* feat(pass-through): add eu/jev decision API pass-through with spend tracking
Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted
decision API that speaks the same POST /v1/systemone contract as TypeSafe.
It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to
https://jev.bevel.software), books spend through the shared decision logging
handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the
cost map
* fix(pass-through): default eu_jev to its canonical eu-jev.com origin
* fix(pass-through): reject custom_body on the eu_jev route
A key limited to other models could send an eu/jev request through
custom_body, because auth checks the outer body while the shared
pass-through forwards custom_body upstream. The route now answers 400
before anything is forwarded, the same way validate_oss_request guards
the laya and bespoke routes
* fix(pass-through): give the eu_jev route request and 200 response schemas
The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and
OA003. The route now documents the eu/jev System One request body and
its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI
document. The lazy OpenAPI snapshot and the dashboard API types are
regenerated with python -m litellm.proxy._lazy_openapi_snapshot and
npm run gen:api
* fix(pass-through): reject streaming on the eu_jev route
A body with stream=true switched the route to streaming, which skips the
decision spend tracking. Reject it with a 400, as validate_oss_request does
for laya and bespoke.
* fix(pass-through): type the eu_jev request body like the other decision routes
* fix(pass-through): check eu_jev decisions against the model eu/jev runs
Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted
model and the legacy names to eu-jeff-latest for key model checks and pricing,
check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when
EU_JEV_API_KEY is unset
* fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases
Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so
model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and
upstream still gets the client's exact body. eu_jev is a registered decisions
provider, so access groups, config deployments, /v1/systemone and /v1/decisions
accept it, and the Admin UI can add it. A release without its own cost map row
is priced as eu-jeff-latest instead of $0
* fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models
* fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it
* fix(eu_jev): take pass-through credentials only from proxy-wide deployments
The eu/jev route now ignores team-scoped deployments when it picks the
upstream key and base, so one team's deployment can no longer redirect
another team's calls, and a base ending in /v1 is not doubled
* fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it
Scope the mapped pass-through and gateway allowlist entries to the decision
route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep
their auth, metering, and gateway behavior from before this PR
* fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it
Drop the decision path from the mapped pass-through list and classify it with one
check that asks the pass-through registry, so an operator's own pass-through at that
path keeps its auth, metering, LLM-route status and litellm_user_api_key handling
from before this PR. The built-in route now yields on a per-request root path and
on any configured method, and the gateway keeps only the exact decision path
* fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding
* test(eu_jev): annotate new test variables with Final
* fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches
Swap an updated pass-through's routes with no await between removal and
re-registration, so the built-in eu/jev route never sees the path as
unconfigured mid-update. The built-in eu/jev handler no longer takes a
configured route's target, and the built-in route also yields on a
method-only match, so GET and PUT on the no-slash form of a slash route
redirect again.
* test(pass-through): annotate the route-swap test's locals as Final
* fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker
* fix(eu_jev): leave a null model out of the upstream decision request
A JSON model of null is limited and counted as eu-jeff-latest, so send the
request upstream without a model field, the way an omitted model goes, since
eu/jev answers a null model with 400 Unknown model
* fix(pass-through): keep route checks on a path another auth endpoint still serves after an update
Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one
* test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it
* refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): support LM Studio decision models
Co-Authored-By: Codex <noreply@openai.com>
* fix(ci): resolve OSV dependency findings
Upgrade the extension MSAL dependency and document expiring exceptions for unused Windows filelock APIs while Python 3.10 remains supported.
Co-Authored-By: Codex <noreply@openai.com>
* Clean up ignored vulnerabilities in osv-scanner.tomlchore: remove unrelated dependency and scanner changes
Removed ignored vulnerabilities related to Windows issues and Python version compatibility.
* chore: remove unrelated dependency and scanner changes
* chore: remove unrelated dependency and scanner changes
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: ishaan-j-berri <ishaan.j@berri.ai>
* test: bill each completed request once and price unpriced returned decision models at the requested model
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(logging,decisions): bill each request once and price returned decision models only when they carry rates
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(decisions): leave the double-billing fix to #47510 and keep the decisions pricing and missing-answers fixes
* fix(decisions): name malformed answer entries and pick the Arize cache-hit span by attribute
* test(decisions): cover a returned-model row with metadata but no rates
* fix(decisions): keep upstream reply text out of the parse error so it stays a 500
* fix(decisions): answer every malformed upstream reply with a 500, whatever text it carries
A reply that fails the decisions response schema now raises
DecisionsResponseValidationError, which the decisions call maps straight to
InternalServerError instead of passing it through exception_type. Its text
matching read an answer key like rate_limit, or a value like "Request timed
out", as a 429 or a 408, so the router cooled down a deployment for a reply
that fails the same way on every retry.
The message also drops the empty location for a reply that is not an
object, the Arize family test loses the cache_hit entries main already
added, and the Bedrock Mantle count-tokens wire test calls
_metadata_deployment, since main renamed the _deployment helper under it.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): support Entra auth for Decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure-ai): select Decisions provider by name
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): prioritize deployment credentials over env key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure-ai): restore api key precedence for Decisions auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): read a deployment's own supports_box_questions flag before the cost map
The box-question check only read the shared cost-map row and required a decisions mode there, so an OpenAI-format deployment whose model_info sets supports_box_questions was refused with a 400, which kept the wire test from #46633 red on main
* fix(decisions): keep the box flag on its deployment and out of request bodies
A model_info that is not a plain dict no longer fails the call, the proxy
decisions endpoint drops a caller-sent model_info, and the router stops
copying supports_box_questions onto the shared cost-map key.
* fix(decisions): keep the box flag shared with same-model deployments like main
Scoping supports_box_questions to one deployment turned box questions on an unflagged sibling of the same backend model from reaching the provider into a 400. Siblings inherit the flag through the shared cost-map row again, as they do on main. Also formats two new tests.
* test(decisions): pin box flag inheritance by an unflagged same-model deployment
* test(decisions): audit box-question deployment flag across providers, routes, and chaos cases
* feat(typesafe): support Jev 1.14 images, box questions, and long-context headers
* fix(cost): drop the typesafe/jev-1.14.0 entry main already prices
* fix(typesafe): send images only with the images header and box questions only to models that answer them
TypeSafe takes assets and input_image parts only when the request carries
TypeSafe-Danger-Zone-Images: true, so without it they follow drop_params like
any other provider that cannot take images. Box support now comes from a
supports_box_questions cost-map flag set on jev-1.14.0 and jev-preview, so
jev-1.13.0 and jev-latest refuse or drop a box question before the call. A
malformed provider_specific_header answers 400 instead of 500.
* fix(typesafe): allow null box confidence and validate asset references and base64
* fix(typesafe): gate box questions on the images header, drop unreadable assets under drop_params, forward opt-in headers on the pass-through
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add decisions and classifier providers to the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): use literal upstream decisions paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: drop duplicate TYPESAFE provider fields entry superseded by main's TypeSafe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): drop stray blank line in provider_info_helpers test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the OSS decisions provider fields test
* fix(decisions): keep Laya and Bespoke env keys on their configured server
Route laya and bespoke Decisions deployments through the same oss_connection
and validate_oss_model rules the pass-through and the classifier router use,
so LAYA_API_KEY and BESPOKE_API_KEY only go to LAYA_API_BASE and BESPOKE_API_BASE,
an api_base with credentials, a query, or a non-HTTP scheme answers 400, and
an unknown checkpoint name answers 400 before any upstream call
* fix(decisions): send the Laya and Bespoke env key to a form-added model on the env server
A deployment added through Add Model always carries an API base, so the
env key only reached deployments with no base. The env key now goes to a
deployment whose base matches LAYA_API_BASE or BESPOKE_API_BASE, an empty
deployment key counts as no key, and the Bespoke placeholder is the name
Ollama serves.
* test(decisions): cover the env-base classifier case in the complexity router test
* fix(decisions): match the env server with or without /v1 and point Ollama users at bespoke/nimble
* fix(decisions): call /v1/systemone once from classifiers and pass-through on a /v1 base
* fix(decisions): strip /v1 from the API base path only, so a host named v1 still works
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): send images to Cloudflare Clef
OpenAI-format input_image parts and the System One images field now reach Cloudflare clef and clef-flash in its images field, with remote URLs downloaded and inlined first. Providers that cannot take images drop them under drop_params and answer 400 otherwise.
* fix(decisions): follow drop_params for images and use the call timeout for image downloads
* fix(decisions): drop a malformed images entry on its own under drop_params
* fix(decisions): refuse images next to input whatever drop_params says
* fix(decisions): bound clef image fetches, honor drop_params for images, and log caller URLs
Refuse more than 4 images for Cloudflare clef before any download, keep the
caller's timeout as one deadline across image fetch retries, accept images
next to input under drop_params, validate image strings and objects, and pass
the caller's image URLs (not inlined bytes) to logging callbacks.
* fix(decisions): check input_image URLs and clef image sizes
* fix(decisions): accept data URL media type parameters and check clef image types
* fix(decisions): count clef images after dropping unsupported types and flag clef vision
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from <provider>/decisions/<model> cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): mark decisions pricing rows with mode decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): sync generated API types for decisions mode
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): update pricing expectations in proxy tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): price classifier calls from decision rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): drop redundant evaluation handler after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): inject pricing row for deployed classifier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): ship the decisions pricing lookup without cost map rows
Restore both cost maps to main. Tests inject their own
<provider>/decisions/<model> rows. The Add Model dropdown hides those
pricing keys and offers mode decisions instead of evaluation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): expect the Decisions mode option in the Add Model form
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): bill decisions and systemone from _decisions cost map fields
Replace the provider/decisions/model pricing rows with <field>_decisions
fields on the model's own row. cost_per_token overlays them for the
decisions, adecisions, systemone and asystemone call types and prices
through generic_cost_per_token; chat, responses and batch keep the plain
fields. gpt-6-luna gains its Decisions rates (input 1e-07, output 0,
cached and cache-write input 1e-07, 2x above 272k tokens).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): satisfy type gate, schema sync and test_utils schema for _decisions fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): import cost_calculator module to keep basedpyright unknown-variable count flat
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): drop stray fireworks import from __init__ and annotate test cost rows for the type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover gpt-6-luna decisions and systemone above the 272k tier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): keep the decisions pricing block clear of main's stale azure comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost_calculator): drop stale azure models doc comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): support Clef-omni audio and video
* fix(decisions): accept Clef-omni media types with parameters and cover the proxy routes
The clip patterns now mirror Cloudflare's input schema, so a data URL or
content_type carrying parameters like audio/webm;codecs=opus is accepted.
The four proxy routes get media tests of their own, the backup cost map
is synced with the new Clef-omni rows, and the docs page moves to the
docs repo as a follow-up.
* fix(decisions): refuse parameterized media-type data URLs and refresh the OpenAPI snapshot
Cloudflare answers 422 to a data URL whose media type carries parameters even though its schema only requires the data:audio/ prefix, so LiteLLM now refuses those with a 400 before the call. The object form keeps passing parameters through, which Cloudflare accepts. The lazy OpenAPI snapshot and the dashboard API types are regenerated for the new audio and videos fields
* fix(decisions): gate media by cost-map flags, treat null clips as none, drop unsupported clips before validation
* fix(decisions): keep the SDK's positional order and gate media flags on evaluation-mode models
* fix(decisions): gate media on both decisions model modes and move the audit cells onto the harness helpers
* test(decisions): cover the Fireworks System One router in the media refusal cells
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(fireworks_ai): add Decisions support for accounts/fireworks/routers/rune on /v1/systemone and /v1/decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(fireworks_ai): point the rune cost row at the public Fireworks docs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): patch litellm.callbacks with monkeypatch in the Fireworks cached-cost test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): call the ai_decide AI Function route for decisions and the auto-router
Databricks serves ai_decide at POST <host>/api/2.0/ai-functions/ai-decide, not
on a serving endpoint's /invocations route. The decisions provider and the
auto-router classifier now accept only the ai_decide model, post to that route
without a model field, unwrap the response envelope, and map a noul answer's
probability to the System One noul field.
* test(databricks): bound the megabyte model name test by scaling, not wall time
* test(databricks): assert the whole OpenAI-format decision and floor the liveliness bound
* test(decisions): move the Databricks OpenAI-format wire row to the ai_decide route
* fix(databricks): keep serving endpoint decisions beside the ai_decide route
* test(databricks): assert the whole classifier request body in the test routing case
* test(databricks): correct System One spend call type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): add litellm.systemone and trim litellm.decisions to the OpenAI shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run tests/unit/systemone in the core-utils unit shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): keep litellm.decisions(state=...) working with a DeprecationWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make tests/unit/systemone a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): surface the litellm.decisions(state=...) deprecation as a UserWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): switch /v1/decisions to the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): build the usage the OpenAI response schema now requires
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): ask signal questions through the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): refuse safety_identifier for System One providers unless drop_params
* feat(decisions): refuse safety_identifier on providers that cannot take it unless drop_params drops it
* test(decisions): cover hosted_vllm in the safety_identifier and per-provider wire tests
* fix(decisions): check a System One request's safety_identifier against the provider too
* fix(decisions): drop a non-string safety_identifier under drop_params
A malformed safety_identifier now follows the drop_params convention in both body shapes: it answers 400 without drop_params and is dropped before the provider call with it. A string identifier on OpenAI stays on the wire either way.
* fix(decisions): let /v1/decisions drop a non-string safety_identifier under drop_params
The route checked the whole body before routing, so a non-string
safety_identifier answered 400 even when the deployment or the body set
drop_params. The route now leaves that field to the Decisions call, which
knows every drop_params source
* refactor(decisions): return the unsupported safety_identifier as a value and raise it in _prepare_call
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(azure_ai): support Microsoft-Decision-1 on the decisions API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): strip project path and API suffix together in decisions base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider
* fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/
* fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor
* fix(ui): name every OSS classifier provider in the classifier radio copy
* feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry
* fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry
* fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router
A /model/new for an auto router lands on one replica. A request routed to
that auto router on a sibling replica missed it, the model read-through
loaded only the auto router's own row, and the pre-routing hook then picked
a tier the sibling had not loaded, so the caller got a 400 "no healthy
deployments for <tier>" until the next periodic reload. An auto router miss
now runs the full add_deployment reconcile, tiers included. Found by the
audit's two-worker rig on this PR's routing cells; CircleCI's one-worker
proxy never opens the window.
* test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier
Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing
cells for the complexity router's databricks classifier on all three
endpoints with the Postgres rows read back, management cells for the
classifier's spend writes and for the evaluation-mode health check of a
Databricks serving endpoint.
* feat(decisions): add hosted_vllm provider
Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move the hosted_vllm placeholder key to constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions
The System One request format moves to /v1/systemone and /systemone. /v1/decisions and
/decisions now accept the OpenAI Decisions API format, translate it into a System One
request, route it through the same pipeline, and translate the answers back.
* fix(decisions): import assert_never from typing_extensions for Python 3.10
* fix(decisions): cap OpenAI-format questions at the System One limit
/v1/decisions accepted up to 200 questions, but the System One request it translates to takes at most 128, so 129 to 200 questions failed with an internal validation error. Both now share MAX_DECISION_QUESTIONS.
* fix(decisions): return 400 for bodies that are not JSON
* test(decisions): send System One bodies to /v1/systemone in integration tests
* feat(decisions): add OpenAI as a Decisions provider
System One requests to openai/ models are translated to OpenAI's Decisions shape on the way out and OpenAI's answers are translated back, so both /v1/systemone and /v1/decisions can route to gpt-6-luna.
* refactor(decisions): move the OpenAI wire translation into llms/openai
* feat(decisions): route both request formats through a shared decisions IR
System One and OpenAI-format bodies now convert to one internal representation, and each provider translates it to its own wire format. OpenAI deployments receive the caller's messages, images, names, typed choice values, level descriptions and safety_identifier unchanged. System One providers return a 400 for image input. Cached and cache-write tokens are priced on both response shapes, and provider response extras survive the round trip.
* fix(decisions): keep provider fields inside System One answers
* fix(decisions): load on Python 3.10 and 3.11 and bill cached tokens once under custom pricing
Decisions IR dataclasses used a mappingproxy default, which is only hashable from Python 3.12, so import litellm failed on 3.10 and 3.11. Decisions usage now reaches cost_per_token as a Usage with prompt_tokens_details, so custom pricing no longer adds cached and cache-write tokens on top of input_tokens that already include them
* fix(decisions): honor litellm OpenAI key and base settings for the openai provider
OpenAI decisions now resolve the key from litellm.api_key, litellm.openai_key, then OPENAI_API_KEY, and the base from litellm.api_base, OPENAI_BASE_URL, then OPENAI_API_BASE, matching other OpenAI calls. Providers own their configured key and base lookup through the endpoint config.
* test(decisions): clear every OpenAI base setting in the proxy OpenAI deployment test
* fix(decisions): send OpenAI instructions for System One questions without them and reject single-option questions
* fix(decisions): return guardrail_information on OpenAI-format decisions when requested
* refactor(decisions): validate proxy request data before reading guardrail settings
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions
The System One request format moves to /v1/systemone and /systemone. /v1/decisions and
/decisions now accept the OpenAI Decisions API format, translate it into a System One
request, route it through the same pipeline, and translate the answers back.
* fix(decisions): import assert_never from typing_extensions for Python 3.10
* fix(decisions): cap OpenAI-format questions at the System One limit
/v1/decisions accepted up to 200 questions, but the System One request it translates to takes at most 128, so 129 to 200 questions failed with an internal validation error. Both now share MAX_DECISION_QUESTIONS.
* fix(decisions): return 400 for bodies that are not JSON
* test(decisions): send System One bodies to /v1/systemone in integration tests
* fix(decisions): return guardrail_information on OpenAI-format decisions when requested
* refactor(decisions): validate proxy request data before reading guardrail settings
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): keep APIConnectionError for unreachable deployments on every provider
`_map_upstream_exception` now maps a status `_handle_error` synthesized for a
non-HTTP failure to `APIConnectionError` before `exception_type` runs, so a
Perplexity (openai-compatible) deployment that cannot be reached no longer
answers `InternalServerError` where main answers `APIConnectionError`.
Audit cells: the unreachable-deployment integration cell is parametrized over
the five providers, OpenRouter connection failures are pinned on chat, stream,
embeddings, decisions and the SDK, a deployment whose provider has no decisions
config gets the 400 naming every supported provider, and a unit cell pins the
mapper alone. The first docstring sentence of the five translation bases is
dropped.
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): register typesafe as a provider so Jev deployments load
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move provider endpoints under llms and validate proxy bodies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add Cloudflare Clef and Strands Decider backends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): register decisions routes for managed agents and gateway
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): use raw regex for cloudflare missing account match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): avoid cast in Cloudflare response unwrapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): default model, evaluation health probe, short Cloudflare names
The proxy validates only state and questions, so a request without a
model falls through to the configured default model like every other
route. Health checks probe evaluation-mode deployments through the
Decisions API instead of failing with an unsupported mode, and
cloudflare/clef and cloudflare/clef-flash get cost-map rows so the short
names resolve a mode and a price. The registry no longer claims typed
decisions for a provider with no backend.
* fix(decisions): let health_check_params override the evaluation probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit the decisions endpoint across providers, limits, health and chaos
Adds the /v1/decisions audit cells: one wire contract per provider (path, key, body and cost-map billing), the gateway-only fields and tags, the sad paths (invalid bodies, unknown model, key checks, api_base in the body, upstream 401/429/500, a 200 without answers, an unreachable upstream), the two evaluation-mode health probes, and three chaos cells (a mixed-failure burst over both routes, a worker SIGKILL mid-burst, an upstream outage and restart on the same port).
The PR's cost case read the upstream observations through the gateway, which answers 404 for that path; it now reads them from the upstream URL. The owned proxy harness takes extra CLI arguments, and its graceful stop waits as long as a worker boot may take, since a worker still starting honors SIGTERM only once it is up and the 30 second wait forced a cleanup under load.
* fix(decisions): send env API keys to a configured api_base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add zero-cost evaluation cost-map entry for Strands Decider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): register the routes through the lazy feature registry
The Decisions router was included at import, ahead of the config and DB
pass-through endpoints, so a pass-through configured at /v1/decisions
was skipped and answered 400 as an unknown Decisions provider. The
routes now register through LAZY_FEATURES, which splices them in after
every eager route, so a pass-through at /v1/decisions keeps its route
while /decisions still serves natively. The lazy OpenAPI snapshot carries
the two paths so the schema shows them before the first call.
The audit cells add the env-key egress to a configured api_base, the
client api_base opt-in shared with chat, the pass-through precedence on
an owned proxy, and the Strands evaluation health check resolved from
the cost map. The integration config exports the Perplexity env key the
first cell needs.
* fix(decisions): keep the Cloudflare api_base message in its transformation and read the audit upstream once per cell
* fix(proxy): let a config pass-through beat a lazily registered route in eager mode
With LITELLM_DISABLE_LAZY_ROUTES set the decisions routes are registered at
startup, so SafeRouteAdder treated a config pass-through at exactly
/v1/decisions as already registered and dropped it. In lazy mode a pass-through
created through the API after the first native call was skipped the same way.
Routes a lazy feature owns no longer count as registered, and a route added at
one of their paths is placed ahead of them, the precedence lazy mode gives a
config pass-through when the feature has not loaded yet.
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
