* fix(bedrock): send native structured output first on Opus 4.8 and Sonnet 5 and fall back to the json tool when Bedrock rejects it
* fix(bedrock): retry with the json tool when Bedrock refuses the native output schema
Count a 400 under output_config.format.schema: as a native output rejection, so a schema Bedrock cannot compile (like a pattern with a nested {n,m} group) gets the json tool retry instead of reaching the caller
* fix(bedrock): make the native output retry send what an opted-out deployment sends
The retry after a native output rejection now reruns the same optional param
mapping completion() ran, with the deployment model_info opted out of
supports_bedrock_native_structured_output, so tool_choice, effort and the json
tool schema match an opted-out deployment instead of a hand-built tool. Each
config keeps only the predicate that decides when native output came from the
opt-in key.
Bedrock batch records for Anthropic models map with the opt-in turned off,
since a batch job has no retry. Models on supports_native_structured_output
keep native output in batch
* fix(bedrock): build the messages retry from the caller's request and keep other errors unchanged
* fix(bedrock): give sync converse stream errors the upstream body as text
* fix(bedrock): let a malformed caller outputConfig reach Bedrock instead of raising
A Converse json_schema request whose caller outputConfig failed litellm's
shape check raised a pydantic error before the request was sent, so the
caller got a 500 where the merge base passed Bedrock's 400 through. The
opt-in check now treats that outputConfig as carrying no native output.
Also opts the vertex wire test's pinned Opus 4.8 json tool deployment out of
native output, and extends the Bedrock native output integration tests.
* fix(bedrock): send a refused caller outputConfig once and give opt-out advice only when native output is unsupported
A caller can send its own outputConfig on Converse. When Bedrock refused that
schema on an opt-in deployment, the retry still carried the caller's textFormat,
so Bedrock got the same request twice and the warning told the operator to turn
off an opt-in that was not the cause. The retry is now skipped when the opted-out
request would still carry a native textFormat, so Bedrock's 400 reaches the
caller after one call as on the merge base. The opt-out advice is logged only for
the account-level "Extra inputs are not permitted" rejection.
* fix(bedrock): keep the caller's continuation message on the native output retry
The Converse transform pops user_continue_message from litellm_params on the first attempt, so a retry built from those params afterwards dropped it. Both Bedrock chat routes now hand the fallback a copy taken before the first attempt.
* test(bedrock): pin additional_drop_params on the /v1/messages native output retry
* fix(bedrock): give the Converse native output retry the fake stream timeout
* fix(bedrock): use native structured output on every Claude model whose Bedrock profiles accept it
Bedrock accepts outputConfig structured output on every reachable profile of
Claude Opus 4.7, 4.8, 5, 5.5, Sonnet 5, 5.5, Haiku 5.5, and Fable 5.1, so
json_schema now goes out natively there instead of as a forced tool, which kept
Bedrock from returning adaptive thinking. Fable 5 rejects outputConfig, so its
keys go back to the tool path. GovCloud keys keep their flags, and the drift
tests let GovCloud lag its commercial base but never lead it.
* fix(bedrock): clean Claude schemas for native structured output and keep recursive schemas on the old path
Bedrock native output rejects minimum/maximum, minItems, uniqueItems, oneOf,
additionalProperties true, and recursive or unresolved $ref, all of which the
forced tool accepted. Claude schemas on the Bedrock native paths now go through
the cleaner the Anthropic native path already uses, and a schema with a
recursive or unresolved $ref keeps the forced tool, inline, or drop path it had.
* fix(bedrock): keep older installs on the json tool with a Bedrock-only native output key
Older installs fetch the model map from main, so flipping supports_native_structured_output
moved them onto native output with their old schema handling. The new
supports_bedrock_native_structured_output key is ignored by older code. Schemas native output
cannot hold (recursive refs, open objects such as dict fields) keep the json tool on the new
key, models on the old key keep their current behavior for those shapes, and the Anthropic
API, Vertex AI, and Azure keep their current behavior.
* chore(cost-map): regenerate the model prices schema for the Bedrock native output key
* fix(bedrock): keep native output to profiles every account accepts and route schemas Bedrock rejects to the json tool
Only Opus 5.5, Sonnet 5.5, Haiku 5.5 and Fable 5.1 keep the Bedrock native output key, since
Opus 4.8 and Sonnet 5 answer 400 on some accounts and Opus 5 garbles escaped characters.
Lookaround patterns and string formats outside Bedrock's list now keep the json tool, objects
typed as a union with null are closed, and the cleaner keeps minItems of 0 or 1.
* fix(bedrock): keep Opus 5.5 and Fable 5.1 on the json tool, since native output stops their adaptive thinking
* fix(bedrock): keep shared $defs in native schemas and send refused schemas to the json tool on every native model
Native output now keeps $defs instead of inlining each use, so a submodel reused across a schema counts once against Bedrock's optional and union limits. Schemas native output cannot hold use the json tool on Converse and Invoke chat for both native keys, and minItems is only kept when it is an integer 0 or 1
* fix(bedrock): keep the json tool unless thinking is requested and bound native output schemas
The Bedrock native output key now applies only when the call asks for thinking, so calls without
thinking keep the json tool as before. Schemas past Bedrock's documented union and optional counts,
or past a grammar size estimate fitted to live probes, keep the json tool on both map keys. Open
objects keep the json tool on the Anthropic API, Vertex AI and Azure AI, where native output returned
them empty. The $ref check no longer expands $defs, so doubling definitions no longer stall the request.
* fix(bedrock): resolve root-path $defs refs so nested Pydantic models keep native output on Invoke
* test(bedrock): cover native structured output and json tool routing through the proxy
Integration cells drive the proxy against a scripted Bedrock peer that enforces Bedrock's structured output schema rules, covering thinking with native output on Converse, Converse streaming, Invoke, /v1/responses and /v1/messages, the deployment opt-in, Fable 5, schema cleaning on every native entry, open objects, and malformed schemas. Vertex, Anthropic and Azure AI cells cover open objects on native-capable models and the cleaner's minItems and nullable object handling. The unit table now covers reasoning_effort none and disabled thinking
* fix(bedrock): keep native output for numeric formats and wide old-key schemas, and keep thinking in json mode
Check string formats only on nodes that can be strings, apply the 64-field cap only to the Bedrock-only key, carry thinking blocks into the json mode message, and send malformed json tool schemas as object schemas.
* fix(anthropic): keep native output for dict fields unless the json tool can be forced
* fix(anthropic): keep native output for dict fields when the caller sends tools
* test(bedrock): close the dangerous-tool-use test schema, since Bedrock native output requires closed objects
* test(anthropic): cover forced and unforced open-object routing, json mode thinking, numeric formats and wide schemas
Add proxy cells for open-object schemas that keep native output when the json tool cannot be forced (a model that refuses forced tools on the Anthropic API and Vertex AI, thinking, caller tools), for the thinking block a json mode answer keeps on Vertex AI and Bedrock Invoke, and for integer and number formats and 65-field schemas on each Bedrock map key. Point the forced open-object cells at the Anthropic API and Azure AI only, since no Vertex AI model in the cost map takes both native output and a forced tool
* fix(bedrock): keep the native output opt-in on the deployment that sets it, and route effort none open objects to native output
* test(bedrock): cover deployment-scoped native output opt-in, register_model and effort none open objects
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 4b7b8efd4c)
* revert(responses): revert "fix(responses): keep gpt-5.4/5.5 tool calls on chat and merge bridged tool calls into one choice" (#44295)
This reverts commit ca1994e403.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): merge bridged tool calls into the same choice as the text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5724117116)
* fix(proxy): stamp the client alias on a copy of each streamed chunk so pricing sees the deployment model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): streamed alias matching a capability rule bills the deployment price
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): assert every streamed chunk carries the client alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): log the client alias on the priced streamed response, the same as non-streamed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 7b432d78d2)
The decision model guardrail backport (#45894) landed on the line after this branch
forked and imports supported_decisions_providers from litellm.decisions.main, which
#45846 replaces here. Match main by exposing it from litellm.decisions.call and point
the guardrail endpoint and its test at that module. schema.d.ts and the lazy OpenAPI
snapshot are regenerated on the merged tree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
schema.d.ts is regenerated with npm run gen:api instead of hand-merged, the Strands logo the Add Model list imports is copied from main, and the picked tests get the import order and Final import this line's ruff asks for
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): add litellm.systemone and trim litellm.decisions to the OpenAI shape
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: run tests/unit/systemone in the core-utils unit shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): keep litellm.decisions(state=...) working with a DeprecationWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make tests/unit/systemone a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sdk): surface the litellm.decisions(state=...) deprecation as a UserWarning
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 25d9b05386)
PR #45667 removed openrouter/typesafe/jev-1.13 from the cost map after #45640 moved OpenRouter decisions billing to usage.cost. The OpenAI-format wire test still read the removed row and raised KeyError on CircleCI integration-providers
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 5894968c5d)
* feat(strands_decider): call a Strands Decider on an AgentCore runtime ARN
A strands_decider deployment whose api_base is a bedrock-agentcore runtime ARN now
calls InvokeAgentRuntime on that runtime, signed with SigV4 from the deployment's aws_*
params or the default chain, reuses one session per runtime, and maps the runtime's
200 error envelopes to 400, 503, or 500
* ci: run the strands_decider unit tests in the llms job
* fix(strands_decider): take the AgentCore session id from the deployment, never the caller
A caller could pick the runtime session through extra_headers, and every new id starts another microVM. The session id now comes from the deployment's agentcore_runtime_session_id (banned in request bodies) or the ARN default, and caller-supplied host, x-amz-*, and x-amzn-bedrock-agentcore-runtime-* headers are not forwarded
* fix(strands_decider): map runtime error replies only on AgentCore ARN calls
* fix(strands_decider): sign plain decisions calls inline, off the AWS signing executor
* test(strands_decider): cover AgentCore runtime ARN calls in integration wire tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit ed9c31cfa8)
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): switch /v1/decisions to the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): build the usage the OpenAI response schema now requires
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): ask signal questions through the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): refuse safety_identifier for System One providers unless drop_params
* feat(decisions): refuse safety_identifier on providers that cannot take it unless drop_params drops it
* test(decisions): cover hosted_vllm in the safety_identifier and per-provider wire tests
* fix(decisions): check a System One request's safety_identifier against the provider too
* fix(decisions): drop a non-string safety_identifier under drop_params
A malformed safety_identifier now follows the drop_params convention in both body shapes: it answers 400 without drop_params and is dropped before the provider call with it. A string identifier on OpenAI stays on the wire either way.
* fix(decisions): let /v1/decisions drop a non-string safety_identifier under drop_params
The route checked the whole body before routing, so a non-string
safety_identifier answered 400 even when the deployment or the body set
drop_params. The route now leaves that field to the Decisions call, which
knows every drop_params source
* refactor(decisions): return the unsupported safety_identifier as a value and raise it in _prepare_call
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 546402c98c)
* feat(azure_ai): support Microsoft-Decision-1 on the decisions API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): strip project path and API suffix together in decisions base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 2fcc780498)
* feat(decisions): add hosted_vllm provider
Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move the hosted_vllm placeholder key to constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 61e5f2dd3c)
Cherry-pick of #45663 adapted to rc/1.105.0, which predates
litellm/decisions/call.py, GuardrailReadOnlyDetails.tsx, LiteLLMBaseModel,
@/lib/decisionModels and the TypeSafe Add Model provider.
- supported_decisions_providers() lists rc's DECISIONS_ENDPOINTS in
litellm/decisions/main.py
- The read-only decision model section moves to
decision_model/DecisionModelDetails.tsx, rendered by guardrail_info.tsx
- DecisionModelCheck extends LiteLLMPydanticObjectBase
- isDecisionMode is inlined and the TypeSafe garden card bundles typesafe.png
- The OpenAPI snapshot and schema.d.ts were regenerated on this branch
- Masked guardrail params are built with BaseLitellmParams.model_validate,
which keeps rc's type-check budget with the two new typed fields
(cherry picked from commit 5f9dd38867)
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): keep APIConnectionError for unreachable deployments on every provider
`_map_upstream_exception` now maps a status `_handle_error` synthesized for a
non-HTTP failure to `APIConnectionError` before `exception_type` runs, so a
Perplexity (openai-compatible) deployment that cannot be reached no longer
answers `InternalServerError` where main answers `APIConnectionError`.
Audit cells: the unreachable-deployment integration cell is parametrized over
the five providers, OpenRouter connection failures are pinned on chat, stream,
embeddings, decisions and the SDK, a deployment whose provider has no decisions
config gets the 400 naming every supported provider, and a unit cell pins the
mapper alone. The first docstring sentence of the five translation bases is
dropped.
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 0e09e99146)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add unified /v1/decisions endpoint for Jev-compatible providers (#44236)
Backport of #44236 to rc/1.105.0.
Cherry-picked from 8b1990b4bc (main).
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions
Backport of #45184 to rc/1.105.0.
Cherry-picked from 3339469e73 (litellm_systemone_endpoint). The playground System One tab is not on this line, so its UI hunks are dropped.
* fix(decisions): import assert_never from typing_extensions for Python 3.10
Backport of the #45184 follow-up to rc/1.105.0.
Cherry-picked from 29e715473c (litellm_systemone_endpoint).
* fix(decisions): cap OpenAI-format questions at the System One limit
Backport of the #45184 follow-up to rc/1.105.0.
Cherry-picked from 7ef132947a (litellm_systemone_endpoint).
* fix(decisions): return 400 for bodies that are not JSON
Backport of the #45184 follow-up to rc/1.105.0.
Cherry-picked from b6bb885324 (litellm_systemone_endpoint).
* test(decisions): send System One bodies to /v1/systemone in integration tests
Backport of the #45184 follow-up to rc/1.105.0.
Cherry-picked from 77872f6cda (litellm_systemone_endpoint).
* feat(decisions): add OpenAI as a Decisions provider
System One requests to openai/ models are translated to OpenAI's Decisions shape on the way out and OpenAI's answers are translated back, so both /v1/systemone and /v1/decisions can route to gpt-6-luna.
(cherry picked from commit b94bbbc318)
* refactor(decisions): move the OpenAI wire translation into llms/openai
(cherry picked from commit d209a85076)
* feat(decisions): route both request formats through a shared decisions IR
System One and OpenAI-format bodies now convert to one internal representation, and each provider translates it to its own wire format. OpenAI deployments receive the caller's messages, images, names, typed choice values, level descriptions and safety_identifier unchanged. System One providers return a 400 for image input. Cached and cache-write tokens are priced on both response shapes, and provider response extras survive the round trip.
(cherry picked from commit 0dd207fde1)
* fix(decisions): keep provider fields inside System One answers
(cherry picked from commit 1348de864f)
* fix(decisions): load on Python 3.10 and 3.11 and bill cached tokens once under custom pricing
Decisions IR dataclasses used a mappingproxy default, which is only hashable from Python 3.12, so import litellm failed on 3.10 and 3.11. Decisions usage now reaches cost_per_token as a Usage with prompt_tokens_details, so custom pricing no longer adds cached and cache-write tokens on top of input_tokens that already include them
(cherry picked from commit 39012f1c22)
* fix(decisions): honor litellm OpenAI key and base settings for the openai provider
OpenAI decisions now resolve the key from litellm.api_key, litellm.openai_key, then OPENAI_API_KEY, and the base from litellm.api_base, OPENAI_BASE_URL, then OPENAI_API_BASE, matching other OpenAI calls. Providers own their configured key and base lookup through the endpoint config.
(cherry picked from commit e465919004)
* test(decisions): clear every OpenAI base setting in the proxy OpenAI deployment test
(cherry picked from commit a2951b318d)
* fix(decisions): send OpenAI instructions for System One questions without them and reject single-option questions
(cherry picked from commit e4081f77c6)
* fix(decisions): return guardrail_information on OpenAI-format decisions when requested
Backport of the #45184 and #45214 guardrail fix.
Cherry-picked from 855f0bb765 (litellm_systemone_endpoint) as merged into #45214 at c19ada66ed (litellm_openai_decisions_provider).
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #44086 to rc/1.105.0.
Cherry-picked from f95446ea39 (main).
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cherry-pick of the squash commit of #44307, which serves gpt-5.6+ chat
completions on the native Bedrock /openai/v1/chat/completions route by
default and adds the chat_completions/ opt-in for gpt-oss and grok.
Conflict resolution: tests/unit/llms/bedrock/test_bedrock_common_utils.py
kept the line's file plus the one test the pick adds, since rc/1.105.0
does not carry #44112's neighbouring tests. The pick's
bedrock_route_for_request builds a MappingProxyType, whose import #44112
brought to main, so the import is added to litellm/llms/bedrock/common_utils.py
here.
(cherry picked from commit 233db9f285)
* test(integration): answer the model-info refresh GET in the mixed MCP responses wire peer
The proxy's periodic model-info refresh sends GET /v1/models to the deployment api_base, which tripped the peer's /responses-only assertion when a tick landed mid-test
* test(e2e): wait for the api-keys URL after clicking Virtual Keys in onboarding
/ui already renders the Virtual Keys heading, so the helper returned before navigation finished. The late route change moved focus and closed the account menu popover in hideLiteAdmin
* test(proxy): stop two unit modules leaking app.openapi_schema and a session-wide Router
test_custom_openapi cached a stripped schema on app.openapi_schema and never cleared it, breaking later openapi route tests. test_proxy_reject_logging built a module-level Router that stayed in the live router registry all session and re-added cost-map keys during a reload. Reset the schema via monkeypatch and make the Router a function-scoped fixture
(cherry picked from commit a292fd409f)
The failure spend-log row from #42695 is written for pass-through routes that
run as LLM API routes, which a config route only does with auth: true. The
test omitted auth and passed only while config wins (#41779) registered config
entries through the typed model, where auth defaults to true. #43962 restored
the pre-config-wins registration, so the route lost that status and the row
was never written. Set auth: true on the route so the test covers the logging
it was written for without depending on that side effect
(cherry picked from commit 1d9cd9b09c)
* test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG
The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context
* test(secret-detection): give the hand-built redaction request an ASGI path
Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path
* test(integration): isolate litellm callback lists per sdk test
usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards
* test(integration): keep the owner-lookup fault proxy off the shared read replica
The owned proxy points DATABASE_URL at a scratch database but inherited
DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared
database and rejected the freshly created key with token_not_found_in_db.
Drop the replica variable like the other scratch-database owned proxies
* test(integration): request every seeded key in the team owner breakdown
The aggregated team activity endpoint now caps breakdown.api_keys at the top
100 keys by default (#43398), so the 300 seeded keys came back as 100 rows.
The test guarantees each key is reported with its own owner, so ask for an
api_key_limit that covers all seeded keys
* test(integration): give every owned Redis its own port in the redis-cache container
On CircleCI every owned Redis ran on the fixed port 16379 inside the shared
redis-cache container. When an earlier server still held that port, the new
one failed to bind, readiness pinged the old server, the pidfile read failed
and cleanup then reported "Owned Redis still serves after shutdown"
Reserve an ephemeral port for the docker-exec path the same way the local
binary path already does, and refuse to start when something already serves
the chosen port so the failure names the real cause
* test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach
The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail
* test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves
vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to
* test(e2e): check only stored message content for a leaked card number
The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there
* test(e2e): assert the proxy decodes token-array embeddings for titan
The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer
* test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking
us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5
* test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests
Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed
* test(e2e): cite the tokenizer and date behind the titan token-array fixture
* test(e2e): let migration seed replicas finish their request-log indexes before cloning
Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema
* test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope
#43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake
* test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list
test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly
(cherry picked from commit 9b5562f89b)
* refactor(types): replace Any with proven types in 9 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin xecguard-adjacent guardrail retry, mcp mixed tools, jwt routing and complexity router paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): keep only live-provable Any removals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(types): type cache_hit as bool | None on the sync success path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(straiker): assert a saved api_version v1 with an sk_agt_ key routes to v3 (#44106)
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(straiker): ignore zombie workers when counting uvicorn children after a kill
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(straiker): describe the saved v1 cell by the v3 route it asserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* fix(azure_storage): keep client call ids from sharing one Data Lake file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(azure_storage): tighten adls_safe_file_name docstring
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_storage): give ids with dot or empty path segments their own Data Lake file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure_storage): cover failure, cache-hit and edge call ids in Data Lake file names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure_storage): run the Data Lake file name cells on two proxy workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move legacy proxy, router and Redis tests into tests/integration
Port 39 legacy tests to the integration tier that owns them, running against
the scripted upstream, local Postgres and Redis, test-owned wire peers and
owned proxies. Delete 5 legacy tests whose contract is already owned by an
existing integration test, and remove the legacy functions, files and helpers
left unused.
* test(integration): cover recovery of a spent key after its budget is raised
Remove the ENFORCE_PRISMA_MIGRATION_CHECK opt-in. When PrismaManager.setup_database returns False (database unreachable, connection retries exhausted, or prisma migrate deploy failing after retries) the proxy now always prints the red failure message and exits 1 instead of serving requests against a database whose schema may be behind the code. The --enforce_prisma_migration_check flag stays as a hidden no-op that prints a one-line deprecation warning so existing container args keep parsing; the env var is no longer read anywhere. The standalone migration entrypoint always runs run_server(("--skip_server_startup",)), and the integration launchers drop the flag.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared exclusion predicate negated a PostgreSQL ANY comparison without handling NULL, so NULL entity ids evaluated to UNKNOWN and dropped out of the Unassigned bucket whenever exclude_*_ids was set. The paginated daily rows Prisma filter had the same NOT IN shape and gets the same IS NULL OR NOT IN treatment
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add LITELLM_DISABLE_LAZY_ROUTES to register optional routers at startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): register eager lazy routes at startup so late eager routes keep precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): share one optional-feature install path between lazy and eager registration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): say eager lazy routes register at worker startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): name the import callable passed to _install
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove a startup hook can drop eager lazy routes for good
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strip the lazy-routes flag from the lazy-mode control proxies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover the lazy warmup route registering a feature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(unit): run tests/unit/proxy/test__lazy_features.py in the proxy-server-core shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for LITELLM_DISABLE_LAZY_ROUTES
Flag spellings, /openapi.json at boot, the warm-up route in both modes, /mcp/proxy ahead of the /mcp mount,
route table and OpenAPI parity with a fully warmed lazy proxy, a broken optional import in both modes, every
client SDK against the completion endpoints, a boot burst with a killed worker, and a restart
* fix(proxy): keep config pass-through routes ahead of eagerly registered features
With LITELLM_DISABLE_LAZY_ROUTES set, features registered before the proxy lifespan
added config pass-through endpoints, so a pass-through overlapping a feature path
(e.g. a self-hosted /langfuse) lost to the built-in route. Restore lazy mode's
registry order once startup finishes, without bringing back routes a startup hook removed
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): accept Converse messages with no content key
A user or tool message whose content key is missing (or null, which the
message cleanup strips) made every Bedrock Converse request fail with
APIConnectionError 'content' before reaching Bedrock. The Converse
transform now reads content with .get for those messages, as it already
did for assistant messages: a content-less user message adds no block
and a content-less tool message becomes a toolResult with empty content.
The str branch also sends the continue message text instead of the
original whitespace-only text.
* fix(bedrock): send the continue message for a content-less user turn
* test(bedrock): type the content-less Converse message test parameters
* fix(bedrock): accept a Converse system message with no content key
* refactor(bedrock): read the system message content with get
* test(integration): audit Bedrock Converse messages without content
Adds the /audit cells for a chat message whose content key is missing or
null on a Converse-routed Bedrock deployment: happy, sad, edge, and chaos
rows through the OpenAI SDK, the Anthropic SDK, and raw httpx against the
scripted upstream, asserting the caller's response, the body the peer
received, and the spend row. The owned-proxy readiness deadline in the
integration harness is now INTEGRATION_PROXY_READY_SECONDS (default 70).
* test(integration): bound stray spend rows in the mid-burst restart cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(mcp): expand team and dashboard grants when listing and serving toolsets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): honor team toolset grants on the responses gateway path and expose a public team permission lookup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(mcp): drop the toolset route docstring tweak so the OpenAPI snapshot stays unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): scope inherited toolset grants by the key's own MCP ceiling
A key that declares any MCP grant of its own keeps only its own toolsets, one
that declares none inherits its team's, and require_key_mcp_access_defined
stops a virtual key inheriting while dashboard sessions and admitted users
still do. Adds the direct, no-grant, admin and key-ceiling integration cases
and makes the LLM gateway toolset case discriminate a scoped toolset from the
aggregate grant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): resolve toolset grants per admitted source and enforce the live team roster
Dashboard sessions and gateway-admitted users now expand into the admitted
subject's per-team sources when resolving toolset grants, so a team-granted
toolset is not capped by the user's own MCP row and is reachable on the
namespaced route. A cached team id no longer grants a toolset unless the live
roster still lists the user, a team lookup fault only drops that team's
inherited grant, and /team/member_add evicts the cached team object so the new
member is authoritative immediately
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): read a key's named object permission before letting it inherit team toolsets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): inject the toolset grant resolver into scope helpers so tests stop patching MCPRequestHandler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop the duplicate admitted_subject_sources wrapper after merging main and follow its renamed resolvers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): honour fresh policy and the session resource scope on pinned toolsets
A pinned toolset scope now reads the toolset through the writer when the admitted session
requires fresh policy, so a tool revoked from the toolset is gone on the next request. A
gateway bearer scoped to one server can only open a toolset that names that server
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep operator-open servers out of toolset gateway urls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): audit cells for team-granted toolsets across every surface
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): bound toolset edit convergence by both cache layers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): drop toolset integration cells that test behavior this PR does not change
Repeat-read byte identity, 20 concurrent calls, a stopped peer and a killed worker are covered
generically by test_mcp_resilience.py and test_mcp_user_env_vars.py. The toolset edit cell asserts
pre-existing cache propagation and flaked locally with connection resets while polling
* chore(mcp): drop mutable-ok suppressions that main's LIT013 now flags as unused
* chore(mcp): keep the require_key_mcp_access_defined read from adding an unknown-argument type error
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The two SpendLogs index migrations shipped in v1.103.0 each break one table shape: the plain CREATE INDEX holds a SHARE lock on a large unpartitioned table and the CONCURRENTLY one fails with 0A000 on a partitioned parent. Both files are now inert and the indexes are built by a table-driven, shape-aware step after migrate deploy: CONCURRENTLY on a plain table, ON ONLY the parent plus per-partition CONCURRENTLY and ATTACH PARTITION on a partitioned one. The migration job builds them synchronously and exits non-zero on failure; a serving proxy that ran migrate deploy itself builds them in the background off the readiness path. A valid index of the same definition under another name is renamed and reused, an invalid one is rebuilt, and extra copies are reported with their DROP INDEX statement instead of being dropped. The migration checker rejects any CREATE INDEX on LiteLLM_SpendLogs or LiteLLM_ErrorLogs in future migrations
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* add test case for /rag/query and stronger auth check
* style(proxy): ruff format auth_checks.py
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): build rag query vector store ids immutably and test the no-registry path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): integration coverage for /v1/rag/query vector store allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): audit cells for /v1/rag/query vector store allowlist
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): consolidate vector store allowlist audit coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type RAG vector store request body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
