Backport of #50276 to lts/1.108.x. tests/integration/_support/redaction_probe.py also sends the standard key the picked integration test reads (main's probe got it in #47192).
Cherry-picked from 3f13c1ca34 (main). Conflicts in litellm_logging.py and the prompt caching check test were main-only neighbours (_get_completed_success_callbacks and the tools/system affinity tests), resolved by adding only this PR's method and test.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #50274 to lts/1.108.x. The line also lacked the get/delete/cancel responses CallTypes members that main got in #49133, so a during_call guardrail made those routes 500 with 'aget_responses is not a valid CallTypes'. The six members are added here (CallTypes, schema.d.ts, the azure call type list), and tests/integration/conftest.py registers the shares_worker_state marker the picked integration test carries.
Cherry-picked from d295f76c4a (main). Conflicts in litellm/types/utils.py, schema.d.ts and three test files were neighbouring main-only call types and tests, resolved by adding only the compact_responses entries and tests from this PR.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The limits test now runs on Haiku 5.5 with thinking, the opt-in row whose
schemas the picks clean, since Sonnet 4.6 keeps the line's schema as sent.
The open object test checks Converse forwards the caller's schema once with
no JSON tool and answers with Bedrock's verdict. The /v1/messages test compares
against a copy taken before the call, so an in-place edit fails it.
The integration tests that expected an open-object schema to move to the
JSON tool on a native model now expect it to stay native, as the line does.
A Bedrock /v1/messages test checks a row-native profile forwards the caller's
schema unchanged, and a Converse handler test checks a sync stream error keeps
its status class and the line's raw body.
Backport line fix, not a pick. Every change keeps a request that worked on
lts/1.108.x working after the picks in this branch.
Models the cost map marks native on every request (Sonnet 4.6, Haiku 4.5,
Sonnet 4.5, Mythos 5.1) send the caller's schema the way the line did, on
Converse, Invoke and direct Anthropic. #46870 moved some of their schemas to
the forced json tool. That turned array roots and typed allOf roots into 400s,
gave prose for open objects with thinking on, and broke tool_choice without
tools. Opt-in models (Haiku 5.5, Sonnet 5.5, Sonnet 5, Opus 4.8) keep the new
native path and its schema checks.
bedrock_mantle/ chat has no retry, so Mantle ignores thinking when it picks
the path, and opt-in models stay on the json tool there as they did before.
The sync Converse stream keeps the line's error handling, so its error classes
and router retries stay the same. The retry reads the raw HTTP error there
instead. It also sends the caller's headers as they were before the first
attempt, so the structured output beta does not leak into it.
main has the same routing gaps.
Backport of #47100 to lts/1.108.x.
Cherry-picked from 52f5a28679 (main).
Conflicts resolved for the line. The line has no per-attempt timeout split, so
main's _native_output_attempt_timeout and timeout_for_optional_params, and the
two tests that cover them, are left out. Both attempts use the call's resolved
timeout, which on a Router stream is stream_timeout, the same limit the line's
faked stream has today. The cost map JSON hunks are left out because the line
already matches them.
Backport of #49785 to lts/1.108.x.
Cherry-picked from fcef9cd0a3 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48278 to lts/1.108.x.
Cherry-picked from 0b66a27528 (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49462 to lts/1.108.x.
Cherry-picked from 948445c828 (main). get_proxy_wide_deployment_credentials hand-ported onto the line's _resolve_matching_deployment_api_key, since the line has no _get_deployment_credentials.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #49301 to lts/1.108.x.
Cherry-picked from 298911d61b (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48640 to lts/1.108.x.
Cherry-picked from 59b8c73930 (main). Dropped the pick's edit to the ValidationError branch the line does not have.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48913 to lts/1.108.x.
Cherry-picked from 84c4c5aa5c (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #48644 to lts/1.108.x.
Cherry-picked from 8f31c7171d (main). Dropped the pick's edit to a stream-stall test the line does not have.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Scope the Prisma migration lock test's advisory lock check to its own database, so a lock another xdist worker holds in its database no longer trips it
Keep in-process MCP peers' idle connections open for 30s instead of 1s, so the gateway's pooled HTTP client doesn't reuse a connection the peer is closing and hang the tool call
Delete database guardrails in the logging-only scope tests through DELETE /guardrails/{id} on the shared gateway instead of raw SQL, so a default-on guardrail stops applying to the next test on the same worker right away
Bound the wedged coordination Redis /customer/update check by the shortest Redis wait the proxy uses, like the burst check, so a slow runner can't fail it while a handler that waits on Redis still does
(cherry picked from commit 4f2c75ddf5)
Takes only #47108's (e85cb4052e) changes to coordination_redis_proxy_config.yaml and test_user_updates_wedged_coordination_redis.py. They set the coordination Redis socket timeout and define the shortest Redis wait bound that #49554's /customer/update fix uses
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
68db022d57: `AssertionError: (('success',),)`).
The test now reads the spend row first, which is written once the proxy has settled the disconnect, and
releases the upstream after it. The proxy's behavior is unchanged; only the test's ordering is.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
(cherry picked from commit 89a194bbd0)
Takes only the max_workers change in test_team_delete_large_membership.py from 7e8be61fa0. The 16-thread key warmup slowed the /team/delete that follows it to 5.4-11.6s locally against the client's 15s timeout, and 2 threads brings it to 4.1-5.4s
* fix(proxy): keep a retried /v1/messages stream's spend log by logging the superseded attempt as a failure
A /v1/messages stream the router retries before any content shares one logging object across both attempts. The superseded attempt's end-of-stream logging ran as a success (a usage-only or placeholder result) and claimed the request's one-shot success log, so the recovered stream's spend row was either missing or written under the superseded attempt's message id. The passthrough logging path now logs an Anthropic stream whose frames carry an `event: error` as a failure with whatever usage the provider reported, the way a dropped stream already does, and the router closes the superseded stream before it opens the retry so that logging is queued first.
* fix(proxy): log a provider error frame through an attempt-scoped logging copy
The error-frame failure stashed the superseded attempt's usage on the logging
object the retry shares, so the retry's success log carried the failed
attempt's token counts. The failure now logs through a copy of the logging
object with its own model_call_details, and the helper moves into the
Anthropic passthrough logging handler so the proxy streaming handler only
routes by endpoint type.
* fix(proxy): scope the error-frame failure log to streams closed at the frame
A stream that ran to its end at the provider's error frame kept logging
through the success route before this PR, and the attempt-scoped failure
log is now taken only for a stream the router closed at that frame, with
the copy made at the close so the retry's deployment never reaches it
* chore: drop duplicate litellm import and mark the stream_ended rebind
* fix(proxy): log only router-superseded attempts as error-frame failures
A client that hangs up at a forwarded error frame closed the stream the same way the router closes a superseded attempt, so a non-retried stream lost its spend. The router now closes a superseded attempt through aclose_superseded_attempt, and only that close routes the attempt to the failure log
* test: type the _settle helper's recorded events
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 8bb8ac7f18)
rc/1.108.0 carries OpenTelemetry 1.45.1 (#48055), #47611 and the #47639 base_model strip, so the OTel traceresponse header, langtrace outage, guardrail concurrent-disconnect and Bedrock lookaround tests expected the old behavior. This takes #48665's hunks for those four files only, without the unhandled to non_object_body rename, which depends on #48438 that rc does not have
(cherry picked from commit 21ab2cb87b, four test files only)
rc/1.108.0 carries the #43935 backport, which enforces gpt-5's supports_none_reasoning_effort and supports_xhigh_reasoning_effort flags, so the test now reads the levels it parametrizes from the cost map like main does. Only this file of #47600 is taken. Its other files follow changes rc does not have
(cherry picked from commit e78f7b4602, tests/integration/providers/test_cursor_responses_wire.py only)
* test(integration): follow #47441 metadata-routed Mantle counting in count_tokens wire tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): build the runtime count body without mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit f5d6e89ad8)
* test: add a manually dispatched integration trial on larger runners
Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers
run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one
* test: run the integration trial on push to this branch
* test: give integration trial workers a local redis-server
* test: run the providers trial on a 64-core runner with 32 workers
The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4
* test: run every integration group in the trial on sized runners
* test: add a scan that refuses to upload integration results holding the license
* test: drop the member-name pass the archive byte scan already covers
* test: use a secret whose base64 and base64url encodings differ
* ci: run integration groups on run-ci pull requests and scheduled main runs
* ci: scope the integration license to a protected environment
Backport note: tests/integration/AGENTS.md keeps rc/1.108.0's text, since main's copy of that paragraph also carries later changes rc does not have
(cherry picked from commit 62b8ee78de)
Backport note: rc/1.108.0 also needs graceful_stop_seconds imported in test_logging_only_scope_chaos.py, which main already imported
(cherry picked from commit c8c0248ac7)
Read spend-log rows through read_rows, since this line's Gateway has no rows method.
Leave out the sync success callback cell: on this line a callbacks-listed logger's
sync log_success_event does not run for async calls (main gained that in #47192).
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* fix(guardrails): drop the response pipeline steps that never ran from x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
* fix(guardrails): keep request steps when a response pipeline blocks before a later request pipeline
(cherry picked from commit 969ded2b54)
* refactor(guardrails): record which policy pipeline steps and deployment-level hooks ran
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails (#49263)
* fix(guardrails): name only the request pipeline steps that ran in x-litellm-applied-guardrails
* test(guardrails): integration cells for request pipeline steps in x-litellm-applied-guardrails
* fix(guardrails): keep the agent cells on the shared proxy and shorten a long suppression line
(cherry picked from commit fbc157437c)
* fix(register_model): keep bedrock geo deployments priced across price data reloads
A request-time pricing registration inserting into litellm.model_cost while a
price data reload walked it could abort the reload (dictionary changed size
during iteration in add_known_models), or trip the lookup map rebuild inside
get_model_info so the deployment replay fell through to the unmapped path and
wrote a raw bedrock/au.anthropic.<model> entry with no token pricing, a $0
shadow re-rolled on every later reload. Walk snapshots of the map, bind the
case-insensitive lookup map locally, and let the unmapped path inherit the
whole built-in entry of a provider get_model_info would accept.
* fix(register_model): keep synthesized zero costs out of inherited built-in pricing
An unmapped alias that inherits its built-in entry now takes only the token
prices the catalogue really carries, so a synthesized zero never lands next to
the caller's custom price. The router regression test reports the writer's
failures instead of swallowing them, and the local names carry Final.
* fix(register_model): match inherited built-ins by the key provider prefix when the value names none
* fix(utils): resolve a tripped lookup from the key's own catalog row before its region sibling
* fix(register_model): keep custom flat prices above inherited tier rates
* fix(register_model): keep the other direction's tier rates when a custom flat price covers one
* fix(register_model): keep a catalog row's tiers when its lookup trips, and snapshot request-path cost map walks
A tripped get_model_info on a key that is itself a catalog row now inherits that row whole, so a custom flat price bills a long prompt at the row's above-threshold rate on both lookup paths instead of flipping until the next reload. The fal_ai, fireworks, ollama and gpt-5 request-path walks over litellm.model_cost iterate a snapshot so a concurrent reload cannot raise mid-request.
* fix(register_model): price route-prefixed geo keys on a tripped lookup and replace cost map entries atomically
* fix(register_model): move every alias of a cost map entry to its registered price
* perf(register_model): read alias names from the shared entry instead of scanning the cost map
Each registration scanned every cost map entry to find the names sharing it,
about 1 ms on the 4,189-entry map, and a proxy request to a custom-priced
deployment registers twice. Aliased entries now carry the names they are
served under, so a registration reads them directly, and an alias the caller
pointed at its own entry keeps that entry. The tests this PR added lose their
docstrings
* fix(utils): keep capability flags and write through on a tripped lookup in register_model
A key the lookup cannot map now seeds its registered entry from the capability
rules its unregistered lookup resolves to, so registering a region alias no
longer turns off supports_system_messages and similar flags. A lookup that
trips mid-walk on a key naming a catalog row writes into that row instead of
leaving a shadow key that goes stale. Adds regression tests for both and for
the request-path snapshot walks.
* fix(utils): retry a tripped registration lookup and rebuild a registration a reload races
A lookup a concurrent write trips is retried instead of writing into the
catalog row its key names, so a per-request price never reprices a shared
row. The inherited entry leaves the matched row's provider behind, so a
provider-less azure_ai key stays priced, and a registration whose map a
reload swapped mid-build is rebuilt from the new map.
* fix(utils): write each registration into the map it was built from and rebuild it when a reload replaced that map
* test(integration): price data reload keeps every registered deployment priced
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 9e9dccb9d4)
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
(cherry picked from commit 93b79d2587)
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
(cherry picked from commit 09e9193514)
* fix(rust): add the dangerous-tool-use beta when an anthropic /v1/messages request carries safeguards
* test(rust): check the dangerous-tool-use beta reaches the wire with safeguards
* fix(rust): derive the dangerous-tool-use beta only for hosts with a beta map column
DeepSeek and the other hosts without a column in anthropic_beta_headers_config.json
keep forwarding safeguards and client betas as sent. Adds the wire-level audit tests.
* fix(rust): add the safeguards beta on Azure AI and Claude Platform hosts
Azure AI and Claude Platform Messages transforms now add the
dangerous-tool-use beta for a non-null safeguards field, through the
Azure AI and Anthropic beta-map columns. The safeguards wire and chaos
tests get unique file names so they do not collide with the Python
fix's tests, the Python-route test asserts the fixed behavior behind a
BUG skip, and the DeepSeek test checks that no beta is added
* fix(rust): keep Claude Platform out of the safeguards beta until a route can prove it
Claude Platform has no beta-map column and no QA account, so it goes back to sending
safeguards with no derived beta, like DeepSeek. Also release held chaos-test requests
before the thread pool waits on them, and mark the no-column test as rstest.
* fix(rust): derive the safeguards beta for Claude Platform and keep the null-cell test off Databricks
Claude Platform on AWS answers 400 to safeguards without dangerous-tool-use-2026-09-03 and 200 with it, so its Messages transform derives the beta through the Anthropic column again. The null-cell case now uses only Bedrock Converse, since Databricks support for the beta is changing in a sibling PR.
* fix(rust): drop a null safeguards key instead of sending it without its beta
Anthropic rejects any safeguards key, null included, when the
dangerous-tool-use beta is missing, so an explicit "safeguards": null on the
Rust gateway still answered 400. The shared Messages transform now drops a
null safeguards key on every host, as the proxy already drops None params
before calling Rust, so it answers 200 with no beta.
The scripted peer in the integration wire tests now rejects the key on
presence, matching the real API.
* test: cover Bedrock Mantle in the null safeguards case and match the vendor's non-list wording
The Rust null-safeguards regression test now also runs for bedrock_mantle, whose beta is keyed on the field being present. The scripted Anthropic peer answers a non-list safeguards value with the real API's "valid array" wording.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 5b042c0db6)
