* fix(deps): upgrade Python dependencies
* fix(deps): lock patched Python dependency versions
* fix(otel): preserve JSON export with the updated SDK
* test(otel): cover JSON export with the updated SDK
* test(otel): update SDK log records and worker discovery
* fix(ui): align Lens peer dependency with patched Next.js
* fix(otel): hand the AgentOps exporter a session it owns for the JWT header
OpenTelemetry 1.45 moved the OTLP HTTP exporter's requests.Session behind a
transport, so writing the minted JWT to the exporter's private _session raised,
was swallowed, and every AgentOps batch went out unauthenticated. The exporter
now creates the session, passes it through the public session= argument and
sets the Authorization header on it
The tests now drive a real export, serving the token endpoint with respx and
capturing the OTLP POST with requests_mock, instead of reading SDK internals
* fix(otel): skip FastAPI instrumentation once the app has started
opentelemetry-instrumentation-fastapi 0.66b1 wraps build_middleware_stack instead
of calling add_middleware, so instrumenting an app whose middleware stack is
already built no longer raises. The call silently did nothing while we still
registered the global response propagator. Check the built stack up front and
skip both, which is what the existing test expects
* test(otel): assert exported requests and span flags instead of SDK internals
OpenTelemetry 1.45 dropped the exporter's private _headers and sets the W3C
random-trace-id flag on new traces, so traceresponse ends in -03. The Langtrace
and Langfuse tests now export through requests_mock and check the URL and headers
the collector receives, and the traceresponse tests expect the server span's own
trace flags
* fix(deps): bump dashboard zod to 4.6.6 and make the vendored Lens use it
zod 4.6.5 is flagged by the dependency scan and blocks osv-scan on every PR. A
root bump alone leaves the vendored Lens tarball, which pins zod 4.6.5 exactly,
on its own nested copy, so a top-level zod override points every consumer at the
root version and the lock carries a single zod 4.6.6
* test(ui): stub the tag summary fetch the client panel retry refreshes
Retrying the model breakdown refetches /tag/summary, which this integration test
left on the real network. When that request failed before the assertion, the
panel switched to its summary error and unmounted the model row, so the test
passed or failed by timing
* fix(deps): drop the duplicated Lens override from the dashboard package.json
* style(ui): format the client panel fetch stub
Backport notes for stable 1.104.x. The dependency files (pyproject.toml, uv.lock, the dashboard package.json and package-lock.json) were left out of this pick and are re-derived on the line in the following commits
ClientUsagePanel.integration.test.tsx does not exist on this line, so its stub hunk is dropped
test_otel_v2_mount.py keeps the line's version because the traceresponse tests the squash edits are not on this line
test_litellm_logging.py keeps the line's version because the Langtrace export test the squash edits is not on this line
test_opentelemetry.py keeps the line's trace import without SpanLimits, which only the newer tests on main use
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
(cherry picked from commit 8f9894fbd6)
Keeps stable/1.104.x's 1.104.6 version in pyproject.toml and uv.lock, so this branch no longer bumps the version. In litellm_logging.py both async success helpers stay, the base's enqueued flag and the claim that #44508 adds, matching main. The two test files keep both sides' new tests
* fix(logging): report pay-as-you-go rates for spilled over PTU requests without base_model
* fix(logging): keep the provider response model so spilled over PTU payloads price the served model
* fix(logging): keep a provider_response_model the response already carries
* fix(logging): read the carried provider_response_model through a typed adapter
---------
Backport of #47655 to stable/1.104.x
Cherry-picked from 6b1f4ed46f (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill every earlier timed-out retry attempt at its own deployment's model rate
* fix(router): read the retry record's base_model through the typed metadata helper
* fix(router): keep the four-record retry breadcrumb cap
* fix(router): carry a deployment's litellm_params base_model on its retry record
* chore(router): place the cast-ok reason on the cast line and list the base_model helper as covered through log_retry
---------
Backport of #47648 to stable/1.104.x
Cherry-picked from 27efc387f7 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): price cost-based routing through model_info.base_model
* fix(router): keep cost-based routing pricing helpers fully typed
* fix(router): drop typing.cast from cost-based routing pricing helpers
* fix(router): read the cost map through a typed accessor in cost-based routing
* fix(router): avoid a Final rebind inside the cost-based routing loop
* refactor(router): share deployment_pricing_models across spend, listing and cost-based routing
* refactor(router): keep the /v1/models listing key inline, share pricing models for spend and routing
* refactor(router): inline the deployment pricing-model wrapper in cost-based routing
* refactor(router): read deployment pricing models through one shared helper
deployment_pricing_models(deployment) now owns the model_info.base_model / litellm_params.base_model /
litellm_params.model unpacking, used by cost-based routing and the /v1/models listing. pricing_models(base_model, model)
stays for the failure-row spend estimate, which already holds the two names
* refactor(router): share the str-keyed mapping validator with cost-based routing
* ci: re-run after flaky test_aiohttp_transport read timeout in llms-providers
---------
Backport of #47617 to stable/1.104.x
Cherry-picked from 0c6731eb5c (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): scope the search free-tier counter to the request's provider
* fix(proxy): read the provider with get so partial spend payloads still get the free tier
* test(proxy): give the grounded spend payload fixture its provider
---------
Backport of #47646 to stable/1.104.x
Cherry-picked from 640f2d865b (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): strip caller base_model from spend pricing and reject lone surrogates in multipart metadata
* fix(proxy): type the parsed multipart metadata so basedpyright does not grow reportArgumentType
---------
Backport of #47639 to stable/1.104.x
Cherry-picked from 2a268951c4 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bill partial Responses API streams when the client disconnects
Streamed /v1/responses requests and the default-route OpenAI and Azure /v1/messages requests behind them logged a 499 with zero spend when the client went away mid-stream, while /v1/chat/completions already billed the chunks it had sent. The Responses API iterator now keeps the response.created payload and can hand the disconnect cleanup a response.incomplete event with counted usage and the served service_tier, and the messages-route byte stream and router wrappers delegate to it.
* fix(proxy): satisfy lint gates for partial Responses API disconnect billing
Make the disconnect helper public so the proxy, router and messages adapter can import it without reportPrivateUsage, type the SSE byte stream as an AsyncGenerator so aclose is known, read litellm_call_id off the logging object, and keep the new tests on Final locals with the recorder injected through dynamic_async_success_callbacks instead of litellm.callbacks.
* fix(router): bill the primary stream's tokens after a mid-stream hop and chat-bridge Responses streams on disconnect
The router's fallback wrapper now folds the primary Responses stream's relayed tokens into the fallback's disconnect event the way _combine_responses_fallback_usage does for a natural end of stream, and LiteLLMCompletionStreamingIterator (chat-only providers served through /v1/responses) builds its disconnect event from the chat completion chunks it collected. stamp_responses_usage_cost is now public so the bridge module can price that event.
* Potential fix for pull request finding 'CodeQL / Log Injection'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
---------
Backport of #47611 to stable/1.104.x
Cherry-picked from f0ae445cc4 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
* feat(proxy): opt-in free-tier allowance for Google Search grounding spend
* fix(proxy): declare the search free tier counter's service target
* test(cost_map): allow search_free_requests in the cost map schema
* fix(proxy): skip cache hits and never free more than the billed cost in the search free tier
---------
Backport of #47494 to stable/1.104.x
Cherry-picked from e63aec029a (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47564 to stable/1.104.x
Cherry-picked from ce63f48ed2 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PR #47394 withheld cache_control_injection_points from the OpenAI SDK extra_body for
every provider on that path, which also covers litellm_proxy. The downstream LiteLLM
proxy applies the point itself, so litellm_proxy keeps forwarding it while OpenAI and the
other OpenAI-compatible providers still never receive it
Backport of #47570 to stable/1.104.x
Cherry-picked from 04e9ed4c52 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
get_standard_logging_object_payload decided custom_pricing from
litellm_params alone, so on an Azure PTU spillover response
model_map_information kept the deployment's zeroed PTU entry while
response_cost already used the standard model rate. Reuse the billing
side LitellmLogging._custom_pricing_for so both read the same decision
Backport of #47567 to stable/1.104.x
Cherry-picked from 4222523261 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): apply the Vertex regional endpoint uplift to batch cost
batch_cost_calculator takes vertex_location and multiplies by the model's
regional_endpoint_uplift_multiplier through get_vertex_regional_endpoint_uplift,
and every batch cost path (aretrieve_batch pricing, completed-batch logging,
native Vertex batch rows, the proxy batch cost poll and its per-line items)
resolves the deployment's location and passes it down
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
* test(logging): point the resolver tests at resolve_vertex_location_for_cost
---------
Backport of #47466 to stable/1.104.x
Cherry-picked from c6c303a618 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrun25 <165297479+Mrun25@users.noreply.github.com>
Backport of #47625 to stable/1.104.x.
Cherry-picked from 6564a490f4 (main).
Ported by hand onto the line's spend and budget code: the charge-once helpers move from model_max_budget_limiter into router_utils.batch_utils as on main; RouterBudgetLimiting keeps the line's websocket skip and uses the line's duration_in_seconds window for the batch marker TTL; the write_started mark covers budget reservations only, since the search free-tier allowance (#47494) is not on this line. Tests adapted to the line's _ProxyDBLogger name; the search-allowance test is dropped for the same reason.
Backport of #38172 to stable/1.104.x.
Cherry-picked from 8327cd6d47 (main).
Prerequisite for #47625: RouterBudgetLimiting reads the provider from the logging payload, so the batch poller's aretrieve_batch events reach router budgets at all. Test file placed at tests/unit/router_strategy/ on this line.
Backport of #47419 to stable/1.104.x.
Cherry-picked from 3124797399 (main).
Ported onto the line's register_model and router startup path without #41523's aliased-entry rewrite: registered fields come from the side table added in the previous commit. Tests adapted to the line: private names (_replay_model_cost_registrations, _response_cost_calculator), the custom-pricing test helpers from main's test_cost_calculator.py, CONVERSE_ANSWER inlined, and the alias-propagation and bedrock gpt-6.1-sol/ARN cases removed since those depend on #41523 and #46128, which are not on this line.
Backport of #47190 to stable/1.104.x.
Cherry-picked from 0ede0e10e7 (main).
Ported by hand onto the line's poller (the ManagedObjectRow protocol is the line's local _ManagedObjectRow; _finalize_unbilled_terminal_job keeps the line's None return). Test conflicts take this PR's assertions; the line-item poller tests from main are not on this line.
Vertex text embedding predict responses report metadata.billableCharacterCount, the
unit Google bills text-embedding-005 and text-multilingual-embedding-002 on. Carry it
into usage.prompt_tokens_details.character_count and, when the model carries
input_cost_per_character, bill that count instead of the token count so the prompt is
charged under one unit. Per-token embeddings (gemini-embedding-001) are unchanged.
Backport of #47460 to stable/1.104.x
Cherry-picked from d29200fa05 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47456 to stable/1.104.x
Cherry-picked from 3fa1587f43 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): apply tool_config cache-control points on chat and /v1/messages
A cache_control_injection_points entry with location tool_config was spread into
the Anthropic request body on /v1/chat/completions, which Anthropic rejected with
400 Extra inputs are not permitted, and the native /v1/messages path dropped the
point silently. Both paths now put the point's control on the last tool Anthropic
caches and never forward the key.
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* refactor(anthropic): validate tool_config inputs instead of casting them
Keeps the lint gates flat: the helper takes untyped tools and points and
narrows them with the existing object validators, the chat transform builds
the body with the marked tools instead of writing into optional_params, and
the handler reads a typed view of kwargs.
* fix(openai): keep tool_config cache points out of extra_body, add integration test
The cache hook hands a tool_config point on for the provider step, and for
providers that call through the OpenAI SDK every non-OpenAI param rides in
extra_body, so OpenAI answered 400 Unrecognized request argument supplied.
The new integration file drives a live proxy against scripted Anthropic,
Bedrock Converse and OpenAI upstreams for /v1/chat/completions and
/v1/messages.
* test(integration): cover tool_config cache points on bridged /v1/messages
Three new cases on /v1/messages for models without a native Anthropic
messages path: OpenAI through the Responses bridge and Gemini through the
chat bridge never see cache_control_injection_points in the body, and
Bedrock Nova through the chat bridge gets the cachePoint on toolConfig
---------
Backport of #47394 to stable/1.104.x
Cherry-picked from 93b79d2587 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Tan Nguyen <tan@pressingly.net>
* fix(cost): bill Gemini audio/image/video prompt tokens at the resolved input rate when no modality rate exists
Co-authored-by: shivam <shivam@berri.ai>
* style(cost): ruff format the modality cost calls
---------
Backport of #47373 to stable/1.104.x
Cherry-picked from 09e9193514 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
* feat(router): let a deployment set a timeout per service_tier
Adds timeout_by_service_tier and stream_timeout_by_service_tier to a deployment's
litellm_params. A request whose service_tier matches a key uses that value in place
of the deployment's timeout or stream_timeout. A timeout the client sends still wins
and requests without a matching tier are unchanged
Co-authored-by: adc77 <amandogra2016@gmail.com>
* style(router): sort the request_timeout_resolver import
* fix(router): read the service tier maps without the untyped params get
* fix(router): look the service tier timeout up through a typed get
---------
Backport of #47406 to stable/1.104.x
Cherry-picked from 62ce515d15 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: adc77 <amandogra2016@gmail.com>
* fix(decisions): log every request field the caller sent on /v1/systemone and /v1/decisions
* fix(decisions): serialize the logged request with model_dump_json so no mutable dict is built
* refactor(decisions): drop the state-only messages pick in function_setup, prepare_call logs the request
* fix(decisions): return the single logged message so no mutable list type is annotated
* fix(decisions): log the caller's raw request fields before validation so a rejected request is still recorded
* fix(decisions): serialize the raw request fields with pydantic_core so question objects and mapping proxies log as JSON
* fix(decisions): log the caller request at the SDK entry so requests rejected before prepare_call are recorded too
* test(decisions): assert the exact BadRequestError for the rejected-questions case
---------
Backport of #47183 to stable/1.104.x
Cherry-picked from 53821f6654 (main)
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #47353 to stable/1.104.x.
Cherry-picked from 2d181e6633 (main).
Adapted to the stable compatibility rule: the bundled DeepSeek tokenizer is selected only on the proxy count routes, through select_count_route_tokenizer, so token_counter, encode and decode, budget reservation, ITPM, context-management triggers and failed-request spend rows keep the line's tokenizer and a request admitted on v1.104.4 is still admitted. A DeepSeek tokenizer that fails to load falls back to tiktoken on the count routes, the way every other bundled tokenizer does. The wire tests read the line's proxy_config.yaml, which has no general_settings or router_settings, and use azure_ai/grok-4 as the control model, since the line's azure_ai/grok-4-1 entry rejects tool_choice. The failed-dispatch spend test is left out, and the wire tests that asserted the DeepSeek tokenizer on those paths now assert the line's count is kept.
Backport of #47182 to stable/1.104.x.
Cherry-picked from 1212260ab1 (main).
Adapted to the line's private helper names (_select_model_name_for_cost_calc, _response_cost_calculator); the new test is appended to the line's test file.
Backport of #43935 to stable/1.104.x.
Cherry-picked from d7620c77a1 (main).
Adapted to the line: the bridge check lives in litellm/main.py rather than litellm/responses/bridge_check.py, the effort helpers keep the line's private names, and the azure gpt-6.1-sol rows get supports_none_reasoning_effort=false (from #46377 on main) since this fix and its tests key on it. Test files keep the line's content plus this PR's own hunks.
Backport of #47184 to stable/1.104.x.
Cherry-picked from 3fb95c8fe3 (main).
Ported by hand onto the line's poller, whose surrounding code predates several main-only batch features: the claim now writes the terminal status and NUL-stripped file_object, a failed spend-log write restores both, and the managed-files postgres_storable_json sanitizer (from #45703 on main) is added to the line since this fix needs it.
Backport of #45151 to stable/1.104.x.
Cherry-picked from 46d440ae96 (main).
Conflicts resolved by keeping the line's fields and adding only the over-100k fields this PR introduces; cost-map entries replaced per model.
Backport of #45108 to stable/1.104.x.
Cherry-picked from 5cd1126870 (main).
Prerequisite for #45151: adds the claude-haiku-5-5 entries that fix corrects. Cost-map conflicts resolved by appending the new entries.
Backport of #43955 to stable/1.104.x.
Cherry-picked from eb5295b20c (main).
Adapted to the line: the transformation lives under anthropic/experimental_pass_through, the integration tests live under tests/integration/providers, and the CLI-beta wire assertion change is dropped since that test is not on the line.
Backport of #45686 to stable/1.104.x.
Cherry-picked from 1aee1a21ed (main).
Test-file import conflict resolved by keeping the line's imports plus this PR's own.
Backport of #45659 to stable/1.104.x.
Cherry-picked from fa7ad80a62 (main).
Test-file conflict resolved by keeping the line's file plus this PR's own additions.
Backport of #47173 to stable/1.104.x.
Cherry-picked from c82e8244a5 (main), applied through the line's earlier resolution c3d07e70d6.
Adapted to the line: the default tool count keeps the line's formatter, since #43417 is not on the line, and only the chat-template path gets the function-definition helpers.
Adapted to the stable compatibility rule: the chat-template tool count is opt-in through token_counter(chat_template_tools=True), which only the proxy count routes pass, so ITPM, budget reservation and context-management estimates keep the line's OpenAI tool framing and a request they admitted on v1.104.4 is still admitted. The tests that asserted the new framing on those paths now assert the line's framing is kept.
Backport of #46430 to stable/1.104.x.
Cherry-picked from 08a24d391a (main), Bedrock changes only.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #45317 to stable/1.104.x.
Cherry-picked from 577d74c1ce (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
