* fix(router): carry a 429 status on cooldown and no-deployment errors
RouterRateLimitError and RouterRateLimitErrorBasic had no status_code, so the
proxy answered an all-deployments-cooled-down request as a generic 500 class
error and logged the bug report notice for every cooldown. Both now carry
status_code 429, which the proxy maps to a rate_limit_error and which the
retry policy already treats as retryable
* fix(proxy): start Sentry at config load and never attach request bodies
Sentry only came up inside set_callbacks on the first LLM call, so boot and
background errors were never reported, and general_settings.alerting: [sentry]
always raised Missing SENTRY_DSN because the alerting path read a dead
litellm.utils pin instead of the live SDK. The proxy now initializes Sentry
once while loading the config whenever sentry is a configured callback or
alerting target, the alerting and failure handlers read the live instance, and
max_request_body_size is pinned to never so prompts and completions stay out
of Sentry events
* fix(tracing): log Lens failures at ERROR and export row counters to Prometheus
A tracing store that fails to start, or a Lens service that rejects a batch,
was only logged at WARNING, so operators running at the default ERROR level
never saw tracing silently go dark. Both now log at ERROR and name the lens
store. The exporter's rows_written and rows_dropped tallies were never
exported, so the two counts are now Prometheus counters
litellm_lens_export_rows_written_total and
litellm_lens_export_rows_dropped_total, registered lazily on the default
registry like the other proxy-level counters and charted on the all-metrics
Grafana dashboard
* test(proxy): drop the litellm import the alerting tests no longer use
* fix(proxy): start Sentry once a DSN is set, from any config source
Boot-time init used to lock Sentry in even with no SENTRY_DSN, so a DSN loaded later from DB environment_variables was never picked up. initialize_sentry now skips when no DSN is set and re-inits when the DSN changes. alerting_handler calls it too, so alerting: [sentry] set through the DB works. Drops the litellm_settings.callbacks check since that config fails to load with sentry in it
* fix(sentry): keep stack frame locals out of events unless PII is opted in
Frame locals carried prompts (messages, request bodies, provider options) into every error event. include_local_variables now follows SENTRY_SEND_DEFAULT_PII
* test(router): drop the test that only pinned the 429 class attribute
* fix(proxy): start Sentry at boot when it is only an input callback
* fix(proxy): keep Sentry init and alerting within the strict ruff and basedpyright ceilings
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
The proxy answers POST /v1/mcp/server with 201 and DELETE /v1/mcp/server/{id} with 202, but handleMCPAPIResponse, handleAPIResponse, Client.sendRequest and resourceLiteLLMMCPServerDelete treated every status other than 200 as a failure. terraform apply created the server on the proxy, then reported an error and never wrote the resource to state, so the next apply created a duplicate, and destroy failed on the 202.
Accept any 2xx in the four status checks, treat an empty or null 2xx body in handleAPIResponse as an empty ModelResponse, and cover each path with httptest tables that fail on the 200-only comparison.
Re-lands the diff from #40723 on current main.
Co-authored-by: matthew-hull-bright <matthew.hull@brightsg.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(cost map): classify token_count_framing and token_count_fallback_model in the schema generator
* ci(cost map): bound token_count_framing to non-negative and token_count_fallback_model to non-empty
* test(cost map): reject a fractional token_count_framing in the generated schema
* test(cost map): pin that a token_count_framing of 0 passes the generated schema
* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): narrow the provider response id with isinstance instead of cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): validate callback fields for current type gates
* test(gcs_pubsub): assert provider response id in spend metadata
* fix(ollama): normalize reused response ids automatically
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf streams take a reasoning or tool-call delta as the first token, tool-call replies had no time_to_first_token
* test(e2e): perf users start at a random offset and jitter each interval by up to 25%, no more lockstep bursts at the planned rate
* test(e2e): small_bodies_5rps sends from the mixed case's callers (two teams, JWT users, keys in and out of teams) through the pre and post guardrail
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking
* fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds
Bedrock Invoke copied every caller thinking key when adding display updates,
so an unknown key still reached the provider with drop_params on. Filter that
copy the same way, and cover each chat call site plus the global
litellm.drop_params path in tests
* fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params
* test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions
* fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel
* feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI
Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing
early-access row, carrying the same limits and capabilities. The AI Studio row
mirrors the early-access entry; the Vertex row differs only in provider and
pricing source. Prices follow once Google publishes them.
Extend the mapped metadata test to cover all three ids, that each routes to its
provider, that its backup copy matches, and that the public rows carry the same
limits as the early-access row.
* feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops
* chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap
* fix(gemini): follow continuation hops in a loop instead of recursion
The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls
* fix(gemini): bill Google Search grounding from every continuation request
Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain
* fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI
A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream
On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued
On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports
continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch
* chore(gemini): name the type errors the dual-mode stream iterator trips
The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors
This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up
* fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens
A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request
A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate
Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk
The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment
* fix(gemini): keep one response id across a continued answer
A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream
A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key
* test(gemini): add integration cells for long decode continuation
Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs
Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once
* fix(vertex_ai): fetch the access token again before each continuation request
A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length
Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged
* fix(vertex_ai): start each continuation stream with an empty parser buffer
When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error
Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async
* fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up
A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token
Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way
* fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer
A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer
The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes
* fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata
A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex
A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not
* fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer
A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0
A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer
Adds the audit's wire tests for both, which fail 24 cells before this change
* test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index
Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed
* fix(vertex_ai): log a continued answer's raw response as one object
A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them
It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted
Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch
* test(integration): check a continued answer logs one merged raw response
A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch
* test(vertex_ai): cover a follow-up tool call and its thought signature in long decode
* test(gemini): pin that a one-request answer logs Google's body unchanged
* test(integration): assert the streamed usage.cost matches the spend row on every cost case
* test(integration): type the streamed usage frames as JsonValue
* test(integration): only require streamed usage.cost when the client received a usage frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses
* fix(ui): send detail on playground Responses image parts and code snippets
* test(messages): cover image detail on the Responses bridge wire
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it
Fixes#31449.
Two OCI GENERIC request-building defects that made Grok-on-OCI unusable
for tool-using coding-agent clients (Codex over the Responses API, Claude
Code over the Anthropic Messages API):
1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only
supports function tools")` for any tool whose type is not "function",
aborting the whole request when a client mixed a built-in/server tool
in with its function tools. Non-function tools are now skipped with a
warning; if none remain, `tools`/`toolChoice` are omitted from the
request instead of sending an empty array.
2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere
model, but OCI rejects it for all served xAI Grok models
(`Model grok-4.20 does not support parameter reasoningEffort`) and for
OpenAI's non-reasoning models (`Unrecognized request argument supplied:
reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-*
and Meta/Google models accept it. `_model_supports_reasoning_effort`
encodes that, `get_supported_openai_params` stops advertising the param
for those models, and `map_openai_params` follows the standard
unsupported-param contract (dropped under drop_params, clear error
otherwise) so `drop_params` / `additional_drop_params` now work as
documented instead of the 400 surfacing from the service.
Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model.
The two existing reasoning_effort normalisation tests move from a Grok
model to openai.gpt-5, which is the case they exercise. 21 new tests; the
OCI suite is 406 passed.
* fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools
Review follow-ups on #40732:
- `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag
of the model's `oci/` catalog entry through `get_model_info` instead of
hardcoding xAI / gpt-oss branches. Catalogued models without the flag
(xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent
from the catalog keep the historical pass-through.
- Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning
models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b`
(pricing from Oracle's public price list, 128K context per OCI docs).
- `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only.
OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output
cap silently disappears) and Llama rejects it with 400, so the catalog's
reasoning flag must not drive that switch.
- After tool adaptation, a `FUNCTION` tool_choice naming a tool that was
skipped (built-in / server tools) is dropped with a warning instead of
being sent to OCI, which rejects the unmatched choice.
Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a
forced built-in now completes and still calls the function tool; Gemini and
gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under
drop_params; gpt-5 keeps maxCompletionTokens.
* style(oci): use an immutable set for the tool-name lookup
Keeps the type-discipline gate (LIT002, mutable collections) within its
budget; the lookup is read-only.
* fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive
Review follow-ups on #40732:
- When tool_choice forces a function that is not among the adapted tools
(a skipped built-in or a misspelled name), send `toolChoice: NONE` instead
of removing the choice. Falling back to the service default would let the
model call any other supplied function even though the caller forced a
specific one, which an agent processing untrusted content must not do.
- Move the #31449 regression tests into the mapped test module
(`test_oci_chat_transformation.py`) instead of a separate file.
Verified live against OCI GenAI: Grok and Llama with a forced skipped
built-in now complete with no tool call; a forced kept function is still
honoured.
* test(oci): point reasoning_effort comments at the in-module gating tests
The referenced module was folded into this file.
* fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels
Non-function tools now follow the unsupported-param contract: a 400 naming
drop_params without it, and dropped with a warning under it, with a forced
dropped tool turning tool calls off. reasoning_effort is gated on the model's
reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the
Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop
sending a param the service rejects
* fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it
* fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name
* fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name
* fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap
* fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR
Under drop_params, non-function tools still drop when function tools remain and
tool_choice does not force a dropped tool. A request whose every tool is dropped,
or whose tool_choice forces one, now answers 400 instead of a tool-less reply the
model can fill with invented results, or a swap to another tool.
The maxTokens routing change for OCI Gemini goes back to base, since it is
outside this fix and an open PR carries it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
GET /v1/mcp/server takes an optional user_id (admin, or the user themselves) and returns the union of the servers each of that user's teams reaches plus allow_all_keys servers. A user in no team gets the unscoped list, a deleted team still listed on the user is skipped, an unknown user is a 404 and a restricted virtual key is refused like for team_id. team_id keeps precedence when both are passed.
The dashboard MCPServerSelector passes the team and user it edits (key edit sends keyData.team_id and user_id, key create sends the picked team and owner, user edit sends the user) and narrows the server options to members of the access groups selected in the same control, keeping allow_all_keys servers and servers already granted so they can still be removed.
Resolves LIT-7784
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fastapi_sso opens a bare httpx.AsyncClient for the non-PKCE token exchange and
userinfo call, so litellm_settings.ssl_verify and SSL_VERIFY never applied and
an IdP on an internal CA failed with an opaque 503. A LiteLLMHttpClientSSO mixin
replays that exchange on litellm's shared SSO client for the generic, Google and
Microsoft providers and surfaces IdP connection and TLS failures as a 401
ProxyException naming the endpoint.
Resolves LIT-9445
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): name the successor model for deprecating deployments
* chore(ui): add successor_model to the generated deprecation api types
* fix(proxy): keep the cost map successor when a deployment overrides only the date
* fix(proxy): read the cost map successor from metadata.successor
The cost map schema rejects a top-level successor_model key, and curated
entries already carry the successor under metadata.successor. Read that key,
and look it up even when the cost map entry has no deprecation date, so a
deployment that sets only its own date still names the successor.
* fix(proxy): keep the first deployment when deduping, let a blank successor hide the cost map one, and name a served successor by its model group
* fix(proxy): name only the deployment's own successor_model, since cost map successors are LiteLLM ids rather than served model names
* fix(proxy): leave out a successor_model the deployment's callers cannot call, with a warning
* fix(proxy): report successor_model as the admin set it and only warn when no model matches it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs
* fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off
* fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off
* test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it
With modify_params on, a trailing tool search turn whose call has a kept result
got a placeholder answer from the factory and went out as a marked tool_use,
but the cache control hook counted marks before that placeholder existed and
skipped the mark, so a request at the cap reached Anthropic with 5 marks and
failed with a 400. The hook now finds client answers on the sanitized messages,
the same list the factory reads.
Also make the replay checks cheaper (prefix check before validation, early
return without provider_specific_fields, and look up client answers only when
a tool call carries cache_control), drop the two audit cells that cover old
gaps outside the tool search fix, and derive the pass-through test timeout
from the graceful stop budget.
* fix(anthropic): keep the duplicate tool result warning out of the cache control hook
The hook finds client answers with only the step of the modify_params sanitize
pass that pairs tool calls with their results, so a request with a duplicate
tool result logs the warning once again instead of once per mark count.
Also restore the Snowflake non-stream and cached-stream tool search cells as
known bugs with pytest.skip instead of deleting them, and put the Anthropic
pass-through test timeout back to 240 seconds.
* fix(anthropic): count cache marks next to a message the pairing step cannot read
With modify_params on and a marked tool call, the cache control hook paired tool
calls with their results before counting. A message without a get method (a
stray string or null in extra_body.messages, or an openai SDK
ChatCompletionMessage) made that pairing raise AttributeError before any
provider call, where main counted the marks and let the request through.
The hook now pairs only when every message can be read the way the pairing
step reads it, and otherwise counts on the messages as sent, like main did.
* fix(anthropic): count cache marks after placing a later system message the way the request builder does
* test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census
* test(anthropic): align the tool search replay cells with main and cover a system message after the search turn
The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882.
The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool
reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell
expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its
mark, and a new cell covers a system message after the trailing search turn in both stream modes
* fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request
* test(anthropic): cover marked tool calls next to a malformed tool id and a later system message
* perf(anthropic): look up client tool answers only when a marked tool call is a server tool call
Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked
tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count
marks
* test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell
* perf(anthropic): place mid-conversation system messages in linear time for models that keep them
* fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time
* test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn
* fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag
* test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router
* fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure
* test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement
* fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests
Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route.
* test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse
* fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow
* test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt
* refactor(test): avoid local list mutation and satisfy type discipline gate
* style(prompt_templates): keep line length under 120 chars for ruff format
* style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401
* style(prompt_templates): eliminate basedpyright errors in function_call_prompt
* fix(prompt_templates): keep tuple and None system content when adding the function prompt
Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged
* fix(prompt_templates): reject unreadable system content without echoing it
Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt
* fix(prompt_templates): leave non-list system content failing as before
None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated
* fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params
* test(prompt_templates): pin the function prompt on every system message
* test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): intercept the bare web_search tool type on chat completions
* fix(websearch_interception): match the bare web_search type only on chat completions
/v1/messages keeps its existing detector, so a non-Anthropic model there no
longer short-circuits a bare web_search tool into raw search results
* test(websearch_interception): cover the bare web_search type on chat and through the messages adapter
* refactor(typing): narrow Any sources in proxy, guardrails, bedrock and responses code
Turns untyped json.loads, response.json(), header lookups and forwarded
kwargs into object or precise types at their source, so downstream code
type-checks without reportAny / reportExplicitAny. No runtime behavior
changes.
* refactor(typing): narrow Any sources in caching, MCP, providers and proxy request code
Decoded JSON, cache reads and header lookups now enter as object or
precise types, prisma tables used by the shadow-eval logger get typed
Protocol views, and a few SDK parameters move from dict[str, Any] to
Mapping[str, object]. No runtime behavior changes.
* refactor(typing): narrow Any sources in proxy server, router, logging and shared utils
* refactor(typing): narrow Any sources in provider transformations, integrations and proxy endpoints
* refactor(typing): narrow Any sources across providers, proxy and router
Deepens the Any cleanup on the hottest files: typed helpers in
core_helpers for attribute and item access, identity TypeAdapter(object)
wraps at untyped boundaries, explicit return annotations on shared
provider and proxy helpers, and Protocols in place of Any parameters.
No intended behavior change.
* test(integration): wait on the right signal in five flaky integration contracts
The Prometheus shared-label tests waited for the latency metric and then read the
request counter, which the logger writes after an await. They now wait for the counter
The auto-router session poll returned on the first 200 while the caller expected two
turns, so it now waits for the expected turn count
The S3 queue cap test let its failing sink recover on a 15s clock that could expire
mid-flush. The sink now stays down until a flush fails after every event is queued
K3 normally takes 168-175s against a 180s limit, so the limit is now 300s
The MCP env var browser spec reused an idle keep-alive socket that the proxy closes
after 5s, so its API calls now send Connection: close
* test(integration): add until_asserted so a wait polls the exact asserts it guards
eventually() takes a predicate separate from the asserts that follow it, which lets a
test wait on one value and then assert on another written later. until_asserted reruns
a block of asserts until they all pass and re-raises the last failure on timeout
The Prometheus shared-label tests and the auto-router session check now use it
