An OpenAPI-backed MCP tool whose upstream answered 401 came back as a
successful tool result carrying the upstream's rejection as its content, so a
caller saw {"error":"invalid_token"} presented as data and the gateway recorded
the request in its own spend log as call_mcp_tool | success.
Three layers each erased the outcome. The request function returned
response.text whatever the status, _handle_local_mcp_tool caught every exception
and returned it as ordinary TextContent, and both dispatch sites then stamped
isError=False unconditionally. Fixing only the first, which is the obvious fix,
changes nothing, because the two above it still map failure onto the
success-shaped value.
The status is now classified where the response is held: a 401 becomes
MCPUpstreamAuthError so the caller is told to re-authenticate, and every other
non-2xx becomes MCPOpenApiUpstreamError, which carries the status and drops the
upstream body rather than serving it as tool content. _handle_local_mcp_tool no
longer swallows, and the call_tool arm keeps the auth error's type. Nothing new
renders these: call_mcp_tool and call_tool_rest_api already turn them into an
isError result naming the status and into a real 401 with WWW-Authenticate, and
the OpenAPI path simply never reached them.
The result is now byte-identical to the regular MCP path for the same failure.
* feat(auto-router): scope shadow eval jobs to multiple keys
A shadow eval job now covers a set of keys instead of exactly one, and each
key carries its own max_turns budget, so one key exhausting its budget leaves
its siblings sampling. The existing job row already is the per-key unit
(api_key_id, max_turns, stopped_at, and the one-active-per-key-and-direction
partial unique index all live on it), so multi-key is grouping rather than
schema surgery: a new group_id column ties N sibling rows written atomically
by one create_many, the API's job id becomes the group id, and pre-existing
jobs backfill group_id = id so their ids keep resolving. The sampler hot path
is untouched; its test file has a zero-line diff
Results come back pooled plus a per-key breakdown and responses list every key
with its own budget, stop state and read-time labels. The dashboard is adapted
minimally to the new shapes (the picker stays single-key and submits a one-key
list); the multi-select picker and per-key table land in the stacked UI PR
* fix(shadow_eval): derive completed from spent budgets and record operator stops
* fix(shadow_eval): stamp stops atomically and freeze counts at the stamp
The stop endpoint wrote stopped_by and stopped_at as two separate updates, so
a failure between them left a job reading stopped while its unstamped legs
kept sampling, and the retry got 400 already stopped. One UPDATE now stamps
stopped_by and every missing stopped_at together, preserving the stopped_at a
leg earned from its own budget via COALESCE
Attempt counts now exclude attempts that land after a leg's stopped_at, so an
in-flight attempt finishing just after an operator stop can never push a
legacy pre-stopped_by job over its budget and flip it from stopped to
completed at read time
* fix(shadow_eval): backfill stopped_by so legacy stops never read as completions
* chore(ui): regenerate api types for the shadow eval stop fields
* fix(shadow_eval): let the stop statement pick one winner under racing stops
Two operators can both pass the derived-status guard in the race window. The
stop UPDATE now claims only legs with stopped_by still null and the endpoint
judges by its row count, so exactly one caller ever gets the 200 and the loser
gets the same already-stopped 400 a late caller gets
* refactor(shadow_eval): make the stop statement the whole state machine
The status guard ran before the UPDATE, so a stop racing the last budgeted
attempt still claimed the job and it read stopped forever instead of
completed. The statement now claims the job only while a leg still samples
inside the window with no stop recorded, and the endpoint reads once after
writing: a racing operator, a same-instant budget spend, and a repeat stop all
get the 400 naming the status the job actually holds. The pre-write guard and
the hand-built response go away
* chore(ui): regenerate api types for the stop route description
The flat-cost rollup reads deployments only from LiteLLM_ProxyModelTable, so a PTU
deployment declared in config.yaml never accrues flat cost. Those deployments live in
llm_router.model_list as plain dicts whose id sits in model_info rather than on the entry,
so they do not satisfy the shape _parse_ptu_model reads.
Adds a frozen record in that shape and a factory that maps a router entry onto it, leaving
_parse_ptu_model byte-identical so the existing cases stand as evidence of no behaviour
change. Nothing calls the factory yet; the caller lands with the loader union.
_decode_model_info also stops handing back valid JSON that is not an object. It decoded
a list or a scalar and returned it as a mapping, so the caller read fields off it and
raised, losing the whole run rather than the one bad deployment.
Both OpenAPI dispatch arms sourced the upstream credential only from the
deprecated global / BYOK mcp_auth_header and never from mcp_server_auth_headers,
so x-mcp-{alias}-authorization was silently dropped on spec_path servers and the
upstream API received no Authorization at all. The managed path already resolves
it through lookup_mcp_server_auth_in_headers, so the two had drifted.
_resolve_openapi_tool_auth now owns that resolution for both arms. A per-server
value is already a complete header value and is forwarded verbatim, while a BYOK
credential keeps its auth-type prefix, so the two are never conflated into
"Bearer Bearer <token>". The resolved credential is also handed to
resolve_openapi_upstream_auth, whose passthrough arm reads it through
_passthrough_token_from_mcp_auth_header and outranks the ContextVar.
server.py loses its inlined copy of the forwarded-header logic along with its
mcp_server is None guards, which are unreachable after the 503 raised above them.
Credit to the earlier analysis and approach in #33349, which this supersedes
against the current v2 credential resolver.
The model resync now mutates the router under MODEL_RECONCILE_LOCK, and the
agent resync shares the new AGENT_RECONCILE_LOCK with the periodic agent
reload, so a reconcile built from a pre-write DB snapshot can no longer evict
or duplicate what a read-through just registered. Every resync checks
should_load_db_object for its object type, keeping read-through consistent
with what the replica is configured to load, and the a2a raise sites tag
ProxyModelNotFoundError as non-retryable so an agent miss no longer burns the
model resync budget.
Resync registry misses with single-row DB fetches (guardrail by unique
name, agent by unique id or name, model by name then id) instead of
full-table loads, and bound them with a global budget of 20 resyncs per
5s window per registry that fails closed without negative-caching the
key.
Access group create/update now trust the reconcile outcome snapshot
captured under the reload lock instead of a post-lock router read, so a
concurrent reconcile can no longer surface a false degraded-serving 500.
Router.upsert_deployment restores the previously served deployment when
the replacement add fails under ignore_invalid_deployments, so a bad
update no longer silently drops a healthy deployment from serving.
Streamed `/v1/messages` `usage.cost` disagreed with the cost the logging callback recorded in three ways: `input_tokens` was read as the whole prompt total, but Anthropic reports it excluding cache tokens, so the non-cached input went unbilled on cache hits; the `cache_creation` 5m/1h split was dropped, billing 1h writes at the 5m rate; and costing by model name alone ignored the deployment's custom pricing, so a negotiated discount still streamed sticker price.
Anthropic usage now goes through `AnthropicConfig.calculate_usage`, the same transformation the non-streaming path uses, and the chunk is priced through the call's logging object when there is one so it inherits `custom_pricing`, `custom_llm_provider`, `base_model` and `router_model_id`, falling back to `completion_cost` by model name.
`calculate_usage` only reads its `usage_object`, so it now takes a `Mapping`.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The aggregated daily-activity queries build their WHERE clause as raw SQL, and
an empty entity list rendered as `"team_id" IN ()`, which Postgres rejects with
`syntax error at or near ")"`. Callers reach that state normally: a caller
without admin view and no explicit team_ids has its scope resolved to the teams
it belongs to, so anyone belonging to no teams, an org admin for instance, sent
an empty list and got a 500 back from /team/daily/activity/aggregated.
The paginated endpoint hands the same empty list to Prisma, which renders
`in: []` and matches nothing, so it kept returning 200 with an empty result set.
Emit FALSE for the empty case so the raw-SQL path lands on the same answer,
mirroring what the api_key filter a few lines below already does.
The fix covers both aggregated queries at once because they share one WHERE
clause builder.
true_passthrough and oauth_delegate forward the caller's own bearer and mint
nothing, so their sessions consume no discovered OAuth endpoint. The discovery
completeness gate still failed them closed with a 503 raised before the upstream
was ever contacted, which the tools/list fan-out swallowed into HTTP 200 with an
empty tool list. Any upstream that publishes no RFC 9728 metadata, including
every OpenAPI-backed internal API, was permanently unusable.
A failed discovery is now fatal only to servers whose OAuth flow the gateway
runs itself. Discovery still runs for the forwarding modes, so /authorize,
/token and /register keep resolving their endpoints, and each keeps its own 400
when they are missing.
MCPServer.is_client_forwarded_token now owns the mode pair that five call sites
spelled inline, one of which had already named it is_client_forwarded_mode.
* feat(bedrock): add a config toggle to disable agent-runtime pass-through
The /bedrock pass-through dispatches agents, knowledge bases, flows, rerank,
retrieveAndGenerate, generateQuery and optimize-prompt to bedrock-agent-runtime,
so an operator who only wants to expose model invoke and converse has no way to
narrow that surface
Adds general_settings.disable_bedrock_agent_runtime_passthrough. When set, those
routes are rejected with a 403 before credentials are fetched or the request is
signed. Plain bedrock-runtime model pass-through is unaffected, and the setting
defaults to off, so existing deployments behave exactly as before
The branch is inverted to an early return for the non-agent-runtime case so the
toggle can reject outright instead of falling through to model extraction, which
would surface a confusing 400 about an unparseable model
* style(bedrock): drop redundant docstrings from the agent-runtime toggle
- ruff format litellm/batches/batch_utils.py
- reconcile the upstream output_file_id=None guard (merged in from
litellm_internal_staging) with BatchCostUsageResult, and count that
batch's error_file_id failures instead of always reporting 0
- fix test_handle_completed_batch_no_output_file_is_zero's tuple
unpacking, which predated the BatchCostUsageResult refactor
- commit the batch_successful_requests/batch_failed_requests fixture
fix to test_spend_management_endpoints.py that was left uncommitted
Both surfaces wrote zero bytes for the whole time-to-first-token, so an
intermediary with an idle read timeout drops a healthy connection before the
first token. They reached neither keepalive engine, which is what #37322 left
open.
The streaming assistants run spends that wait inside the awaited call that
produces its response, since create_response buffers the first chunk, so it
takes the same open_sse_before_first_byte seam the native routes use. The A2A
route only contacts the upstream agent once its body iterator is first pulled,
so nothing is awaited before the response exists and the gap has to be filled
from inside the stream instead; wrap_sse_stream_with_keepalive_pings already
does that and now takes the filler as a parameter, so A2A gets an SSE comment
its JSON-RPC clients discard rather than Anthropic's ping event.
Off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds.
