An OpenAPI-backed MCP tool whose upstream answered 401 came back as a
successful tool result carrying the upstream's rejection as its content, so a
caller saw {"error":"invalid_token"} presented as data and the gateway recorded
the request in its own spend log as call_mcp_tool | success.
Three layers each erased the outcome. The request function returned
response.text whatever the status, _handle_local_mcp_tool caught every exception
and returned it as ordinary TextContent, and both dispatch sites then stamped
isError=False unconditionally. Fixing only the first, which is the obvious fix,
changes nothing, because the two above it still map failure onto the
success-shaped value.
The status is now classified where the response is held: a 401 becomes
MCPUpstreamAuthError so the caller is told to re-authenticate, and every other
non-2xx becomes MCPOpenApiUpstreamError, which carries the status and drops the
upstream body rather than serving it as tool content. _handle_local_mcp_tool no
longer swallows, and the call_tool arm keeps the auth error's type. Nothing new
renders these: call_mcp_tool and call_tool_rest_api already turn them into an
isError result naming the status and into a real 401 with WWW-Authenticate, and
the OpenAPI path simply never reached them.
The result is now byte-identical to the regular MCP path for the same failure.
Two or more reasoning keyword matches promoted a request straight to the
REASONING tier no matter what the weighted score said, so "hi, step by step,
pros and cons" scored 0.100 and still bought the most expensive tier.
Require the score to clear the simple_medium boundary before the override
applies. Promotion from MEDIUM or COMPLEX is unchanged; only prompts the
scorer already placed in the cheapest band stay there.
* feat(auto-router): scope shadow eval jobs to multiple keys
A shadow eval job now covers a set of keys instead of exactly one, and each
key carries its own max_turns budget, so one key exhausting its budget leaves
its siblings sampling. The existing job row already is the per-key unit
(api_key_id, max_turns, stopped_at, and the one-active-per-key-and-direction
partial unique index all live on it), so multi-key is grouping rather than
schema surgery: a new group_id column ties N sibling rows written atomically
by one create_many, the API's job id becomes the group id, and pre-existing
jobs backfill group_id = id so their ids keep resolving. The sampler hot path
is untouched; its test file has a zero-line diff
Results come back pooled plus a per-key breakdown and responses list every key
with its own budget, stop state and read-time labels. The dashboard is adapted
minimally to the new shapes (the picker stays single-key and submits a one-key
list); the multi-select picker and per-key table land in the stacked UI PR
* fix(shadow_eval): derive completed from spent budgets and record operator stops
* fix(shadow_eval): stamp stops atomically and freeze counts at the stamp
The stop endpoint wrote stopped_by and stopped_at as two separate updates, so
a failure between them left a job reading stopped while its unstamped legs
kept sampling, and the retry got 400 already stopped. One UPDATE now stamps
stopped_by and every missing stopped_at together, preserving the stopped_at a
leg earned from its own budget via COALESCE
Attempt counts now exclude attempts that land after a leg's stopped_at, so an
in-flight attempt finishing just after an operator stop can never push a
legacy pre-stopped_by job over its budget and flip it from stopped to
completed at read time
* fix(shadow_eval): backfill stopped_by so legacy stops never read as completions
* chore(ui): regenerate api types for the shadow eval stop fields
* fix(shadow_eval): let the stop statement pick one winner under racing stops
Two operators can both pass the derived-status guard in the race window. The
stop UPDATE now claims only legs with stopped_by still null and the endpoint
judges by its row count, so exactly one caller ever gets the 200 and the loser
gets the same already-stopped 400 a late caller gets
* refactor(shadow_eval): make the stop statement the whole state machine
The status guard ran before the UPDATE, so a stop racing the last budgeted
attempt still claimed the job and it read stopped forever instead of
completed. The statement now claims the job only while a leg still samples
inside the window with no stop recorded, and the endpoint reads once after
writing: a racing operator, a same-instant budget spend, and a repeat stop all
get the 400 naming the status the job actually holds. The pre-write guard and
the hand-built response go away
* chore(ui): regenerate api types for the stop route description
The flat-cost rollup reads deployments only from LiteLLM_ProxyModelTable, so a PTU
deployment declared in config.yaml never accrues flat cost. Those deployments live in
llm_router.model_list as plain dicts whose id sits in model_info rather than on the entry,
so they do not satisfy the shape _parse_ptu_model reads.
Adds a frozen record in that shape and a factory that maps a router entry onto it, leaving
_parse_ptu_model byte-identical so the existing cases stand as evidence of no behaviour
change. Nothing calls the factory yet; the caller lands with the loader union.
_decode_model_info also stops handing back valid JSON that is not an object. It decoded
a list or a scalar and returned it as a mapping, so the caller read fields off it and
raised, losing the whole run rather than the one bad deployment.
Both OpenAPI dispatch arms sourced the upstream credential only from the
deprecated global / BYOK mcp_auth_header and never from mcp_server_auth_headers,
so x-mcp-{alias}-authorization was silently dropped on spec_path servers and the
upstream API received no Authorization at all. The managed path already resolves
it through lookup_mcp_server_auth_in_headers, so the two had drifted.
_resolve_openapi_tool_auth now owns that resolution for both arms. A per-server
value is already a complete header value and is forwarded verbatim, while a BYOK
credential keeps its auth-type prefix, so the two are never conflated into
"Bearer Bearer <token>". The resolved credential is also handed to
resolve_openapi_upstream_auth, whose passthrough arm reads it through
_passthrough_token_from_mcp_auth_header and outranks the ContextVar.
server.py loses its inlined copy of the forwarded-header logic along with its
mcp_server is None guards, which are unreachable after the 503 raised above them.
Credit to the earlier analysis and approach in #33349, which this supersedes
against the current v2 credential resolver.
The proxy resolves its own public origin from PROXY_BASE_URL before it
looks at anything on the request, so a developer who has that set for
their own deployment watched 65 OAuth discovery, redirect_uri, and client
registration cases fail against an origin no test ever asked for
An autouse fixture now clears it for every unit test, matching the host
AWS config isolation that already sits beside it, and the tests that do
exercise a configured public origin keep setting it in their own body
The model resync now mutates the router under MODEL_RECONCILE_LOCK, and the
agent resync shares the new AGENT_RECONCILE_LOCK with the periodic agent
reload, so a reconcile built from a pre-write DB snapshot can no longer evict
or duplicate what a read-through just registered. Every resync checks
should_load_db_object for its object type, keeping read-through consistent
with what the replica is configured to load, and the a2a raise sites tag
ProxyModelNotFoundError as non-retryable so an agent miss no longer burns the
model resync budget.
The gemini 3.6 flash batch rates landed at half the standard rates in
94a29e0708, so the hardcoded standard-rate expectation started failing
on staging and red-lit misc / Run tests on every PR.
Resync registry misses with single-row DB fetches (guardrail by unique
name, agent by unique id or name, model by name then id) instead of
full-table loads, and bound them with a global budget of 20 resyncs per
5s window per registry that fails closed without negative-caching the
key.
Access group create/update now trust the reconcile outcome snapshot
captured under the reload lock instead of a post-lock router read, so a
concurrent reconcile can no longer surface a false degraded-serving 500.
Router.upsert_deployment restores the previously served deployment when
the replacement add fails under ignore_invalid_deployments, so a bad
update no longer silently drops a healthy deployment from serving.
The strict-denial pin only asserted a 401, so any unrelated 401 (a bad key, a deleted key) would have kept it green while tag routing silently broke. The harness now keeps the 401 response body, the way it already does for 429s, and the pin asserts the tag-routing denial message.
The earlier sweep only caught the conformance suite in tests/llm_translation.
Groq retired llama-3.1-8b-instant alongside llama-3.3-70b-versatile, and four
tests under tests/local_testing still call them for real, so litellm_router_testing
and both local_testing shards 404 with model_not_found.
Only the sites that leave the process move. The chunk fixtures in
test_stream_chunk_builder, and the cost and routing tests that never open a
socket, keep the old ids because the string is data there, not a request.
* feat(otel): attribute Prisma database spans to PostgreSQL instead of localhost
Prisma reaches PostgreSQL through a query engine on loopback, so transport
instrumentation attributes database waits to localhost and operators cannot
tell the work is PostgreSQL or correlate it with the database's own metrics.
Datastore service spans now carry db.system.name, db.system, db.operation.name
and, for PostgreSQL, server.address, server.port and db.namespace derived from
DATABASE_URL, and are emitted as CLIENT spans. Only host, port, database and
schema are read, so no credential reaches an exporter. Endpoint attributes are
omitted when a read replica is configured, because routing is decided per Prisma
call underneath the span.
* fix(otel): reject a mis-split DSN authority and name socket-only databases
An unencoded '/' in the password truncates the URL authority, so urlparse
reports the username as the host and the password tail as the database, which
put credential material in db.namespace. Postgres drivers reject that DSN
outright, so the only safe reading is no endpoint at all.
A hostless 'postgresql:///litellm' is a valid local-socket DSN that Prisma
accepts, and it now yields db.namespace with no server address rather than
nothing. The default schema is matched case-insensitively, since an unquoted
PostgreSQL identifier folds and one deployment must yield one namespace.
* fix(otel): keep a non-default schema in db.namespace
Prisma quotes the schema name, so a DSN with ?schema=PUBLIC provisions a
second schema alongside public rather than reusing it. Observed on a live
proxy: the PUBLIC schema came up with its own 70 tables next to public's 78,
and a key created under one was not visible under the other.
Case-folding the two into a single namespace therefore reported two different
schemas as the same database, which is the misattribution this feature exists
to remove. Match the default literally.
* fix(otel): reject any DSN whose userinfo fell outside the authority
An unencoded '#' or '?' in the password sends the tail to the fragment or
query, leaving the path empty, so the marker check on the database segment
never fired and urlparse's hostname (the database username) was exported as
server.address.
The stranded userinfo '@' is the general tell for every mis-split, so guard on
that instead of enumerating the characters that cause it.
* fix(otel): allow an at-sign inside a well-formed DSN query
The previous guard rejected any DSN whose userinfo at-sign fell outside the
netloc, which also caught libpq parameters that legitimately carry one, so
?application_name=svc@prod and ?user=admin@company.com lost their endpoint
attributes.
Discriminate instead: a PostgreSQL DSN never has a fragment, its database name
cannot hold an unencoded at-sign or slash, and an at-sign in the query is only
suspicious when the query did not parse as parameters.
* fix(otel): resolve the database endpoint per span instead of once per process
The endpoint was cached for the process lifetime on the premise that
DATABASE_URL is deployment-static. It is not. The RDS IAM refresh rebuilds the
URL from DATABASE_HOST/PORT/NAME/SCHEMA on every rotation, the reconnect path
re-reads DATABASE_URL, and the DB-backed environment_variables config overlay
sets arbitrary keys post-startup with no blocklist covering DATABASE_*. A
process that had genuinely failed over kept exporting the old server.
Read the environment per span, which is also what Prisma connects with, so the
span can no longer name a different server than the one serving the query;
get_secret_str consulted a secret manager first and could diverge from it. Only
the parse is memoized, keyed on the URL.
* fix(otel): reject a question-mark mis-split whose tail parses as parameters
A '?' in a password strands the rest of the authority in the query, and that
tail can still parse as key=value, so testing only for an unparseable query let
the login through as server.address. One spelling hijacked the host= parameter
and put the password suffix there directly.
A legitimate at-sign in a query always follows a database path, and a
'?'-mis-split never leaves one, so require both.
* refactor(otel): drop the DSN parse cache that retained rotated credentials
The cache was keyed on the full DATABASE_URL, so up to eight complete DSNs,
each carrying a password or a retired IAM token, stayed referenced for the
process lifetime and outlived the rotation that replaced them. Nothing reached a
span, but a heap dump or crash report would have surfaced them.
Parsing costs about four microseconds against a span emission that costs orders
of magnitude more, so the cache bought nothing worth that.
* fix(otel): avoid a set construction the tightened LIT002 budget rejects
* fix(otel): refuse an ambiguous DSN authority instead of guessing at it
A password holding both an unencoded slash and a query-like tail defeated all
three shape checks: the slash left a clean path carrying the password
remainder, the query still parsed as parameters, and no fragment survived. The
login went out as server.address, the password's leading digits as server.port
and the rest as db.namespace.
A DSN whose at-sign sits in a query parameter is indistinguishable from that
mis-split by any property of the parse; both leave no userinfo, a host, a port
and a path. Guessing wrong publishes a credential fragment, so the ambiguity
now resolves to refusing the endpoint. Such a DSN loses server.address and
db.namespace and keeps the rest of the span; percent-encoding the at-sign
restores them.
Also honour port= alongside host=, which libpq pairs and this read ignored.
* docs(otel): fix a spliced sentence and a stale cache claim in db_endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
gemini/imagen-4.0-generate-001 now 404s with "no longer available, please
update your code to use models/gemini-3.1-flash-image", and
groq/llama-3.3-70b-versatile reached its deprecation_date of 2026-08-16, which
is the day the llm_translation and image_gen jobs went red.
Only the two live call sites move. The remaining references to the old ids sit
in offline transformation tests, where the string is just a routing key and no
request leaves the process.
A file uploaded through Router.acreate_file lands in the account of the
deployment that stored it, so a cross-group fallback silently stores the
file with the wrong provider and every later batch or fine-tuning call
against the returned id permanently fails. Extend the provider-scoped
fallback pin that already covers input_file_id and training_file to file
creation, so the original provider error surfaces instead.
litellm.completion(thinking=True) crashed pre-network in is_thinking_enabled
with a retryable APIConnectionError ('bool' object has no attribute 'get'),
so the router burned retries on a deterministic failure and proxy clients got
a traceback instead of a usable response.
validate_and_fix_thinking_param now coerces thinking=True to the enabled dict
with the default medium budget and drops thinking=False, and the remaining
dict-assuming thinking accessors (base config, bedrock converse, deepseek)
guard with isinstance so raw bools can never crash a transform.
* test(ui): repoint the e2e locators at the post-antd form controls
Nine Playwright specs went red after the tremor and antd removals, none of
them because the product broke. The specs selected on markup those libraries
owned: tremor's TextInput stamped data-testid="base-input", the antd toast
facade rendered .ant-notification, and the team member modal's email field was
an .ant-select. Removing the libraries deleted those hooks silently.
Repoint each onto a user-facing locator that survives the next migration:
getByLabel for the key name, the MCP tool argument and the two cache pricing
fields, getByRole("combobox") for the team member email search, and the toast
container for the add-model success message.
The pricing fields needed a source change to be reachable at all. antd's
Form.Item used to assign the field name as the input id and tie the label to
it; the react-hook-form rewrite renders FieldLabel with no htmlFor and lets
FormField generate an opaque control id, so both cache inputs lost their
accessible name and could only be told apart by placeholder, which they share.
Pin the id back to the field name and point the label at it.
* fix(ui): let FormField own the pricing field label instead of hand-rendering one
The previous commit gave the cache cost inputs an accessible name by rendering
a FieldLabel with htmlFor next to the FormField. FieldLabel forwards Label's
props, and Label only accepts children, so next build failed type checking.
FormField already renders a label wired to the control id it generates, and
FormField.test.tsx covers that association, so passing label through is both
type-safe and less markup. The read-only branch keeps its plain FieldLabel,
which has no control to point at.
The pass-through method contract asserted that every route accepts GET, POST,
PUT, DELETE and PATCH. That holds for routes fronting a REST API, where
narrowing the set turns a request the upstream would have served into a 405,
but it does not hold for Amazon Comprehend Medical: it speaks AWS JSON 1.1,
which is POST-only, and its handler signs a POST upstream and requires a JSON
object body. Registering the other four verbs there would only produce routes
that always fail body parsing, so the route is POST-only on purpose and the
blanket assertion is the part that is wrong.
Carry a narrow exemption list instead of dropping the check. Exempted routes
are still pinned to an exact method set, so widening or narrowing one fails,
and a companion test asserts every exempted path is still registered so the
list cannot outlive the routes it covers.
Two suites broke because they stood in for production objects with stand-ins
that no longer answer the same way.
The redaction test faked a ResponsesAPIResponse and then reassigned
builtins.isinstance so the fake would pass the type check. Redaction now gates
on a tuple of accepted types, and the patched isinstance only recognised the
bare class, so the fake fell through to the generic branch and the assertions
ran against a plain dict. Building a real ResponsesAPIResponse drops the
builtins patch entirely and exercises the same type gate production takes.
The batch rate limiter tests constructed _PROXY_BatchRateLimiter with
parallel_request_limiter=None even though the parameter is not optional. That
stayed harmless until the output-token estimate started reading the limiter,
which turned it into an AttributeError. Inject the limiter the proxy injects,
sharing one InternalUsageCache the way _add_proxy_hooks does.
The aggregated daily-activity queries build their WHERE clause as raw SQL, and
an empty entity list rendered as `"team_id" IN ()`, which Postgres rejects with
`syntax error at or near ")"`. Callers reach that state normally: a caller
without admin view and no explicit team_ids has its scope resolved to the teams
it belongs to, so anyone belonging to no teams, an org admin for instance, sent
an empty list and got a 500 back from /team/daily/activity/aggregated.
The paginated endpoint hands the same empty list to Prisma, which renders
`in: []` and matches nothing, so it kept returning 200 with an empty result set.
Emit FALSE for the empty case so the raw-SQL path lands on the same answer,
mirroring what the api_key filter a few lines below already does.
The fix covers both aggregated queries at once because they share one WHERE
clause builder.
