The auto-routed model group was only reachable through the
x-litellm-model-id response header. SDK and framework callers that do not
expose response headers had no way to read it, and under streaming there
was no body surface at all.
The response body `model` field is deliberately restamped back to the
client-requested alias on both paths, which is correct OpenAI semantics,
so this adds a separate namespaced `router_model_name` key instead of
redefining `model`. The key is written on non-streaming bodies and on
every SSE chunk, including the streaming fast path, and is emitted only
when an auto-routing strategy actually selected the deployment.
After a mid-stream fallback moves the request off the group the router
picked, the key is omitted rather than continuing to claim the original
tier. The router marker already supports per-chunk fallback signals via
`x-litellm-attempted-fallbacks` headers; this wires that signal into
the gate so no stale tier is claimed after a fallback fires.
Also removes a redundant function-local import in the streaming
generator that shadowed the module-level one for the whole function.
The Prisma query engine is a separate process whose resident memory grows with
what it is asked to hold and glibc never returns it, so a pod's memory floor
ratchets up to its worst statement and stays there for the life of the worker.
#34956 bounded a spend-log flush by payload bytes, which caps that floor when
prompts are stored and does nothing when they are not: rows carrying only
attribution metadata run about 1.2 KB, so a 1000-row statement is roughly
1.2 MB, the 2 MB byte budget never binds, and every statement stays at 1000
rows forever.
The engine charges per row as well as per byte. Measured on a container running
the same engine build (5.4.2) against real Postgres, with rows shaped like a
store_prompts_in_spend_logs=false deployment, writing the same 200,000 rows:
rows/statement engine RSS still resident after the flush
1000 179 MB
500 91 MB
250 41 MB
100 19 MB
None of those statements came near the byte budget, so the whole difference is
row count. The floor is a plateau rather than a leak: 1,000,000 rows written at
1000 per statement settles around 229 MB and stops climbing.
Adds SPEND_LOG_WRITE_BATCH_MAX_ROWS, default 100, applied alongside the
existing byte budget so whichever binds first splits the statement. Both are
needed, since bytes are what track a prompt-carrying row and rows are what
track the engine's per-row bookkeeping.
One consequence worth naming: a flush now issues more statements, and a
statement that fails under a poison flood costs one insert before any
isolation runs, so the irreducible floor rises by the statement count. The
isolation budget still caps the amplification on top of that, and the tests
assert the bound derived from the configured row cap rather than a constant.
Per-model budgets were three separate things pretending to be one. The
enforcement check, the post-call increment and the info endpoints each derived
their own cache key, so a budget could refuse traffic at 429 while /key/info
reported zero usage, and a Bedrock model id never matched a budget keyed on the
bare family name. /user/new echoed a model_max_budget back and stored an empty
dict, and nothing enforced a user-scoped per-model budget at all.
One owner now builds the counter key from the configured budget model, and
enforcement, the increment and the info endpoints all read it. Bedrock ids
resolve through the model-cost map. Auth carries the user's budget onto the
token on every branch that reaches the spend hook, including JWT and
auto-registration. Native passthrough attaches the three budget metadata keys
its StandardLoggingUserAPIKeyMetadata does not carry, so /anthropic/... and
/bedrock/... traffic is counted and capped like /v1/chat/completions.
The dashboard gains the per-model budget editor it never had, on the key create,
key edit and internal-user edit forms. It is read-only without an enterprise
license, matching the write gate the proxy already enforces, and an untouched
budget is left out of an update so an unrelated edit cannot trip that gate.
The editor hydrates from either BudgetConfig spelling, since model_max_budget is
a plain dict that the proxy stores exactly as the client sent it, and it carries
through the fields it does not model. Without both, editing one model would drop
another model row entirely and silently discard its tpm_limit and rpm_limit.
/user/info refreshes its local copy of the user field by field after a save, so
model_max_budget joins that list. Left out, a saved cap read back as the old one
when the form was reopened, and clearing the row to recover would then wipe the
value that had actually persisted.
A zero-dollar cap is the strictest limit expressible, not the absence of one,
so it is enforced rather than skipped on falsiness, spend exactly at the cap is
refused the way every sibling budget check already refuses it, and a counter
that was never written reads as zero spend rather than as unknown. The usage
endpoints read every counter in one batched lookup, so a large model_max_budget
cannot fan out into one concurrent cache call per configured model.
Every auth path honours the same zero-cost skip flag, so none of them can refuse
a free request that another serves. The custom-auth helper gains the flag it
never had, which also changes its pre-existing key and end-user checks.
The compaction summary gate checks the user scope alongside the key and end-user
ones. This file propagates all three budgets into the summary subrequest, so
enforcing only two let compaction increment a counter it could not be refused by.
Custom auth attaches the user's budget to the token unconditionally, since the
post-call spend hook reads it there: gating the attach on the same condition as
enforcement left the counter uncharged whenever the request was not itself
enforceable. An entry that will not validate is skipped rather than raised on,
so one malformed scope cannot abort every other scope's increment or turn a
config typo into a 500.
The edit forms re-seed the budget editor when a different key or user is loaded.
Its rows are seeded once and cannot re-read their own value prop, so without
this a save wrote the previously loaded record's budgets onto the current one.
Only the built-in provider pass-through routes carry the budget metadata.
get_model_from_request deliberately resolves no model for a user-defined
pass-through, since its body is forwarded verbatim and names an upstream model,
so attaching there would charge a counter nothing on that route can refuse.
tests/litellm/ was a second mirror beside tests/test_litellm/ that no workflow,
Makefile target, or CircleCI job ever named. Its other 33 files were reconciled
during August 2026; this one stayed behind under a ci-coverage-allowlist entry
asking a later pass to decide which of its five orphan behaviours still hold.
They no longer hold as written: 25 of its 32 cases fail against today's code,
because the file froze on the day it stopped being collected and the endpoints
kept moving. Three of the five are already covered by the live twin, and better.
test_get_request_base_url_xff_trust_gate parametrizes the trust gate in both
directions, including the exact untrusted-caller case the orphan asserted, and
the standard and legacy protected-resource shapes are both exercised through
use_standard_pattern.
The other two were the only tests anywhere for validate_trusted_redirect_uri
under that same gate, so they are ported rather than dropped, rebuilt on the
live file's request-mock conventions. Both directions are load-bearing: forcing
is_request_from_trusted_proxy to True fails the untrusted case, forcing it to
False fails the trusted one.
313 tests pass in the live file, up from 311. Dropping the dead file clears one
zero-assert TQ001 violation, so its ceiling ratchets down with it.
GPT-5.6 Sol, Terra and Luna reached the bedrock-runtime data plane on
2026-08-17, separately from the existing bedrock-mantle path. On runtime
they are served only through cross-region inference profiles, so
bedrock/us.openai.gpt-5.6-* had no cost map entry and fell through to the
Invoke route, which rewrites the token cap to max_tokens and is rejected
as unsupported_parameter on both /v1/chat/completions and /v1/responses.
Register the Geo and Global profiles as bedrock_converse so routing
reaches Converse, which AWS documents and serves for these models, and
price each profile from its own published rate table. No bare key: the
control plane reports inferenceTypesSupported INFERENCE_PROFILE with no
on-demand throughput, so a bare id is not invocable.
Declare the published cache-read and cache-write rates. Bedrock rejects
an explicit cachePoint block for these models, so supports_prompt_caching
stays off, but it caches long prefixes implicitly and reports the cache
tokens in usage either way. Without the cost fields a cache-read turn
bills only its uncached tokens: measured against live Bedrock, a
15609-token cached prefix came to $0.000176 instead of $0.00876095.
Clients that resend a long prefix every turn are the worst affected.
Reasoning stays unadvertised. Converse rejects the Anthropic-shaped
thinking block LiteLLM sends for reasoning_effort; the shape these models
accept is additionalModelRequestFields {"reasoning": {"effort": ...}},
which needs a transform change tracked by #34105. Advertising it without
that change is what made the earlier attempt in #37307 fail.
Applies the review suggestions. The cost map now carries the rates published on
https://scx.ai/pricing, GLM-5.2 at 0.61 in, 0.22 cached, 1.98 out and
Qwen3.8-Max at 1.65 in, 0.21 cached, 4.99 out per million tokens, and cites that
page as the source rather than a third party gateway. The provider link is
corrected to https://docs.litellm.ai/docs/providers/scx_ai to match the page
that shipped as scx_ai.md. Both the primary files and their backup mirrors are
updated.
Resolves the three conflicts against the JSON provider registry refactor. The
hardcoded api.scx.ai base-url branch in get_llm_provider_logic.py is dropped in
favour of the generic JSONProviderRegistry.get_by_base_url lookup, which reads
the same base_url and api_key_env from providers.json and additionally honours
an explicitly passed api_key. constants.py and types/utils.py keep both the
cognition and scx-ai entries added on either side.
Master-key auth stamps the stable alias litellm_proxy_master_key instead of the
raw key, so spend logs carry a readable, non-secret identifier for those rows.
The new redaction path only recognized sha256 and hashed-jwt shapes, so it
hashed that alias and broke continuity with every master-key row written
before this change. The alias joins the recognized non-secret values, still
behind the same provenance gate, so a caller who sends the alias string as
their own bearer token still gets it hashed.
The proxy conftest already snapshots master_key and prisma_client around every
test, because a value left behind on litellm.proxy.proxy_server poisons the rest
of the xdist worker. llm_router has the same problem. The PTU rollup reads the
running router out of sys.modules, so a router a sibling test left behind lands
in its deployment scan and three test_ptu_flat_cost_rollup tests fail or pass
depending on how xdist happens to split the shard.
The hashed-jwt branch trusted the value's shape alone, so a caller-supplied key in that shape was stored unhashed. Both pass-throughs now require the value to match the auth-time user_api_key_hash, and the shape check is a full match.
The spend-log helper no longer treats a 64-hex shape as proof a value was already hashed, so this case has to say where the hash came from. Reconciles the test that came in with #31799 against that change.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
sagemaker_chat never put X-Amzn-SageMaker-Inference-Component on the request, so any endpoint
backed by inference components answered 400 INFERENCE_COMPONENT_NAME_MISSING and the call never
reached the container. The legacy sagemaker provider has built that header from model_id since
#8889, and this brings the chat provider in line. It goes on in validate_environment, which runs
before the request is SigV4-signed, so the signature covers it
The request body also always named the endpoint rather than the served model, which containers
that validate the body's model answer with a 404. hf_model_name now becomes the body's model
when it is set, and endpoints that do not set it keep sending exactly what they send today
Add sync and async regression tests for the BaseLLMHTTPHandler streaming
path, which forwards provider response headers for the ~30 providers that
ride the generic handler and had no coverage. Also drop redundant setup
prose from the moonshot invoke test docstring.
`members_with_roles` is a denormalized JSON snapshot written at add-time.
`_update_team_members_list` backfilled `user_id` from `user_email` but never
the reverse, so a member added by `user_id` alone was stored with
`user_email=None` permanently - and `/team/info` returns that blob verbatim
with no join to `LiteLLM_UserTable`, so the Admin UI's member table renders
"-" for a user that plainly has an email.
Fix both ends:
- write path: `_resolve_member_identity` resolves identity both ways off the
user rows the add just touched, so new roster entries stop being born blank.
- read path: `/team/info` fills blank emails from `LiteLLM_UserTable` in one
indexed `user_id IN (...)` query, repairing rows already in the database.
Members that already carry an email are passed through untouched and cost
no query, so this only ever turns a null into the right value.
The cost map shipped cognition/swe-1.7 at $2.50 in / $12.50 out per million with
$1.00 cache reads. Those are the Lightning numbers. Cognition's own model list at
https://docs.devin.ai/desktop/models has uid swe-1-7 at $0.50 / $2.50 with $0.20
cache reads, and uid swe-1-7-lightning at $2.50 / $12.50 with $1.00 cache reads,
so every swe-1.7 call has been costed at 5x since the entry landed.
swe-1.7 now carries the standard rates and the Lightning tier gets its own entry,
in both cost map copies. The source field on both moves to the desktop models page,
which is the one that lists both tiers.
* test: enforce PT012 so a pytest.raises block cannot hide dead assertions
`with pytest.raises(...)` stops at the first statement that raises. Anything
sequenced after it inside the block never runs, so an assertion written there is
never checked and the test still reports green.
Two sites were doing exactly that, and both assertions turned out to be wrong
once they started running. tests/llm_translation/test_prompt_factory.py asserted
the bedrock rejection names "requires at least one non-system message", which
holds. tests/proxy_unit_tests/test_proxy_server.py asserted the prisma startup
failure mentions "httpx.ConnectError", which never appears: the failure is an
httpx.ConnectError whose message is "All connection attempts failed", so that
test now asserts the type. Its DATABASE_URL override moves to monkeypatch, since
the old restore sat below the assertion and leaked the invalid URL into every
later DB test the moment the assertion started being able to fail.
The remaining 72 sites are rewritten without changing what they exercise: setup
that cannot raise moves above the block, a nested `patch` moves outside it, and
bodies with real control flow (a stream drain, an if/else on sync_mode, a
retry loop) move into a local closure the block calls.
Fixing PT012 unmasked two B017s, since ruff only reports a blind
pytest.raises(Exception) once the block holds a single statement.
tests/proxy_unit_tests/test_auth_checks.py narrows to the ProxyException
can_key_call_model actually raises. tests/local_testing/test_completion_cost.py
was asserting vertex_ai/medlm-medium has no cost entry, which stopped being true
at some point; that dead first half is gone and the rest of the test, which
checks medlm pricing resolves above zero, now runs instead of being skipped.
* chore(ci): ratchet TQ004 to 768 after the prisma test moved to monkeypatch
SCIM group members were matched against litellm user ids only. An identity
provider that lists people by email or by the OIDC subject therefore matched
nothing, and the member fell through to placeholder creation.
Since #37688 made a failed member creation fail the group sync rather than drop
the member, that fallthrough is no longer quiet: the placeholder is created with
user_email set to the member value, the duplicate-email check rejects it, and the
whole group push answers 500. So on current staging a group listing anyone by
their email fails outright, every other member in the payload included.
An unmatched member id is now looked up across sso_user_id and user_email in one
query. Searching either field first would hide a value that names one account by
its SSO identity and another by its email, and hand the group to whichever was
searched first. The two are not compared alike: an email is matched the way
new_user matches one before accepting a new account, case-insensitively, because
matching more strictly than the layer that would reject the placeholder is what
turned an id whose casing differed from the stored email into that same 500. An
SSO identity is matched exactly, since OIDC defines sub as case-sensitive and
nothing folds its case on the way in.
An exact user_id hit is checked the same way rather than trusted outright, since a
value can be one account's id and another's SSO identity or email. That is not a
corner case: the placeholders this bug provisioned are keyed by the very id the
provider keeps pushing, so on a tenant that already has them the placeholder wins
the id lookup and the real account can never be matched. Refusing names the
problem instead of silently landing on the placeholder again. Those rows still
have to be deleted before the real account resolves; making the sync heal itself
needs a trustworthy way to tell a placeholder from an account someone created, and
created_via lives in caller-writable metadata, so it is left to a follow-up.
A value that names more than one account is refused with a 400 naming the id
rather than attributed to one of them.
Removals resolve too, since the roster holds canonical user ids and a directory
removes people by the id it added them with. A removal counts the members one
value names: the id as written when the roster holds it verbatim, which is how an
earlier release recorded a member it could not match, together with the members it
resolves to. Counting only the accounts on the roster keeps someone removable
after a second account takes their email, which resolving table-wide would not,
and counting both ways of naming a member together stops one value revoking two
people when it is one member's canonical id and another's email. A value naming
two of the group's own members is undecidable and fails rather than guessing or
reporting a removal it did not perform.
Resolves LIT-5383
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Pre-call processing rewrites request_data["model"] for aliasing and routing, so
matching either key let a routed model count as the client's own name and put the
wrapper model back on an Azure Model Router row.
A chunk carrying usage is stored as a pre-restamp copy, so an alias-restamped
stream reaches disconnect billing with its first chunk still on the deployment
model and every later chunk on the client's name. That is the same shape Azure
Model Router produces, and the previous guard read it as a routed model and
left the alias on the row, which is the unpriced name this PR set out to stop.
Compare the assembled model against the name the proxy stamps chunks with, so
the alias goes back to the deployment's model and the routed model stays.
Pricing a frame through the request's own logging object is what makes custom
deployment pricing work, but _response_cost_calculator does not only return a
number. It also stamps cost_breakdown onto the live logging object, and on a
pricing failure it writes response_cost_failure_debug_information into
model_call_details.
On an ordinary proxy stream that is harmless, because the success handler
recomputes cost_breakdown at end of stream and overwrites whatever the frames
left behind. The pass-through handlers are the problem: they compute their final
cost with a bare completion_cost call and never touch cost_breakdown again, so a
breakdown derived from one mid-stream frame would survive to the end and land in
the spend log's metadata. response_cost itself is unaffected either way, so this
was a reporting surface bug rather than a billing one, but the spend row would
have gone from null to a populated breakdown for a partial frame.
Snapshot both writes and put them back once the cost is read, so pricing a frame
stays a read as far as the rest of the request is concerned. The returned cost is
unchanged, so nothing about the injected usage.cost moves.
team_allowed_routes and admin_allowed_routes only matched exact strings or named route groups, so a whole prefix of pass-through endpoints had to be listed route by route in config. Match trailing-wildcard patterns with the same helper the key-level allowed_routes check uses, so "/prefix/*" covers endpoints registered later.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every live together_ai call in CI has answered 503 Service unavailable since
2026-08-20, across two runs 2.5 hours apart, while Together's status page
reported no incident in either window. These are real calls, not replayed
cassettes: the VCR layer runs filter_non_2xx_response, so a 503 is never
written to a cassette and cannot be replayed back.
Qwen/Qwen2.5-7B-Instruct-Turbo does not appear anywhere on Together's monitored
component list, whose Qwen entries are all Qwen3.x, so a model-level outage
there would never surface as an incident. The same 503 already forced
test_basic_rerank_together_ai to be skipped on a different together_ai model,
so per-model 503s are an established failure mode here rather than a platform
outage.
openai/gpt-oss-20b is the cheapest together_ai entry that carries real pricing
and the capabilities these suites exercise, at $0.05/$0.20 per 1M tokens with
function calling, response schema and tool choice. Together monitors it as a
served component. The retired model also carries null pricing in the cost map,
which is its own liability now that unpriced models are blocked.
test_multiple_deployments.py keeps the old id: it is a router fallback list
that is green today, and busting its cassette to prove a point would trade a
passing test for a live call this change cannot vouch for.
a369cb0da7 made completion() hand the prefixed model back to responses(), so
that responses() running get_llm_provider() a second time becomes a no-op
instead of stripping a prefix the model id owns. That was deliberate, and it
shipped with its own unit test, but it left two older assertions behind still
expecting the bare id.
#37744 corrected the openai one in test_openai.py. This is its azure sibling,
which llm_translation_testing has been failing on ever since.
Only the expected value moves. The neighbouring custom_llm_provider assertion
already passes and stays as it is.
The logging-object pricing applies to streamed /v1/chat/completions too, not
just Anthropic message_delta, so a deployment with negotiated per-token prices
now gets that price in the streamed usage.cost there as well. Nothing asserted
that half. Adds the discounted and the sticker-fallback case for the OpenAI
chunk shape, plus the branch where the pricer raises and the frame falls back
to model-name pricing instead of breaking the stream.
The disconnect billing path was stamping the wrapper's model over whatever
stream_chunk_builder assembled. For Azure Model Router that throws away the
routed model: the proxy deliberately leaves those chunks unrestamped so the
builder can pick the real model off a later chunk, and overwriting it prices
the row at the router alias instead.
Only apply the wrapper's model when the builder did not find a model beyond
the first chunk's, which is every case except Model Router.
