A chunk carrying usage is stored as a pre-restamp copy, so an alias-restamped
stream reaches disconnect billing with its first chunk still on the deployment
model and every later chunk on the client's name. That is the same shape Azure
Model Router produces, and the previous guard read it as a routed model and
left the alias on the row, which is the unpriced name this PR set out to stop.
Compare the assembled model against the name the proxy stamps chunks with, so
the alias goes back to the deployment's model and the routed model stays.
Pricing a frame through the request's own logging object is what makes custom
deployment pricing work, but _response_cost_calculator does not only return a
number. It also stamps cost_breakdown onto the live logging object, and on a
pricing failure it writes response_cost_failure_debug_information into
model_call_details.
On an ordinary proxy stream that is harmless, because the success handler
recomputes cost_breakdown at end of stream and overwrites whatever the frames
left behind. The pass-through handlers are the problem: they compute their final
cost with a bare completion_cost call and never touch cost_breakdown again, so a
breakdown derived from one mid-stream frame would survive to the end and land in
the spend log's metadata. response_cost itself is unaffected either way, so this
was a reporting surface bug rather than a billing one, but the spend row would
have gone from null to a populated breakdown for a partial frame.
Snapshot both writes and put them back once the cost is read, so pricing a frame
stays a read as far as the rest of the request is concerned. The returned cost is
unchanged, so nothing about the injected usage.cost moves.
team_allowed_routes and admin_allowed_routes only matched exact strings or named route groups, so a whole prefix of pass-through endpoints had to be listed route by route in config. Match trailing-wildcard patterns with the same helper the key-level allowed_routes check uses, so "/prefix/*" covers endpoints registered later.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every live together_ai call in CI has answered 503 Service unavailable since
2026-08-20, across two runs 2.5 hours apart, while Together's status page
reported no incident in either window. These are real calls, not replayed
cassettes: the VCR layer runs filter_non_2xx_response, so a 503 is never
written to a cassette and cannot be replayed back.
Qwen/Qwen2.5-7B-Instruct-Turbo does not appear anywhere on Together's monitored
component list, whose Qwen entries are all Qwen3.x, so a model-level outage
there would never surface as an incident. The same 503 already forced
test_basic_rerank_together_ai to be skipped on a different together_ai model,
so per-model 503s are an established failure mode here rather than a platform
outage.
openai/gpt-oss-20b is the cheapest together_ai entry that carries real pricing
and the capabilities these suites exercise, at $0.05/$0.20 per 1M tokens with
function calling, response schema and tool choice. Together monitors it as a
served component. The retired model also carries null pricing in the cost map,
which is its own liability now that unpriced models are blocked.
test_multiple_deployments.py keeps the old id: it is a router fallback list
that is green today, and busting its cassette to prove a point would trade a
passing test for a live call this change cannot vouch for.
a369cb0da7 made completion() hand the prefixed model back to responses(), so
that responses() running get_llm_provider() a second time becomes a no-op
instead of stripping a prefix the model id owns. That was deliberate, and it
shipped with its own unit test, but it left two older assertions behind still
expecting the bare id.
#37744 corrected the openai one in test_openai.py. This is its azure sibling,
which llm_translation_testing has been failing on ever since.
Only the expected value moves. The neighbouring custom_llm_provider assertion
already passes and stays as it is.
The logging-object pricing applies to streamed /v1/chat/completions too, not
just Anthropic message_delta, so a deployment with negotiated per-token prices
now gets that price in the streamed usage.cost there as well. Nothing asserted
that half. Adds the discounted and the sticker-fallback case for the OpenAI
chunk shape, plus the branch where the pricer raises and the frame falls back
to model-name pricing instead of breaking the stream.
The disconnect billing path was stamping the wrapper's model over whatever
stream_chunk_builder assembled. For Azure Model Router that throws away the
routed model: the proxy deliberately leaves those chunks unrestamped so the
builder can pick the real model off a later chunk, and overwriting it prices
the row at the router alias instead.
Only apply the wrapper's model when the builder did not find a model beyond
the first chunk's, which is every case except Model Router.
models.litellm.ai and released litellm versions read
model_prices_and_context_window.json from main at runtime, so Kimi K3 is
missing from the hosted catalog even though the entry is in review for
litellm_internal_staging in #37552. This copies that entry onto main so
the catalog picks it up on its next fetch.
Data only: the cost map and its backup copy, no code changes. Pricing
matches Moonshot's published rates ($3/M input, $0.30/M cache read,
$15/M output, 1,048,576-token context). The fireworks_ai and Azure
Foundry kimi-k3 variants are separate work in #37512 and #37658; neither
touches the native moonshot/kimi-k3 key.
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
The support matrix says cognition serves /v1/responses, but the README row
left that column blank, so the two disagreed. Every other provider row tracks
the matrix, so bring this one in line.
The Model-Specific Limits rows now carry Input TPM and Output TPM, and a
limit the operator removes is sent as an explicitly empty map so
/project/update actually drops it instead of leaving the stored quota
enforced behind a UI that shows it gone.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
cache_read_input_tokens and cache_creation_input_tokens are pydantic extras
on Usage, not declared fields, so filling them in created keys that were not
there before rather than replacing a None. Readers that test for presence
then took the new zero as authoritative: the spend log writer skipped its
own copy from prompt_tokens_details, turning a real cache read of 500 into
0, and the prometheus provider cache counters stopped incrementing.
Carry the prompt_tokens_details counts up before defaulting to zero, so a
partial row reports the same cache numbers a complete one does. Renamed the
helper to say what it now does.
The swe-1.7 rates were briefly lowered to the standard tier. The docs page
records the API-served swe-1.7 as the Cerebras-served Lightning tier, so put
the matching rates back rather than have the cost map and the docs disagree.
Cognition also answers /v1/responses through the chat-completions bridge, the
same as every other provider in the JSON registry, so the endpoints support
matrix should say so instead of under-declaring it.
The swe-1.7 rates were carried over from the closed prior attempt and
match SWE-1.7 Lightning, 5x the SWE-1.7 Max and Medium rates the vendor
publishes. swe-1.6 was already on the standard tier, so the two entries
disagreed with each other. Both now read 0.5 in, 2.5 out, 0.2 cached per
million tokens.
Also drops the redundant registry comment in constants.py.
The passthrough tests and their coverage registry rows pointed at the internal
ticket id, which does not resolve for anyone following a link from
status.litellm.ai. Each test docstring and registry rationale now names the
GitHub issue it pins: #36086 for the two prefix routing cases, #36087 for the
file list cursors, #36523 for streamed Responses cost, and #36646 for
embeddings spend.
Every pod schedules the budget reset job, so a fleet re-read the whole due
population and wrote it back against one Postgres at the same calendar
boundary, multiplying a single sweep by its replica count. The job now takes
the shared PodLockManager lease, so one pod sweeps per tick. A deployment with
no Redis keeps its previous behavior, and a Redis that cannot answer sweeps
unguarded rather than stranding every expired budget at its cap.
The per-window scan read every row carrying budget_limits in one statement, so
its cost grew with the deployment's key count. It is now keyset-paginated and
walks to the end of the table on every sweep. A per-run cap would need a resume
position, and no pod can hold one because the lease rotates between ticks, so
the strictly advancing cursor is what terminates the walk.
Found and updated rows were also JSON-serialized into the service hook's
metadata and into debug lines on every chunk, on the event loop, whether or not
anything consumed them. The hooks now carry counts, and the debug payload is
deferred until a record is actually emitted.
Resolves LIT-4793
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
a369cb0da7 made the chat-to-responses bridge return the routing prefix on the
model it passes to responses(), so responses() re-resolving the provider is a
no-op instead of stripping a second prefix. It updated the bridge's own unit
tests but not this one, which still asserted the stripped id and has been
failing llm_translation_testing since that change landed.
The provider still receives gpt-5.4: responses() strips the openai/ prefix on
its own resolve, one layer later than this assertion used to sit. The stale
comment claiming the prefix is stripped before routing goes with it.
SCIM roster writes were swallowed, so a group or user push returned 200 while the
team roster never received the membership. Surfacing the failure fixes that, but
aborting on the first failed write leaves the rest of the batch unattempted on top
of unrolled-back, which is worse than what it replaces.
Every roster write in a reconciliation is now attempted, and the ones that did not
land are reported together, naming each failed add and remove. Rollback would be the
other option and it is not safe here: the compensating write can fail too, and it can
strip a membership that pre-dated the push. SCIM reconciliation is idempotent, so a
named partial failure is what the IdP's next push needs to close the gap.
The reported status still follows the failures, so a unanimous 404 stays a 404 and
only a batch whose failures disagree falls back to 500.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The read replica never received the operator's DB pool settings, so its
Prisma pool fell back to `num_physical_cpus * 2 + 1` and the configured cap
was not enforced. Both startup paths now pass the same params to the reader:
the CLI, and the componentized entrypoints that go through
`DatabaseURLSettings.apply_to_env`.
Only pool and timeout params are inherited, through a single allowlist both
paths share. Anything that decides which tables a query resolves against
stays on the writer, including entries smuggled in through
`database_extra_connection_params`, so a writer `search_path` cannot repoint
reader queries. Params the operator pinned on the replica URL still win.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cognition serves an OpenAI-compatible /v1/chat/completions endpoint, so it has been onboarded as
custom_llm_provider: openai. That books its traffic as OpenAI, which means OpenAI-specific cost
discounts and provider-level reporting apply to it.
Registers cognition through the JSON provider registry: a providers.json entry with
COGNITION_API_KEY and COGNITION_API_BASE, LlmProviders.COGNITION, the constants.py provider lists,
cost map entries for swe-1.6 and swe-1.7, the provider endpoints matrix, the dashboard provider
fields, and tests. JSON providers can now also be resolved from their base url alone, so an
api_base pointing at a known provider no longer falls through to an unresolved provider.
* test: point the live web search, groq and vertex image suites at models that still exist
Three CircleCI jobs on the staging-to-main promotion are red because the models
their live suites call have been retired by the providers, not because anything
in litellm changed.
openai/gpt-4o-search-preview now answers "has been deprecated" (its dated id
gpt-4o-search-preview-2025-03-11 carries deprecation_date 2026-07-23), so the
two web search conformance tests and the web search cost tracking test move to
gpt-5-search-api, the current search model. It keeps mode chat,
supports_web_search and a search_context_cost_per_query map, so the cost
assertion still resolves.
groq/llama-3.1-8b-instant reached its deprecation_date of 2026-08-16 and Groq
answers "does not exist or you do not have access to it". It follows
groq/llama-3.3-70b-versatile to groq/openai/gpt-oss-120b, the same replacement
PR #37422 already picked. The proxy config that job boots routes on a */*
wildcard, so no config change is needed.
vertex_ai/imagen-3.0-fast-generate-001 404s with "was not found or your project
does not have access to it". Google retired the whole Imagen family across
Vertex and the Gemini API, so there is no Imagen id left to point at. The class
is removed rather than repointed: Vertex image generation is already covered
live by TestVertexAIGeminiImageGeneration on vertex_ai/gemini-2.5-flash-image,
and the Imagen request and response transformations keep their offline coverage
in tests/test_litellm/llms/vertex_ai/image_generation/.
Only live call sites move. Remaining references to the old ids sit in offline
cost-map and transformation tests, where the string is a lookup key and no
request leaves the process.
* chore(lint): ratchet the TQ005 ceiling down to the count this branch reached
Removing the retired TestVertexImageGeneration class cleared one TQ005
violation, so the gate demands the limit come down with it.
make lint-budget-update only lowers a limit by the delta a branch cleared, and
this ceiling already sat 2 above the base count, so the tool landed on 2834
while the gate wants the limit at or below the 2832 this branch reached. The
remaining 2 are that stale headroom, which is exactly what the gate is asking
to reclaim.
The edit model is reached through the image generation path with fal's
image_urls param; /v1/images/edits is not wired for fal_ai and errors.
Point supported_endpoints at /v1/images/generations and say so in the
entry notes.
* test(ci): serve /moderations from the canned OpenAI mock
The otel proxy E2E job points its `openai/*` wildcard deployment at the
canned mock, and #37492 made `get_model_list` agree with
`get_available_deployment` on bare model names. /moderations now resolves
`omni-moderation-latest` to that wildcard deployment the way
/chat/completions already did, so the request lands on the mock, which
never implemented the route and answers a bare 404.
Add /moderations and /v1/moderations to the mock, returning an
OpenAI-shaped response with one result per input item.
* style(ci): annotate the new moderations locals as Final
The monitors are separate servers with their own password, so the data node's Entra or
IAM token has no standing there. Dropping the provider only when a Sentinel password was
configured left it in place for unauthenticated monitors, where redis-py sends it as an
AUTH the monitor rejects and async Sentinel discovery fails.
The generic container handler returned response.content for endpoints marked
returns_binary before it ran any status or error check, so a non-2xx answer
from the provider was handed back to the caller as raw bytes. Asking for the
content of a container file that does not exist returned the provider's 404
error body as an opaque payload instead of raising.
Move the check ahead of the binary short-circuit and apply it to every
container file endpoint, falling back to the response text when the error body
is not JSON.
Entering a custom model name on the Add Model form crashed the whole page
to "This page couldn't load" (React error #185, maximum update depth
exceeded), taking the provider credential fields down with it, so the
model could never be created.
ConditionalPublicModelName kept a `tableKey` counter and bumped it from
an effect on every run to force the mappings table to remount. That was
harmless under antd, whose useWatch handed back the stored array. React
Hook Form's useWatch returns a fresh array each render, so the effect's
dependency changed every render, the effect bumped state again, and the
render loop never settled.
The table is driven by its `data` prop, so the remount counter buys
nothing: drop it, key the effects off the selection contents rather than
the array identity, and write model_mappings only when they actually
change. The two `react-hooks/set-state-in-effect` suppressions on this
file, which were recording exactly this bug, go with it.
A JWKS fetch had no retry, so a single connect timeout to the identity provider
failed authentication outright, and once the cached copy expired there was
nothing to fall back on. How that surfaced depended on the outage shape:
httpx.ConnectTimeout was missing from DB_CONNECTION_ERROR_TYPES so it fell
through to the generic auth handler as a 401 with an empty detail, while a read
timeout took the database path and reported a healthy database as unreachable.
Transport failures are now retried three times with a short backoff, and the
last-known-good JWKS stays usable for a bounded window past public_key_ttl.
That window is public_key_stale_ttl, a new config field defaulting to 3600s and
settable to 0 to fail closed. It is checked on every read against the current
setting rather than baked into the cache entry when it is written, so lowering
it binds immediately instead of waiting for entries written under the old value
to age out, which matters because a shared cache survives the restart an
operator performs to make the change take effect. A copy whose write time
cannot be established is not servable. Only httpx.TransportError unlocks the
stale copy, so an identity provider that answers at all, including with a
narrowed key set, revokes on the next refresh. Every stale serve logs the kid
it authenticated, how long ago that copy was refreshed, and how long until it
stops being trusted.
A sustained outage is remembered for 30s per key url, so it costs one fetch per
window instead of three timeouts per request serialised behind the refresh lock.
Non-200 JWKS responses now raise instead of being cached as the key set, which
previously let an error body overwrite the last-known-good copy. An unreachable
identity provider with no cached copy left returns 503 auth_provider_unavailable.
Resolves LIT-5524
Co-authored-by: Yassin Kortam <yassin@berri.ai>
