Pre-call processing rewrites request_data["model"] for aliasing and routing, so
matching either key let a routed model count as the client's own name and put the
wrapper model back on an Azure Model Router row.
A chunk carrying usage is stored as a pre-restamp copy, so an alias-restamped
stream reaches disconnect billing with its first chunk still on the deployment
model and every later chunk on the client's name. That is the same shape Azure
Model Router produces, and the previous guard read it as a routed model and
left the alias on the row, which is the unpriced name this PR set out to stop.
Compare the assembled model against the name the proxy stamps chunks with, so
the alias goes back to the deployment's model and the routed model stays.
Pricing a frame through the request's own logging object is what makes custom
deployment pricing work, but _response_cost_calculator does not only return a
number. It also stamps cost_breakdown onto the live logging object, and on a
pricing failure it writes response_cost_failure_debug_information into
model_call_details.
On an ordinary proxy stream that is harmless, because the success handler
recomputes cost_breakdown at end of stream and overwrites whatever the frames
left behind. The pass-through handlers are the problem: they compute their final
cost with a bare completion_cost call and never touch cost_breakdown again, so a
breakdown derived from one mid-stream frame would survive to the end and land in
the spend log's metadata. response_cost itself is unaffected either way, so this
was a reporting surface bug rather than a billing one, but the spend row would
have gone from null to a populated breakdown for a partial frame.
Snapshot both writes and put them back once the cost is read, so pricing a frame
stays a read as far as the rest of the request is concerned. The returned cost is
unchanged, so nothing about the injected usage.cost moves.
team_allowed_routes and admin_allowed_routes only matched exact strings or named route groups, so a whole prefix of pass-through endpoints had to be listed route by route in config. Match trailing-wildcard patterns with the same helper the key-level allowed_routes check uses, so "/prefix/*" covers endpoints registered later.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The logging-object pricing applies to streamed /v1/chat/completions too, not
just Anthropic message_delta, so a deployment with negotiated per-token prices
now gets that price in the streamed usage.cost there as well. Nothing asserted
that half. Adds the discounted and the sticker-fallback case for the OpenAI
chunk shape, plus the branch where the pricer raises and the frame falls back
to model-name pricing instead of breaking the stream.
The disconnect billing path was stamping the wrapper's model over whatever
stream_chunk_builder assembled. For Azure Model Router that throws away the
routed model: the proxy deliberately leaves those chunks unrestamped so the
builder can pick the real model off a later chunk, and overwriting it prices
the row at the router alias instead.
Only apply the wrapper's model when the builder did not find a model beyond
the first chunk's, which is every case except Model Router.
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
cache_read_input_tokens and cache_creation_input_tokens are pydantic extras
on Usage, not declared fields, so filling them in created keys that were not
there before rather than replacing a None. Readers that test for presence
then took the new zero as authoritative: the spend log writer skipped its
own copy from prompt_tokens_details, turning a real cache read of 500 into
0, and the prometheus provider cache counters stopped incrementing.
Carry the prompt_tokens_details counts up before defaulting to zero, so a
partial row reports the same cache numbers a complete one does. Renamed the
helper to say what it now does.
Every pod schedules the budget reset job, so a fleet re-read the whole due
population and wrote it back against one Postgres at the same calendar
boundary, multiplying a single sweep by its replica count. The job now takes
the shared PodLockManager lease, so one pod sweeps per tick. A deployment with
no Redis keeps its previous behavior, and a Redis that cannot answer sweeps
unguarded rather than stranding every expired budget at its cap.
The per-window scan read every row carrying budget_limits in one statement, so
its cost grew with the deployment's key count. It is now keyset-paginated and
walks to the end of the table on every sweep. A per-run cap would need a resume
position, and no pod can hold one because the lease rotates between ticks, so
the strictly advancing cursor is what terminates the walk.
Found and updated rows were also JSON-serialized into the service hook's
metadata and into debug lines on every chunk, on the event loop, whether or not
anything consumed them. The hooks now carry counts, and the debug payload is
deferred until a record is actually emitted.
Resolves LIT-4793
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
SCIM roster writes were swallowed, so a group or user push returned 200 while the
team roster never received the membership. Surfacing the failure fixes that, but
aborting on the first failed write leaves the rest of the batch unattempted on top
of unrolled-back, which is worse than what it replaces.
Every roster write in a reconciliation is now attempted, and the ones that did not
land are reported together, naming each failed add and remove. Rollback would be the
other option and it is not safe here: the compensating write can fail too, and it can
strip a membership that pre-dated the push. SCIM reconciliation is idempotent, so a
named partial failure is what the IdP's next push needs to close the gap.
The reported status still follows the failures, so a unanimous 404 stays a 404 and
only a batch whose failures disagree falls back to 500.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The read replica never received the operator's DB pool settings, so its
Prisma pool fell back to `num_physical_cpus * 2 + 1` and the configured cap
was not enforced. Both startup paths now pass the same params to the reader:
the CLI, and the componentized entrypoints that go through
`DatabaseURLSettings.apply_to_env`.
Only pool and timeout params are inherited, through a single allowlist both
paths share. Anything that decides which tables a query resolves against
stays on the writer, including entries smuggled in through
`database_extra_connection_params`, so a writer `search_path` cannot repoint
reader queries. Params the operator pinned on the replica URL still win.
Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Cognition serves an OpenAI-compatible /v1/chat/completions endpoint, so it has been onboarded as
custom_llm_provider: openai. That books its traffic as OpenAI, which means OpenAI-specific cost
discounts and provider-level reporting apply to it.
Registers cognition through the JSON provider registry: a providers.json entry with
COGNITION_API_KEY and COGNITION_API_BASE, LlmProviders.COGNITION, the constants.py provider lists,
cost map entries for swe-1.6 and swe-1.7, the provider endpoints matrix, the dashboard provider
fields, and tests. JSON providers can now also be resolved from their base url alone, so an
api_base pointing at a known provider no longer falls through to an unresolved provider.
A JWKS fetch had no retry, so a single connect timeout to the identity provider
failed authentication outright, and once the cached copy expired there was
nothing to fall back on. How that surfaced depended on the outage shape:
httpx.ConnectTimeout was missing from DB_CONNECTION_ERROR_TYPES so it fell
through to the generic auth handler as a 401 with an empty detail, while a read
timeout took the database path and reported a healthy database as unreachable.
Transport failures are now retried three times with a short backoff, and the
last-known-good JWKS stays usable for a bounded window past public_key_ttl.
That window is public_key_stale_ttl, a new config field defaulting to 3600s and
settable to 0 to fail closed. It is checked on every read against the current
setting rather than baked into the cache entry when it is written, so lowering
it binds immediately instead of waiting for entries written under the old value
to age out, which matters because a shared cache survives the restart an
operator performs to make the change take effect. A copy whose write time
cannot be established is not servable. Only httpx.TransportError unlocks the
stale copy, so an identity provider that answers at all, including with a
narrowed key set, revokes on the next refresh. Every stale serve logs the kid
it authenticated, how long ago that copy was refreshed, and how long until it
stops being trusted.
A sustained outage is remembered for 30s per key url, so it costs one fetch per
window instead of three timeouts per request serialised behind the refresh lock.
Non-200 JWKS responses now raise instead of being cached as the key set, which
previously let an error body overwrite the last-known-good copy. An unreachable
identity provider with no cached copy left returns 503 auth_provider_unavailable.
Resolves LIT-5524
Co-authored-by: Yassin Kortam <yassin@berri.ai>
A request can name more than one model, through a comma-separated model or target_model_names on
the batch and fine-tuning routes, and the gate only looked at the string case, so an unpriced model
riding alongside a priced one went through and billed. Check every candidate and name the unpriced
ones in the 403
Aliases had the same problem on the other side: a group that prices itself through its model_info
block lands in the cost map under its deployment id, and the explicit-cost check walked the raw
model list by group name, so an alias pointing at that group read as unpriced. Resolve the group
through the router the way the pricing check already does
Also correct the 403 copy. Providers that return their own usage cost still bill for these models,
so the accurate claim is that litellm has no pricing of its own for them
Some IdPs, ADFS among them, return only `sub` from UserInfo and put the real
identity claims in the ID token or the access token. Those users land in the
Admin UI with no username, email, groups or teams.
Adds an opt-in `GENERIC_INCLUDE_TOKEN_CLAIMS` that merges token claims into the
UserInfo response before the existing `GENERIC_USER_*_ATTRIBUTE` mappings run.
Precedence is UserInfo, then id_token, then access token, and it applies to both
the PKCE and non-PKCE login flows. With the flag unset, behavior is unchanged.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Adds a chat_completions route module to litellm-core, mirroring the messages
route, plus Anthropic Messages and Bedrock Converse provider configs. The
per-model `rust: true` opt-in now covers /chat/completions for both providers.
The core accepts an allowlisted subset (text conversations, non-streaming) and
returns CoreError::Unsupported for anything else, so tool calls, multimodal
content and streaming fall back to the Python path transparently.
Resolves LIT-5698
tiktoken's BPE merge loop is quadratic in the length of a single regex piece, so a long
run of one repeated character turns a multi-MB payload into minutes of CPU. Encoding in
bounded chunks makes that linear, at a drift of at most ~1 token per chunk boundary.
Chunking alone only makes the stall shorter, so the async paths now count in a worker
thread: tiktoken releases the GIL for its Rust encode, so the loop keeps serving other
requests while a count is in flight. The /utils/token_counter endpoint awaits the new
atoken_counter, and the router's async deployment selection counts off-loop and hands
the result to _pre_call_checks instead of making it count inline.
The chunk size knob is bounded to [1, 4096]: a non-positive value used to raise or
silently report zero tokens, and an arbitrarily large one restored the quadratic cost
this exists to remove. Out-of-range and unparseable values warn and fall back to 1024.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Budget reservation tokenized every request twice, once for the max-cost
estimate and once for the input-cost estimate, and again per pricing
candidate. Tokenizing is O(prompt) and ran inline, so admitting one large
request stalled every other request the worker was serving.
Count the input tokens once per request and reuse the counts for both
estimates. Prompts above 30K characters of input text are counted in a
worker thread so the event loop stays free. The size heuristic renders the
body rather than walking its values, so tool-schema property names count
toward the threshold, and it sizes every field the counter tokenizes,
tool_choice included.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Agent registry CRUD (/v1/agents*) sat in agent_routes, which feeds
llm_api_routes, so DISABLE_LLM_API_ENDPOINTS returned "LLM API routes are
disabled for this instance." for every Admin UI Agents tab call. Split the
group the same way MCP is split: agent_inference_routes stays on the data
plane, agent_management_routes joins management_routes, and agent_routes
remains their union for keys configured with allowed_routes=["agent_routes"].
Non-admin callers reached agent CRUD through llm_api_routes before, so the
management paths also join self_managed_routes and the llm_api_routes virtual
key carve-out; the handlers already scope reads by role and 403 non-admin
writes.
Both new groups are tuples, so check_route_access now takes a Sequence and
matches wildcards through a generator instead of materializing an
intermediate list on every call.
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(scim): fail group sync when a member add or user creation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(scim): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A streaming chat completion that ends early (client disconnect, or the proxy
cutting the stream at LITELLM_MAX_STREAMING_DURATION_SECONDS) wrote a spend log
row with spend 0.0, prompt_tokens 0 on the proxy-cut path, and no cache fields
in usage_object. The proxy restamps chunk.model in place to the client-facing
alias, so the partial response rebuilt from those chunks priced the unmapped
alias and came out at 0. The failure path also rebuilt usage without the
request messages, so prompt tokens counted to 0, and a cut stream never sees
the final usage event that normally zero-fills the cache fields.
Restamp the rebuilt partial response with the wrapper's real model before cost
calculation on both the disconnect and the failure paths, pass the request
messages when rebuilding usage on the failure path, and zero-fill missing
cache usage fields the way completed streams already do.
Peer workers previously kept their startup value for block_requests_for_models_without_pricing
until a restart, so a toggle from the UI only took effect on the worker that served the request.
* test: merge three stranded twins into the files that shadow them
The second mirror's last four files each share a filename with a live test, so
the previous commit could not move them. Three of the four turn out to be plain
additions: their classes collide with nothing in the live file, so the tests are
extra coverage that has sat unrun rather than a competing version of anything.
Appending them takes the three files from 156 collected tests to 196, and all
196 pass. The 40 recovered are 13 OCI cases covering key normalization,
credential validation, complete-URL building and image-url transformation, 15
management-endpoint cases covering empty-value handling and the premium check,
and 12 DeepSeek thinking-parameter cases.
One assertion had to change. test_map_reasoning_effort_none_does_not_enable_thinking
asserted that reasoning_effort='none' leaves no thinking key, while the handler
maps it to {'type': 'disabled'} on purpose, documented in map_openai_params as
the OpenAI-style way to ask for thinking off. The test's stated intent holds,
since disabled does not enable anything, so it now asserts the disabled mapping
instead of the key's absence. Two imports moved to module scope for the
appended code, and no live test was touched.
test_discoverable_endpoints.py is the one left. Its twin grew from 1268 lines
to 9434, 25 of its assertions fail against today's code, and only 5 of its 19
tests have no counterpart, so deciding what survives that rewrite is a
judgement about the endpoints rather than a merge. The allowlist now holds
exactly that file and that reasoning.
* test(oci): stop the OCI suite reading credentials from the environment
validate_environment falls back to os.environ for every OCI credential and only
defaults the region when OCI_REGION is unset, so on a machine with OCI
configured the missing-credential test finds credentials it never passed and the
default-region test builds a URL for the ambient region. The suite then passes
or fails depending on who runs it.
A fixture drops the seven OCI variables for the four classes this branch added
and for TestOCIChatConfig, which had the same dependency before any of this and
fails the same way: with OCI_USER and friends exported, two of its cases fail on
origin/litellm_internal_staging today.
clean env: 83 passed
ambient OCI env: 83 passed
Same numbers either way, where the pre-existing file gave 68 passed / 2 failed
under the second.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
store_user_oauth_credential refused to overwrite any existing row that did not
decode as an OAuth2 payload, which conflated two states: a live BYOK secret that
reads back as plaintext, and ciphertext written under a LITELLM_SALT_KEY the proxy
no longer holds. The second is unrecoverable by any caller, so refusing preserved
nothing and instead wedged the user out of the OAuth flow permanently, since
re-authorizing is their only recovery.
The guard now raises only when the existing value is genuinely readable. An
undecryptable row is logged and replaced by the newly authorized token.
Both read paths were equally silent: get_user_oauth_credential and
list_user_oauth_credentials (which backs the bulk prefetch) each dropped an
undecryptable row indistinguishably from "user never authorized", so an operator
saw an upstream 401 and no hint that a credential had failed to decrypt. Both now
warn with the user and server ids, never the stored value.
A deployment that overrides any cost_per field, including at zero, now counts as priced so it is not blocked as unpriced
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.
Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:
- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
on the failure path, so the NameError, not the assertion, is what ran.
test_llm_guard_error_raising is the worst: it passes today with content
safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
supports 3.10. Guarded behind the exceptiongroup backport that anyio already
pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
a helper that catches HTTPException by a name it never imported, so the
challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
test_claude_agent_sdk, and an unused helper in test_end_users calling a
function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
FileNotFoundError behind a NameError.
Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
* feat(proxy): redact or drop individual batch records instead of rejecting the file
A single record tripping a guardrail rejected the whole upload, which is unusable for a file
holding thousands of rows. A record a guardrail rewrites is now submitted in its rewritten
form, a record it blocks is left out, and the create response reports every changed record by
both custom_id and line so a caller can reconcile against the file it sent. The same outcome
is written to the proxy log and to request metadata, so it is not visible only to the caller.
A rewritten record goes straight to a spool and only its offset is carried, so a masking
guardrail touching most rows of a large upload does not build a second copy of the file on the
heap, and the rewrite runs off the event loop the way the sibling full-file validation does.
Both proxy-injected metadata keys are captured from the record and restored exactly, including
an explicit null, so a masked row keeps the tags that decide how it is attributed.
A record is dropped only when a guardrail judged its content. `GuardrailRaisedException` now
carries `blocked_content` for that, because half its raise sites in the repo signal an
unreachable or unparseable backend under a fail-closed policy, and treating those as blocks
would turn "refuse this request" into "drop this record and submit the rest". The default is
off, so a raise that does not say what it means aborts the upload instead of silently
shrinking the file.
* fix(proxy): only drop a batch record on a verdict the guardrail actually reached
A guardrail that reports a technical failure as an HTTPException carrying a block status was
read as a content block, so an unreachable backend under a fail-closed policy quietly shrank the
file instead of failing the upload. Two in-tree integrations do exactly that, and one of them
defaults to fail-closed, so the broken configuration was the default one. Such an exception is
raised `from` the underlying error, which is a deliberate statement that something else caused
it, and no content verdict in the repo is raised that way, so the chain now settles it. Implicit
context is left alone, since a block raised inside an unrelated `except` would read as a failure.
Two annotation errors in the same family: the one GuardrailRaisedException subclass in tree never
opted into blocked_content, so a real block took the whole upload down with it, and straiker's
block helper is reached both from its verdict and from its fail-closed handler, so it claimed a
verdict for an outage. The helper now takes the flag from its caller.
A record could also opt itself out of the chain. Guardrail selection reads a body-level
`guardrails` key ahead of the proxy-injected list, and online that key can only add to the key
and team selection, never replace it, so a batch record naming an empty list skipped every
guardrail that was not default_on and was still reported as scanned. Every injected key is now
stripped before dispatch and restored afterwards.
A guardrail that reroutes a record to another model is honoured on the online path by rewriting
the model, which the scan read as a rewrite and submitted in the same file, sending content to
the provider the reroute existed to avoid. Every record of a batch file goes to one provider, so
the upload is refused instead, naming the line.
The scan spool is closed on the paths that never read it back.
* fix(proxy): give the scan the metadata bag guardrails actually read, and close its spools
The narrowed request metadata was installed under `litellm_metadata` only, but a record is
scanned as the chat request it describes, and the guardrails that pick a policy from a request
header read `metadata` instead. Noma choosing an application and Aim choosing a user both look
there, so the header allowlist added for them did not reach either one and a batch record was
still evaluated under the fallback policy. The scan metadata now goes into both bags, which are
both stripped and restored, so neither survives into the record that ships.
The scan spool was closed on the paths that abort, which are exactly the paths where it is
empty, and left open on the one path where it holds the rewritten records. Nothing closed the
rewrite output either, where before this feature the uploaded handle belonged to Starlette. The
upload now owns both and closes them however it exits.
* fix(proxy): register the scan spool before the rewrite can fail
The scan spool was added to the request's cleanup list only after the rewrite returned, so a
rewrite that raised, which for a spilled file can be as ordinary as the disk filling up, jumped
to the handler with the list still empty and left the scan's own handle open. The rewrite also
left its half-written output behind on that path, since nothing owns that handle until it is
returned. Both now close.
