Budget reservation tokenized every request twice, once for the max-cost
estimate and once for the input-cost estimate, and again per pricing
candidate. Tokenizing is O(prompt) and ran inline, so admitting one large
request stalled every other request the worker was serving.
Count the input tokens once per request and reuse the counts for both
estimates. Prompts above 30K characters of input text are counted in a
worker thread so the event loop stays free. The size heuristic renders the
body rather than walking its values, so tool-schema property names count
toward the threshold, and it sizes every field the counter tokenizes,
tool_choice included.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
The shared logging ThreadPoolExecutor uses an unbounded work queue, so
sync callbacks that fall behind request arrival pin every queued payload
in memory until the task restarts. Cap queued-plus-running work with a
semaphore, shed submissions past the cap, and warn at most once every 30
seconds naming the knob that raises it. No caller of the shared executor
reads the returned future, so shedding is safe.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Agent registry CRUD (/v1/agents*) sat in agent_routes, which feeds
llm_api_routes, so DISABLE_LLM_API_ENDPOINTS returned "LLM API routes are
disabled for this instance." for every Admin UI Agents tab call. Split the
group the same way MCP is split: agent_inference_routes stays on the data
plane, agent_management_routes joins management_routes, and agent_routes
remains their union for keys configured with allowed_routes=["agent_routes"].
Non-admin callers reached agent CRUD through llm_api_routes before, so the
management paths also join self_managed_routes and the llm_api_routes virtual
key carve-out; the handlers already scope reads by role and 403 non-admin
writes.
Both new groups are tuples, so check_route_access now takes a Sequence and
matches wildcards through a generator instead of materializing an
intermediate list on every call.
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(scim): fail group sync when a member add or user creation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(scim): apply ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
REDIS_URL-based async clients and every async connection pool dropped the
managed-identity credential the caller configured, so they connected
unauthenticated against an auth-enforcing Redis. The conversion from
redis_connect_func to a CredentialProvider now happens once, before any
branch, and covers the url, sentinel, cluster, and pool paths alike.
Also adds credential_provider to the cluster kwargs allowlist, which
silently filtered it out.
A streaming chat completion that ends early (client disconnect, or the proxy
cutting the stream at LITELLM_MAX_STREAMING_DURATION_SECONDS) wrote a spend log
row with spend 0.0, prompt_tokens 0 on the proxy-cut path, and no cache fields
in usage_object. The proxy restamps chunk.model in place to the client-facing
alias, so the partial response rebuilt from those chunks priced the unmapped
alias and came out at 0. The failure path also rebuilt usage without the
request messages, so prompt tokens counted to 0, and a cut stream never sees
the final usage event that normally zero-fills the cache fields.
Restamp the rebuilt partial response with the wrapper's real model before cost
calculation on both the disconnect and the failure paths, pass the request
messages when rebuilding usage on the failure path, and zero-fill missing
cache usage fields the way completed streams already do.
Peer workers previously kept their startup value for block_requests_for_models_without_pricing
until a restart, so a toggle from the UI only took effect on the worker that served the request.
Route fal.ai's openai/gpt-image-2 endpoints through a dedicated transformation that maps OpenAI image params (n, size, quality, output_format) into fal's schema, and register the model in the cost map.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do
POST /model/new answers 400 when PTU fields are set without a team_id, a
ptu_effective_from, or the count and rate together. config.yaml ran none of
those checks, so the same deployment loaded and served, billing per token
while accruing no flat cost, with nothing logged.
The rule moves into litellm_core_utils.ptu_pricing so both paths state it
once. Registration refuses such a deployment and names it, and the proxy's
ignore_invalid_deployments keeps that to the one entry. Only enforced while
PTU cost attribution is enabled, so a proxy that never opted in is unchanged.
* refactor(ptu): build the refusal message in the module that owns the rule
router.py raised a message it composed itself, which put proxy-facing
wording on the shared SDK surface. ptu_config_error now takes the
deployment name and returns the whole sentence; the endpoints still ask
without a name and their 400 bodies are unchanged.
JSON-RPC 2.0 types `id` as string, integer or null, but
LiteLLMSendMessageResponse annotated it as a bare required `str`. Pydantic v2
dropped v1's int-to-str coercion, so an upstream agent echoing an integer id was
rejected outright, and a null id, which section 5 requires for an error that
cannot be correlated to a request, was rejected too. Both surfaced as -32603 with
a pydantic ValidationError in the message: five distinct 500s on
/a2a/{agent_id}, across message/send and tasks/get.
Everything around the model already handled the full union: the endpoint reads
the id off the body as Any, its helpers are typed `str | int | None`, the error
builder takes `object`, and the streaming path passes the id through untouched.
The response model was the only narrowing left.
Backfilling an id the agent omitted keeps the caller's type too, since JSON-RPC
requires the response id to equal the request id and a caller that sent 7 cannot
correlate a response carrying "7".
`bool` is excluded from the integer half even though it subclasses `int`, so a
boolean id is stringified rather than relayed as 1 or 0, where it would collide
with a real integer id another in-flight request may be using.
* feat(ci): freeze the conftest save/restore inventory so it can only shrink
* fix(ci): resolve the named constant a conftest save loop iterates
* fix(ci): match the snapshot shape instead of a list of blessed dict names
* feat(ci): fail a branch that clears TQ violations without lowering the ceiling
A limit that only ever falls is not the same as one that falls when it can.
Clearing violations and leaving the ceiling above the new count let the same
violations return later under a limit nobody moved, so the gate now fails on
that and names `make lint-budget-update` as the fix. It needs both head below
base and head below limit, so headroom already in the base is never blamed on
the branch that happens to run next.
Drops the seeded-rule exemption from the ratchet along with it. Its stated
reason was that the base tree predates a rule introduced on this branch, but
base counts are measured with the current checker, so such a rule is counted at
the base too and its grandfathered total was never at risk of reading as fixed.
Removing the exemption is what lets a newly seeded rule ratchet like the six
that came before it.
The base scan is skipped when the branch touches neither the test tree nor the
checker, since neither count can have moved.
* feat(ci): assert .github/workflows holds only workflows, correctly named
* style(tests): annotate the hygiene test module's names with Final
* fix(ci): report a .yaml workflow as a naming finding, not a stray
GitHub reads .yml and .yaml alike, so WF001 telling you to move a valid
.yaml workflow to .github/scripts/ was wrong advice. WF001 now covers only
files that are not workflows at all, and the .yml spelling this directory
keeps moves to WF004, which says to rename rather than relocate.
WF001 also never looked into subdirectories, since GitHub does not read
them either; the message now says so. The directory is injected rather
than read off a module constant, so the cases are testable without
monkeypatching.
* test: merge three stranded twins into the files that shadow them
The second mirror's last four files each share a filename with a live test, so
the previous commit could not move them. Three of the four turn out to be plain
additions: their classes collide with nothing in the live file, so the tests are
extra coverage that has sat unrun rather than a competing version of anything.
Appending them takes the three files from 156 collected tests to 196, and all
196 pass. The 40 recovered are 13 OCI cases covering key normalization,
credential validation, complete-URL building and image-url transformation, 15
management-endpoint cases covering empty-value handling and the premium check,
and 12 DeepSeek thinking-parameter cases.
One assertion had to change. test_map_reasoning_effort_none_does_not_enable_thinking
asserted that reasoning_effort='none' leaves no thinking key, while the handler
maps it to {'type': 'disabled'} on purpose, documented in map_openai_params as
the OpenAI-style way to ask for thinking off. The test's stated intent holds,
since disabled does not enable anything, so it now asserts the disabled mapping
instead of the key's absence. Two imports moved to module scope for the
appended code, and no live test was touched.
test_discoverable_endpoints.py is the one left. Its twin grew from 1268 lines
to 9434, 25 of its assertions fail against today's code, and only 5 of its 19
tests have no counterpart, so deciding what survives that rewrite is a
judgement about the endpoints rather than a merge. The allowlist now holds
exactly that file and that reasoning.
* test(oci): stop the OCI suite reading credentials from the environment
validate_environment falls back to os.environ for every OCI credential and only
defaults the region when OCI_REGION is unset, so on a machine with OCI
configured the missing-credential test finds credentials it never passed and the
default-region test builds a URL for the ambient region. The suite then passes
or fails depending on who runs it.
A fixture drops the seven OCI variables for the four classes this branch added
and for TestOCIChatConfig, which had the same dependency before any of this and
fails the same way: with OCI_USER and friends exported, two of its cases fail on
origin/litellm_internal_staging today.
clean env: 83 passed
ambient OCI env: 83 passed
Same numbers either way, where the pre-existing file gave 68 passed / 2 failed
under the second.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
MCP egress prefixed the configured scheme unconditionally, but callers legitimately supply
both a bare token (from a stored credential) and an already-schemed value (passed through
from the caller's x-mcp-auth or Authorization header). The second shape produced
Authorization: Bearer Bearer <jwt>, which upstream servers reject as a malformed token. It
presented intermittently because a resolved stored credential arrives via extra_headers and
overwrites the doubled header, so only users without one always failed.
strip_auth_scheme drops one leading scheme before the header is rebuilt. It matches the
scheme case-insensitively per RFC 7235 and requires a credential behind it, so both a token
that merely begins with the scheme text and a scheme with nothing behind it are left intact.
MCPAuth.authorization stays verbatim because that auth type means the caller owns the whole
header value.
For MCPAuth.basic the normalization has to happen in update_auth_value rather than at
header-build time: to_basic_auth has already encoded the whole "Basic <credentials>" string
by then, so no prefix is left to find. A schemed value whose remainder decodes is already
encoded and is reused; one that does not decode is the bare pair with the scheme written in
front of it, and is encoded rather than forwarded as an invalid header.
The same doubling reached OpenAPI-backed servers through _format_byok_openapi_auth_header. A
non-BYOK server short-circuits _resolve_byok_mcp_auth_header, so that formatter also receives
the deprecated global x-mcp-auth, which is already a complete header value.
store_user_oauth_credential refused to overwrite any existing row that did not
decode as an OAuth2 payload, which conflated two states: a live BYOK secret that
reads back as plaintext, and ciphertext written under a LITELLM_SALT_KEY the proxy
no longer holds. The second is unrecoverable by any caller, so refusing preserved
nothing and instead wedged the user out of the OAuth flow permanently, since
re-authorizing is their only recovery.
The guard now raises only when the existing value is genuinely readable. An
undecryptable row is logged and replaced by the newly authorized token.
Both read paths were equally silent: get_user_oauth_credential and
list_user_oauth_credentials (which backs the bulk prefetch) each dropped an
undecryptable row indistinguishably from "user never authorized", so an operator
saw an upstream 401 and no hint that a credential had failed to decrypt. Both now
warn with the user and server ids, never the stored value.
A deployment that overrides any cost_per field, including at zero, now counts as priced so it is not blocked as unpriced
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
AKS workload identity injects AZURE_CLIENT_ID, AZURE_TENANT_ID, and
AZURE_FEDERATED_TOKEN_FILE into the pod, and never a client secret.
Reading that bare client id as a managed identity sent the pod to IMDS,
which has no identity attached to it, so the token request failed and the
federated token was never exchanged.
AZURE_FEDERATED_TOKEN_FILE now wins over the bare client id and infers
DefaultAzureCredential, whose chain reaches WorkloadIdentityCredential
before ManagedIdentityCredential. DefaultAzureCredential passes
AZURE_CLIENT_ID to both legs, so a plain user-assigned managed identity
still reaches the same identity it does today.
This is the credential path Azure recommends for passwordless Postgres on
AKS, and it also fixes the Azure OpenAI token provider, which infers its
credential the same way.
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the
test tree for names that do not exist. That matters more in tests than in
product code: a NameError inside a test whose body is wrapped in
`except Exception: pass` is swallowed, and the test reports green forever.
Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and
`make lint-ruff`, and clears every existing violation:
- 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only
on the failure path, so the NameError, not the assertion, is what ran.
test_llm_guard_error_raising is the worst: it passes today with content
safety disabled entirely. It now asserts the 400 and its detail body.
- 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still
supports 3.10. Guarded behind the exceptiongroup backport that anyio already
pulls in below 3.11.
- 9 missing imports (json, openai, Any, Final, HTTPException), including one in
a helper that catches HTTPException by a name it never imported, so the
challenge path it exists to detect raises NameError instead.
- 5 annotations naming types imported inside the function body, hoisted to
module scope or TYPE_CHECKING.
- 2 blocks of dead code: everything after a pytest.fail in
test_claude_agent_sdk, and an unused helper in test_end_users calling a
function defined in a different module.
- 1 error-path f-string in the router-settings doc test that masked the real
FileNotFoundError behind a NameError.
Only F821 for now. Widening the select list means ratcheting thousands of
pre-existing findings, so rules go in one at a time with their violations
already fixed.
* feat(proxy): redact or drop individual batch records instead of rejecting the file
A single record tripping a guardrail rejected the whole upload, which is unusable for a file
holding thousands of rows. A record a guardrail rewrites is now submitted in its rewritten
form, a record it blocks is left out, and the create response reports every changed record by
both custom_id and line so a caller can reconcile against the file it sent. The same outcome
is written to the proxy log and to request metadata, so it is not visible only to the caller.
A rewritten record goes straight to a spool and only its offset is carried, so a masking
guardrail touching most rows of a large upload does not build a second copy of the file on the
heap, and the rewrite runs off the event loop the way the sibling full-file validation does.
Both proxy-injected metadata keys are captured from the record and restored exactly, including
an explicit null, so a masked row keeps the tags that decide how it is attributed.
A record is dropped only when a guardrail judged its content. `GuardrailRaisedException` now
carries `blocked_content` for that, because half its raise sites in the repo signal an
unreachable or unparseable backend under a fail-closed policy, and treating those as blocks
would turn "refuse this request" into "drop this record and submit the rest". The default is
off, so a raise that does not say what it means aborts the upload instead of silently
shrinking the file.
* fix(proxy): only drop a batch record on a verdict the guardrail actually reached
A guardrail that reports a technical failure as an HTTPException carrying a block status was
read as a content block, so an unreachable backend under a fail-closed policy quietly shrank the
file instead of failing the upload. Two in-tree integrations do exactly that, and one of them
defaults to fail-closed, so the broken configuration was the default one. Such an exception is
raised `from` the underlying error, which is a deliberate statement that something else caused
it, and no content verdict in the repo is raised that way, so the chain now settles it. Implicit
context is left alone, since a block raised inside an unrelated `except` would read as a failure.
Two annotation errors in the same family: the one GuardrailRaisedException subclass in tree never
opted into blocked_content, so a real block took the whole upload down with it, and straiker's
block helper is reached both from its verdict and from its fail-closed handler, so it claimed a
verdict for an outage. The helper now takes the flag from its caller.
A record could also opt itself out of the chain. Guardrail selection reads a body-level
`guardrails` key ahead of the proxy-injected list, and online that key can only add to the key
and team selection, never replace it, so a batch record naming an empty list skipped every
guardrail that was not default_on and was still reported as scanned. Every injected key is now
stripped before dispatch and restored afterwards.
A guardrail that reroutes a record to another model is honoured on the online path by rewriting
the model, which the scan read as a rewrite and submitted in the same file, sending content to
the provider the reroute existed to avoid. Every record of a batch file goes to one provider, so
the upload is refused instead, naming the line.
The scan spool is closed on the paths that never read it back.
* fix(proxy): give the scan the metadata bag guardrails actually read, and close its spools
The narrowed request metadata was installed under `litellm_metadata` only, but a record is
scanned as the chat request it describes, and the guardrails that pick a policy from a request
header read `metadata` instead. Noma choosing an application and Aim choosing a user both look
there, so the header allowlist added for them did not reach either one and a batch record was
still evaluated under the fallback policy. The scan metadata now goes into both bags, which are
both stripped and restored, so neither survives into the record that ships.
The scan spool was closed on the paths that abort, which are exactly the paths where it is
empty, and left open on the one path where it holds the rewritten records. Nothing closed the
rewrite output either, where before this feature the uploaded handle belonged to Starlette. The
upload now owns both and closes them however it exits.
* fix(proxy): register the scan spool before the rewrite can fail
The scan spool was added to the request's cleanup list only after the rewrite returned, so a
rewrite that raised, which for a spilled file can be as ordinary as the disk filling up, jumped
to the handler with the list still empty and left the scan's own handle open. The rewrite also
left its half-written output behind on that path, since nothing owns that handle until it is
returned. Both now close.
completion() strips the litellm routing prefix before it dispatches to the
responses bridge, but responses() runs get_llm_provider() again, so a model id
that itself starts with the provider name lost a second prefix and reached the
provider as a name it does not know. Handing responses() the prefixed model
back makes its own resolve a no-op: across the 3061 cost map entries, 76 reach
the responses bridge and only the four perplexity Agent API models change.
