Key objects share the 200-entry UserApiKeyCache in-memory store with teams,
end users, tags and memberships, so churn in those objects evicts hot keys
and forces a LiteLLM_VerificationToken lookup on the next request. Route
bare hashed-token keys to a dedicated InMemoryCache inside UserApiKeyCache
while keeping Redis, TTL, serialization and invalidation shared
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The release image installs only the proxy extras, and psutil is a locust and mirakuru dev
dependency, so /debug/memory/summary answered with an error and no ram_usage_mb on the e2e
gate. Fall back to /proc/self/statm and /proc/meminfo on Linux when psutil cannot be imported
On the release gate the e2e tests only see the nginx router, and the chart's
ingress sent /debug/memory/summary to the backend catch-all, so the RSS check
measured the backend pod instead of the gateway workers that serve the failing
requests. Render it as an Exact gateway path next to /test, name the host in the
summary response so workers behind one origin never collide on pid alone, and
key the harness readings by (origin, hostname, pid)
The cascade zeroed end-user spend with a single update_many whose where
clause enumerated every dependent user id. Prisma compiles that IN-list
into one prepared statement carrying one bind variable per customer, and
PostgreSQL caps a statement at 32,767 of them. Once a shared budget had
more dependents than that the statement could not be parsed at all, so
the atomic cascade rolled back, budget_reset_at never advanced, and the
tier stayed due on every later tick forever. Customers sitting at their
cap were blocked indefinitely with only a recurring log line to show for
it.
End users now match on budget_id like every other gated table, plus a
NULL-budget_id branch for the implicitly created rows that carry no link
and ride the default tier. The statement's bind count now tracks the
number of expiring tiers rather than the customer population, so a reset
costs the same whether a budget has ten dependents or a million.
Fixes#40564
Claude-Session: https://claude.ai/code/session_01Hn5E8Jz1LjGLFyiYxBRcBW
* feat(rust_bridge): count budget-check input tokens in Rust on all LLM routes
Rust counts input tokens from the raw JSON body with the GIL released inside the existing budget reservation, covering every LLM route the auth dependency guards. It only fires for models on the Anthropic tokenizer when a budget is set, and Python counts whenever Rust is off, missing, or declines a body shape.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(rust): count byte-level BPE tokens without the GPT-2 split regex (#40594)
The oniguruma run of the ByteLevel pre-tokenizer regex is about 90% of
encode_fast on a 100k token body (100 ms of the ~110 ms Rust admission
count in the gateway pod). A hand-written scanner that yields the same
pieces, then feeds the model directly, counts the same text in 10 ms.
It only engages for tokenizers with the Anthropic shape (optional NFKC,
ByteLevel without prefix space, no post-processor) and falls back to the
full encoder when the text contains an added token. Parity with
encode_fast is tested on random texts, the pieces are compared with the
real pre-tokenizer, and the \p{L}/\p{N}/\s tables are checked against
oniguruma for every code point.
NFKC runs through unicode-normalization-alignments, the crate and
Unicode tables NormalizedString::nfkc already uses, so the fast path
normalizes exactly what the full encoder would. Using the newer
unicode-normalization crate changed the count for 171 code points that
gained compatibility decompositions after Unicode 9 (U+32FF, U+A7F1..).
The fast normalizer is compared with the tokenizer's for every scalar
value and on random texts.
The scanner is built without mutable state: byte_char and mapped_len replace the const table builders and the reusable mapped buffer, and iter::successors replaces the stateful piece iterator. byte_chars_match_the_byte_level_alphabet checks the byte mapping against ByteLevel for every scalar value.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rust_bridge): bound concurrent token-count encodes and share the Anthropic tokenizer predicate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Encode complete non-Claude source names and include source_model in the
Claude Code listing. Preserve configured route and alias precedence,
normalize once before model policy checks, and select CLI models using
explicit source identity instead of name stripping or positional joins.
Resolves LIT-7360
Claude-Session: https://claude.ai/code/session_01WyqeRhfZGm26zAnHx9P3kq
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* feat(guardrails): map each guardrail scan id to its guardrail, stage and provider
Adds the x-litellm-guardrail-scan-metadata response header, a JSON list of
{guardrail, stage, provider, scan_id} entries, next to the existing
comma-separated x-litellm-guardrail-scan-id header. Prisma AIRS records the
execution stage for every scan and OpenAI Moderation now records its
moderation id too. The new metadata key is internal: client-supplied values
are stripped and it is exposed through the UI CORS allow list.
Resolves LIT-6018
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): cap the scan metadata response header at a configurable length
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): hardcode the scan metadata header cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
error_status_code only read status_code, so a ProxyException raised
before routing (which stores its status as the string code) answered
500 with its 4xx type through the rerank, images, realtime, files, and
pass-through tails. It now falls back to a decimal code. A 408 maps to
timeout_error instead of invalid_request_error.
Tail regressions for rerank, images, realtime calls, and the chat
pass-through fail at the merge base with ('None', 'None'); the new
files-test helpers are fully typed.
* fix(proxy): load db credentials inside the model reconcile so a worker never serves a model before its credential
* fix(proxy): load db credentials in the model read-through so a request miss never adds a model before its credential
* fix(proxy): read credentials from the writer db before the router update and look a credential up once
* test(proxy): assert the credential is loaded when db models reach the router instead of the call order
An include entry that matches both a file next to the config that declares it and
one next to the root config now warns naming both, so a config that resolves to a
different file than it used to says so instead of quietly serving other models.
Also from reviewing that change:
- an empty root object in a bucket fails the boot again instead of coming up empty
- a YAML syntax error in a bucket object logs its own line naming the object
- an include already loaded is skipped before it is read rather than after
- reading a config out of GCS builds the plain bucket client, so it needs no
enterprise license and starts no flush loop that nothing ever cancels
Keep reading an include left beside the root config, with a warning naming where it
was found, so a nested include written against the old rule still boots.
Also build one S3 client per config load rather than one per included object, treat an
empty included object as an empty config instead of failing the boot, and point the
error a dropped bucket include raises at the bucket error logged with it.
Reading a config from a bucket ran a blocking boto3 GET straight from the
event loop for every object in the include tree, and on GCS it built a new
bucket client per object, each one starting a flush task that never ends.
S3 reads now go through a worker thread, and one bucket client serves the
whole include tree.
A config loaded from a GCS or S3 bucket skipped include processing entirely,
so every model, guardrail, and setting behind an `include` was silently
dropped. Both bucket types shared the same branch in `get_config`, which
never called `_process_includes`, and that helper only ever read from disk.
The merge now lives in one async helper that takes the loader as a
dependency, so disk and bucket configs share the same semantics: list values
extend, everything else overrides, nested includes are followed, and the
`include` key is stripped. Bucket entries resolve as object keys relative to
the config object's prefix, with a leading `/` meaning the bucket root, and
an include that cannot be read now raises instead of being skipped.
_numeric_form_type only peeled a single ReadOnly layer, so a field still
wrapped in Required/NotRequired was read as non-numeric and dropped from the
mapping. Which qualifiers survive get_type_hints varies by interpreter version
and by include_extras, so on Python 3.10 NotRequired[ReadOnly[int]] reached the
check intact and the field was silently skipped, which is what turns the mapped
test red on the 3.10 leg only.
Peel Required/NotRequired/ReadOnly/Annotated in any order and nesting instead.
The one production caller feeds a schema with no qualifiers, so the resulting
mapping is unchanged on every interpreter in the matrix, but a field written the
house-convention way stops being dropped.
The proxy's exception tails defaulted `type` and `param` to the four-character
string "None", which is neither a known OpenAI error type nor the JSON null the
nullable `param` field is typed as, so a client's error handler matched nothing
and fell into its generic branch.
Lifts the helpers PR #39521 added for the unified LLM endpoints into
litellm/proxy/common_utils/openai_error_payload.py and calls them from the file,
rerank, image, realtime, anthropic, and pass-through route families, plus the
shared handle_exception_on_proxy handler that the management, batches,
fine-tuning, credential, SCIM, guardrail, and customer routes funnel through.
The remaining families (proxy_server, auth, health, spend tracking, and
management endpoints) follow in separate PRs so each slice stays QA'able on a
live proxy.
Every field of a multipart form arrives as a string, so `n` reached the
provider as "2" and Bedrock Nova Canvas rejected the request with
"expected type: Number, found: String". Restore the type the request
schema declares at the boundary where the form is parsed, driven by the
schema's own type hints so the helper covers any int- or float-typed
field on any multipart endpoint.
A model access group could gate which models a caller reaches but never how
much that group of callers could spend in total. Capping a shared pool meant
setting a per-entity budget on every key by hand, which caps each key
separately and still leaves no way to read what the group cost.
Spend is attributed to a group only when the group's name appears on an
allowlist the caller was granted (key, team, team-member scope, project or
org) and that group serves the requested model. Asking for a model that
merely belongs to a group attributes nothing, because nothing about the
caller named the group. Levels are unioned rather than ranked, so a team
granted "*" whose member is scoped to one group still counts as gated by
that group.
Enforcement runs on both paths tags already use: a reservation counter on
the pre-call path and a read-time max_budget check inside the existing
concurrent budget gather, so the ceiling still holds under
disable_budget_reservation.
Adds LiteLLM_ModelAccessGroupBudgetTable, which is the only place a group is
ever a row: the groups themselves stay free-text strings in
model_info.access_groups, so a row exists only once someone gives that group
a budget. GET, PUT and DELETE /access_group/{name}/budget manage it, and
/access_group/{name}/info now carries the spend and budget alongside the
models.
* feat(proxy): opt-in budget rollover carrying overage into the next window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): zero under-cap rows before decrementing over-cap rows in cascade resets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(langfuse): support langfuse_environment as a per-key dynamic callback param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse): type the langfuse_environment constructor param
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse): only pass environment when the SDK client supports it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse): drop the request-body metadata test for langfuse_environment
The proxy bans request-body callback params by default (derived from
_supported_callback_params in auth_utils), so the metadata channel this
test asserted is rejected with a 401 on the proxy. The supported channel
is admin-set key/team callback_vars, with LANGFUSE_TRACING_ENVIRONMENT
as the deployment-wide fallback.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(langfuse): validate langfuse_environment, avoid redundant clients, honor it in langfuse_otel
Closes the review gaps on the langfuse_environment param:
- Validate values against Langfuse's environment pattern at save time
(/key/generate, /key/update, /team callback all 400 on e.g. 'Production'
instead of 200-then-silently-dropping every trace server-side) and at
logger init; non-string values are str()-coerced instead of crashing
the SDK's regex check per event.
- Treat empty/whitespace values and values equal to the deployment-wide
LANGFUSE_TRACING_ENVIRONMENT as non-dynamic so an environment-only
override that changes nothing no longer mints a duplicate SDK client
against MAX_LANGFUSE_INITIALIZED_CLIENTS.
- langfuse_otel now reads the per-key/team langfuse_environment from
standard_callback_dynamic_params instead of only the env var.
- Advertise the param on the discovery surfaces: callback_configs.json
(langfuse + langfuse_otel), the dashboard callback registry, and the
/team/{team_id}/callback docstring (schema.d.ts regenerated).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* style: ruff format langfuse files
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(lint): remove duplicate test import, LIT002 dict literal, and mock-echo otel test
- drop redundant in-function import of callback_config_error (F811)
- avoid the `or {}` mutable literal in _set_langfuse_specific_attributes (LIT002)
- rewrite the dynamic-env otel test to observe span.set_attribute output
instead of patching litellm internals (TQ002/TQ008)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <yucheng@berri.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(proxy): reset a stuck team member's budget
A per-team-member budget check reads a cross-pod spend counter that
nothing ever invalidates. Once a member exceeds their per-member
budget, resetting the key's spend, raising the user's or the team's
own budget, or issuing a new key all leave the member stuck, because
none of them touch this counter or its cached membership object.
Add POST /team/{team_id}/member/{user_id}/reset_spend to reset a
member's tracked spend, and invalidate the same cached state from
/team/member_update when it raises a member's own budget, so that
path also takes effect immediately instead of waiting on the
membership cache's TTL. Name the entity in the check's error message
so a stuck member is diagnosable from the 429 body alone.
* fix(proxy): close reset-vs-floor-read race and surface double Redis write failure on member spend reset
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast spend reset as a SET so the handler's self-delivered message cannot erase the reset guard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit null fields from the invalidation message so plain evictions keep the old wire format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every repository handed its `.table` back untyped, so a dozen modules had
each grown a private `_PrismaTableActions` Protocol to paper over it. They
had drifted: some declared `update` as returning the row, others the row or
None, and none agreed on whether `find_many` was covariant
Replace all of them with a single `TableActions[RowT_co]` in
`litellm/repositories/prisma_protocols.py`, keyed to the prisma row each
repository is bound to. Query inputs stay `Mapping[str, object]` so callers
keep passing plain dicts, and `find_many` returns `Sequence` so the row type
stays covariant
Typing the nullable returns honestly surfaced paths that were already
crashing. A team admin could never edit or delete a memory entry owned by
their team: the write-auth check fed a raw prisma row to a helper that
expects the domain model, so `members_with_roles` arrived as plain dicts and
the request died as a 500 instead of applying the edit. Non-admin members hit
the same 500 in place of the 403 they were owed, so refusal and breakage were
indistinguishable. `/v2/model/info?user_models_only=true` dereferenced a
missing user row rather than returning the 400 the route already had, three
team routes dereferenced a team deleted between the read and the write, and
the agent registry dereferenced a missing agent instead of naming it
basedpyright drops 2,132 errors, 1,454 of them reportAny and 73
reportExplicitAny. The dashboard's generated types pick up `string[]` where
they had `unknown[]` for a team's members, admins and models
Type the annotations that landed in the last 24 hours and ratchet the lint budgets down accordingly. No behavior change.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
get_form_data collapsed the FormData multidict with dict(form) before the loop
that rebuilds `foo[]` arrays ever ran, so a request sending
timestamp_granularities[]=word and timestamp_granularities[]=segment reached the
provider as ["segment"] with the first value silently dropped. Read the multidict
with multi_items() instead.
The test could not catch it because its mock was a plain dict carrying the same
key twice, which Python collapses exactly the way the bug did. Every request.form
mock that fed get_form_data now returns real FormData.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
Every pod schedules the budget reset job, so a fleet re-read the whole due
population and wrote it back against one Postgres at the same calendar
boundary, multiplying a single sweep by its replica count. The job now takes
the shared PodLockManager lease, so one pod sweeps per tick. A deployment with
no Redis keeps its previous behavior, and a Redis that cannot answer sweeps
unguarded rather than stranding every expired budget at its cap.
The per-window scan read every row carrying budget_limits in one statement, so
its cost grew with the deployment's key count. It is now keyset-paginated and
walks to the end of the table on every sweep. A per-run cap would need a resume
position, and no pod can hold one because the lease rotates between ticks, so
the strictly advancing cursor is what terminates the walk.
Found and updated rows were also JSON-serialized into the service hook's
metadata and into debug lines on every chunk, on the event loop, whether or not
anything consumed them. The hooks now carry counts, and the debug payload is
deferred until a record is actually emitted.
Resolves LIT-4793
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A dropped connection anywhere in the budget reset tick used to abort the whole
phase, so every due key, user, team and budget tier stayed unreset until the
next tick ten minutes later. Route the job's DB calls through
call_with_db_reconnect_retry so a transport blip costs one reconnect instead.
Reads replay on any transport error, since re-running a SELECT has nothing to
double-apply. Writes are non-idempotent, a reset assigns spend = 0
unconditionally, so they narrow to DB_RETRY_SAFE_ERROR_TYPES: only a
ConnectError proves the statements never reached the database. A post-send
error like ReadError or ReadTimeout leaves the commit outcome unknown, and
replaying one that already landed would erase whatever was spent since, so
those keep the pre-existing behaviour of failing the tick.
Resolves LIT-5372
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
