A body model_id picks what runs upstream for Bedrock (model or inference
profile), SageMaker (inference component) and Vertex (private endpoint),
while auth, logs and spend see the deployment's model. Ban it on LLM routes
for every provider, unlocked only by configurable_clientside_auth_params on
each deployment the request can reach. The proxy-wide
allow_client_side_credentials flag no longer unlocks it.
metadata.model_id stays allowed since nothing reads it from metadata.
Admin routes keep model_id as a deployment id.
* fix(bedrock): answer 400 for service tiers Bedrock cannot serve, forward reserved
Converse used to drop any service_tier outside default, flex, and priority
with no error, rewrite auto to default, and drop reserved. Now reserved and
standard_only map to Bedrock tiers, auto sends no tier, and any other value
answers 400 unless drop_params is on. Nova batch rows honor the row's
drop_params.
* fix(bedrock): keep a Responses batch row's drop_params and keep it out of passthrough rows
* test(bedrock): annotate service tier test locals as Final and drop the row helper's kwargs
* fix(bedrock): refuse a service tier on Nova batch rows unless drop_params, and honor the deployment's drop_params
* fix(bedrock): map standard and any-case service tiers, and drop tiers from Claude batch rows under the deployment's drop_params
* fix(bedrock): forward the tiers OpenAI-format batch rows take, accept auto on Claude rows, and label tiers in lowercase
* fix(bedrock): mark the service tier block field read-only
* fix(bedrock): read drop_params given as a string on batch rows, and cover the Nova row mapper's drop_params
* test(prometheus): sort the service tier label test imports
* fix(bedrock): send the Invoke route's service tier as a header and name the accepted tiers in the 400
* fix(bedrock): report and bill the tier Bedrock served on Invoke calls and streams, and send the tier header only to Nova
* fix(bedrock): return copies when stamping the served tier, read its header without re-encoding the others, and pin it per stream chunk
The Nova Invoke mapper and the event-stream decoder now return model_copy results instead of writing service_tier in place. One shared reader finds the X-Amzn-Bedrock-Service-Tier response header case-insensitively, so a non-ASCII request id no longer breaks the stream decoder, and the passthrough uses the same reader. Tests now vary the served tier in both directions, check every chunk, and cover a stream with no tier header or an empty one
* test(bedrock): check the served tier on a stream whose headers come as a plain dict with an upper-case name
The non-ASCII request id stream test now also carries X-AMZN-BEDROCK-SERVICE-TIER in a plain dict and asserts every chunk reports that tier, so a case-sensitive header reader or a lookup by the canonical name fails it
* refactor(bedrock): drop the passthrough's own service tier header reader that main re-added
The passthrough already reads the served tier through the shared case-insensitive reader in common_utils, so the httpx.Headers copy main brought back with the embeddings change has no callers and references a constant this branch removed
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): keep carried end-user spend when a budget reset is retried
End-user pages commit before the budget window advance. When the advance
failed, the tier stayed due and the next tick replayed every page, so a
rollover-capped row already cut to its carryover (15 -> 5) matched the
under-cap zero and lost it, and spend accrued since the first pass was
zeroed again. Each end-user reset now stamps the tier window it closes
in a new spend_reset_window column and skips rows already stamped with it.
Fixes#50898
* fix(proxy): leave never-reset tiers unguarded and test each tier's own window
A tier with no budget_reset_at falls back to the old unguarded reset
instead of stamping created_at, which an admin clearing the reset time
would reuse and silently skip end users. Tests now give tiers distinct
windows and cover a second window closing.
* test(proxy): give the end-user ledger its own fake prisma client
* fix(proxy): pin a reset time on tiers without one before end-user pages commit
* fix(proxy): read the reset-time pin from the writer and skip tiers another sweep advanced
* fix(proxy): keep linked rows of a tier another sweep advanced out of the rollover reset
* fix(proxy): order the end-user reset window migration after upstream's same-timestamp migration
* fix(proxy): reset end users again when an admin moves a tier's reset time back to a closed window
* fix(proxy): clear end-user reset stamps in the same transaction as the tier's reset time
A failed /budget/update used to leave the new reset time written with the
stamps still set, so repeating the request found the time unchanged and
skipped the clear. The update and the clear now commit together, and a
reset time in the future no longer scans the tier's end users
* fix(proxy): name each end-user reset window by the tier's last write
Stamp end-user resets with the tier's updated_at read on the writer instead
of its reset time. A failed window advance leaves updated_at as it was, so the
retry still skips the end users already reset, while the advance or any
/budget/update moves it and opens a new window. That covers a forced reset on
the default end-user tier and drops the stamp clear, so /budget/update is the
one-row update it is on main. The writer read also leaves out a tier another
sweep already advanced, which a lagging read replica still showed as due
* fix(proxy): name end-user reset windows by the tier and its advance count
Tiers advanced in one batch share updated_at, so a customer moved between
them could skip the new tier's reset, and any unrelated tier edit during a
failed advance reopened the window. A budget_reset_count that only the
window advance bumps, prefixed with the tier id, names the window instead.
* fix(proxy): keep carried spend for customer teams and members on a retried reset
Customer team and member pages commit before the window advance like end-user
pages, so they now stamp the window they close and a retry skips them too. A
member runs on its unexpired override when that has its own duration, else on
its template.
The window name now joins the tier id, its creation time, its count of closed
windows and its reset time, so a duration change or a tier recreated under the
same id opens a new window even when the advance failed.
* test(integration): list every budget column and relation in the budget update field check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(mcp): check the tool dict conversion without patching the MCP handler
* refactor(mcp): let the semantic filter take its MCP tool loader as a parameter
* refactor(mcp): type the semantic filter's default tool loader
Annotate the chosen loader as _OpenAIToolLoader and narrow its return to
FunctionToolParam, so a default that returns raw MCP tools fails the type
check. Drop the unused patch import.
* fix(logging): keep a content-block system prompt in logged messages
A /v1/messages request whose top-level system prompt is a list of content
blocks never reached the logged messages, because only a plain-text system
prompt was prepended. Accept a non-empty list too, so every logger that reads
messages sees it. Claude on Vertex also normalizes a plain-text system prompt
into blocks, so it was dropped there as well.
Fixes#36402
* fix(logging): type the logged system message
* fix(logging): type the system message without a mutable annotation, close the test client
* fix(proxy): keep x-litellm-spend-logs-metadata on auth-rejected failure spend logs
Budget, invalid-key and other auth-stage rejections are raised before the
request headers are merged into metadata, so the persisted failure row lost
the caller's spend logs metadata header. The auth failure handler now parses
it with the same pre-call helper and stamps it into the metadata bucket it
already uses for the requester IP and user agent.
* fix(proxy): type the spend logs metadata header parser honestly
The parser takes the request headers as a str mapping and returns the parsed
JSON as object, which is what successful requests already store. The auth
failure path keeps only JSON objects. The precedence regression now also
asserts the caller's request body is left untouched.
* fix(proxy): stamp the auth failure spend logs metadata into the route's bucket
A chat request whose body also carried litellm_metadata got the header in
litellm_metadata, which the failure spend row never reads on chat routes.
The header now goes into the bucket picked by metadata_variable_name_for_request
for the route, and the failure row lifts spend_logs_metadata from that bucket
the same way it already lifts the requester IP and user agent. The regression
tests now assert on the built failure spend row.
* fix(proxy): keep the auth failure spend logs metadata fix inside the auth handler
Restores the failure spend row's lifted fields to main, so failures after auth
on litellm_metadata routes keep their current logging. The auth handler now
writes the parsed header into metadata, which the failure spend row reads on
every route, and into the route's own bucket for callbacks on litellm_metadata
routes.
* fix(proxy): make the spend logs metadata header field read-only and test the real spend writer
The header collector now sets spend_logs_metadata when it is built, so the
field can be ReadOnly with the same stored value. The auth failure tests run
the real DBSpendUpdateWriter and read the queued spend log row from a fake
database client instead of an autospec writer.
* test(proxy): read auth failure spend rows from a real unconnected PrismaClient
Replaces the mocked database client and the class-level queue byte reset.
The test now enqueues into a real PrismaClient's own queue and dequeues
its row in cleanup, which keeps the shared byte counter balanced.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(model-prices): drop Flex prices from Vertex AI Gemini 4 Argon and Hydrogen
* test(cost-calc): stop comparing Flex prices between Vertex AI and Gemini API Gemini 4 Argon
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
A cost map with a top-level value that is not an object made a reload answer 500
after the new table was already published, so one worker billed new prices while
the others kept the old ones. The document is now refused before anything changes,
which also lets boot fall back to the bundled map instead of failing the import.
Also restores behavior the Rust cost map move changed for SDK callers:
- a document that starts with a UTF-8 or UTF-16 byte order mark loads again
- an alias_of row that carries other fields is served as its target
- register_model(url) fetches nothing when LITELLM_OFFLINE_MODE is set
- register_model(None) is a no-op again
- the public names RESERVED_TOP_LEVEL_KEYS, FALLBACK_GENERALIZATIONS_KEY,
RETRYABLE_FETCH_STATUS_CODES and git_blob_id import again, and so does
utils._runtime_registered_model_cost, which downstream test code replaces
litellm_core_utils/AGENTS.md now says which removed names come back.
* fix(helm): route every gateway allowlist path to the gateway in the componentized ingress
The ingress gateway prefix list had drifted from gateway/routes/allowlist.py, so requests for decisions, systemone, extraction, traces, logs, live sessions and several provider passthroughs fell into the backend catch-all, and /eu_jev/v1/systemone returned 404. The missing paths are added, the exact gateway paths move into a list, and a unit test now fails whenever the chart and the allowlist disagree in either direction
* fix(helm): keep extraPaths that repeat a built-in gateway path rendering on upgrade
Values written for an older chart add the newly built-in gateway prefixes through ingress.extraPaths with service gateway, which the duplicate check now rejected, so helm upgrade failed to render. Such an entry is skipped, and a built-in path pointed at another Service or port is still rejected. Also asserts that the new exact path and a new prefix render to the gateway Service
* revert(helm): drop the extraPaths skip for built-in gateway paths
Keeps this PR to the missing paths and the drift test. An extraPaths entry that repeats a now built-in gateway path is rejected at render time as before, and operators remove it on upgrade. The checks that the new paths render to the gateway Service stay
* fix(helm): skip an ingress.extraPaths entry identical to a built-in route
Values that added a path through extraPaths before the chart routed it stopped rendering once the path became built in, since the duplicate check rejected every repeat. A repeat with the same path, pathType, Service and port (by number or the http port name) now renders nothing, and a repeat that routes elsewhere is still rejected
* refactor(helm): limit the ingress fix to gateway-only routes and the drift test
Drops the extraPaths identical-route skip, so the duplicate check is the same as on main. /v1/traces and /v1/logs leave the gateway list and keep going to the backend, since the backend also serves them and their gateway entries date from the retired OTLP ingest. The drift test now requires only paths that the gateway alone keeps
* test(helm): require every gateway exact path the backend does not list exactly
A backend prefix such as /debug/ made /debug/memory/summary count as shared, so the drift test stopped requiring its Exact gateway rule. Exact paths now count as shared only when the backend lists the same exact path
* chore(helm): keep the gateway prefix comment as it is on main
* fix(helm): fit the componentized ingress under the default ALB rule quota
The AWS Load Balancer Controller makes one listener rule per Ingress path and an ALB allows 100 rules by default, so the 113 single-host paths would fail with TooManyRules. Gateway prefixes are now listed in pairs, and on alb each pair is one rule, its first prefix as the Ingress path with a use-annotation forward to the gateway and the second added through a conditions annotation. The default alb render drops to 62 rules for the same path patterns, and nginx, monolith and component-host renders keep one path per prefix
* fix(helm): order grouped ALB rules by their shorter prefix and keep gateway conditions
Each gateway prefix pair now lists its shorter prefix first and uses it as the Ingress path, so a longer extraPaths child of either prefix still sorts ahead of the grouped rule. Conditions an operator set for the gateway Service are carried onto every grouped rule, which uses a generated backend name. The grouped annotations and paths are written inline from the pair list
* fix(proxy): keep a customer budget reset from being undone by a stale hierarchy cache entry
The reset job now evicts the customer_org_alias entries of the organizations it resets, and a
hierarchy load that began before an eviction can no longer cache what it read. The cache hands
a lease out on a miss and refuses the write when the key was evicted since, on this worker and,
with enable_redis_auth_cache, on any worker through a generation key checked in Redis.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop auth cache memory before the generation write and skip an empty alias eviction
delete cleared memory only after its new Redis round trip, so a deleted entry stayed readable on
the handling worker while Redis was slow. The reset job also broadcast an empty alias eviction for
a cascade with no customer organizations.
* fix(proxy): delete an auth cache entry and void its leases in one script on the bulk path too
The bulk delete wrote the generation and sent the DEL as two round trips, so a failed generation
write still let the DEL run and an in-flight load could re-cache what it read.
* fix(proxy): delete the auth cache entry before the generation write inside the script
A Redis at its memory limit with noeviction refused the script at its leading SET, so the DEL
never ran and a deleted entry stayed in Redis. With the DEL first the script still deletes it.
* test(proxy): mark the new reset cache tests' bindings Final and use rstest for the new Rust test
* fix(proxy): refuse a leased auth cache write for a key no generation key can guard
* test(proxy): undo the unrelated reformat of the spend ledger parity tests
* test(redis-store): mark the updated memory-only auth cache test with rstest
* chore(proxy): drop two pointer comments from the cache lease code
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: inherit team models for dashboard sessions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: authorize dashboard models from owner grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve dashboard model listing filters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): attribute dashboard team model spend to granting teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Keep dashboard fallbacks on the original billing team
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Preserve admitted dashboard billing through primary model rewrites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Reject dashboard batches that need team model grants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Respect wildcard exclusions in dashboard model listings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): log Titan embedding replies on the invoke passthrough as embeddings
A native /bedrock/model/<titan-embed>/invoke call returned its vector, but success logging read the reply as a chat completion and crashed, so the call was never priced or written to spend logs. Fixes#51455
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): log binary Titan embeddings and bill input images on /invoke passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): charge text tokens alongside the image on Titan multimodal /invoke passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): add the 4K output rate to runwayml/seedance2_5
* test(cost-map): cap per-second, per-image and per-request prices at 1000 and drop the seedance exceptions
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: configure Aria access for internal users
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: keep Aria session access within complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: match Aria role helper to authorized role type
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: type Aria settings and allow permitted team selection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The cache copied Python's lazy expiry heap. Deleting or re-arming a key
left a stale heap record, and a longer-lived key at the top kept those
records in memory without bound. A TTL past the end of Duration also
panicked while the lock was held, which poisoned it for every later call
The cache now keeps one index record per entry in a BTreeSet and removes
it with the entry, so there is nothing to compact. A write picks
Expiry::KeepExisting or Expiry::Refresh, update() reads and writes under
one lock, and deadlines saturate
The Redis stores' memory tier uses update() and drops its second mutex.
RouterCooldowns no longer panics on a cooldown too long to measure, and
both crates lock with parking_lot
* fix: eagerly build usage schemas before parallel requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: tolerate upstream schema rebuild changes in race probe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Move usage schema race regression to SDK integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(types): move the Usage schema race regression into the unit suite
The probe needs no proxy or upstream, so it belongs with the other Usage tests
and runs on every pull request. It now builds each model with real token counts,
asserts them in the dump, and falls back to a plain start barrier when pydantic's
model_rebuild cannot be inspected.
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add customer organization budget groups tab
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add cap policy controls to customer budget groups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Clarify per-group cap policy help text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Fix customer budget groups formatting and lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(customer-portal): save budget group configuration atomically
* refactor(customer-portal): name unchanged budget group state
* fix(customer-portal): chunk removed budget reference checks
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The three params were dropped without an error on the online path and in Vertex batch rows,
and a PDF embedding reported 0 tokens because its DOCUMENT token count failed usage validation.
They now go in embedContentConfig online and in embed_content_config inside each batch row's
request, and DOCUMENT tokens are counted at the text rate.
Resolves LIT-10384
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(sdk): stop requiring redis to import the Messages streaming path
redis_update_buffer.py and dynamic_rate_limiter_v3.py imported RedisError at module
level. Both sit on the import chain that the SDK /v1/messages stream loads, so an SDK
install without redis lost every streaming success callback. Import RedisError only
where a Redis call can actually raise it.
* style(tests): space the optional redis import fixture per ruff format
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(router): stop a Responses fallback from replaying a delivered tool call
A tool-only Responses stream carries no output text, so a retriable error or
response.failed after a completed function call was treated as a failure before
output and the fallback replayed the whole request, running the tool twice.
Check for a started tool call before the before-output shortcut so the primary
error surfaces instead. Pre-output fallback and text continuation are unchanged.
* test(router): read the delivered tool call events one at a time
Consume the created and output_item.done events with anext and expect the
primary error on the next read, so the raises block holds a single statement.
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): drop a customer override expiry sent without an override
* test(proxy): clear the override before deleting the row so the integration test leaves no budget behind
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): report PDF tokens and cost on Gemini embedContent responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): mark PromptTokensDetails fields ReadOnly and type the new usage tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
