temp_budget_increase was only applied on the DB-fetch path of _user_api_key_auth_builder, so a key served from the auth cache reverted to its original max_budget and was wrongly blocked with BudgetExceededError once spend crossed the original budget while staying under the effective budget.
Move _update_key_budget_with_temp_budget_increase out of the DB-only branch so it runs for every resolved token regardless of source. The cache stores the original budget and each cache hit returns a fresh model_copy(), so this never double-applies.
Fixes#25760
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The JWT first-login upsert in get_user_object creates the user row by
merging default_internal_user_params straight into table.create, so a
configured budget_duration landed with budget_reset_at NULL. The reset
sweep now heals such rows (PR #33623), but until the next sweep the row
shows a null reset time and its first window starts at the sweep instead
of one full duration after creation. Compute budget_reset_at at creation
like every other write path (/user/new, UI SSO, /key/generate, /team/new)
already does
Continues #29762. Response models (Message, Choices, Usage) delete unset
optional fields in __init__ so model_dump matches the OpenAI spec. Each delete
routed through pydantic's BaseModel.__delattr__, whose per-call
ModelMetaclass.__getattr__ lookup and _check_frozen dominate construction. When
the target is a declared field already present in __dict__ on a non-frozen
model, delete it with object.__delattr__ directly; that is exactly what pydantic
2.13 does for that case, minus the metaclass getattr and the frozen check. It
falls back to the previous super().__delattr__ path for extras, private
attributes, cached properties and missing names, so behavior is unchanged.
Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
Internal users seeded from default_internal_user_params (SSO/JWT first-login
upsert, or /user/new without an explicit budget_reset_at) get budget_duration
set but budget_reset_at = NULL. The ResetBudgetJob user/team queries filter on
{"budget_reset_at": {"lt": now}}, which never matches NULL, so these rows are
never reset: their spend accumulates for the lifetime of the row and silently
exceeds max_budget with no periodic reset.
The budget-table query already handles this by OR-ing in a
{budget_reset_at IS NULL AND budget_duration IS NOT NULL} branch. Apply the
same pattern to the user and team reset queries in PrismaClient.get_data.
Adds a regression test asserting both the user and team reset queries select
NULL-budget_reset_at rows that have a budget_duration.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
Continues #29761. Delta.__init__ set roughly ten attributes through pydantic's
__setattr__ and then deleted the five OpenAI omits on every chunk. Those keys
are extra fields (extra='allow'), so this builds __pydantic_extra__ and
__pydantic_fields_set__ directly after the parent init instead of round-tripping
each field through __setattr__/__delattr__. The resulting __dict__,
__pydantic_extra__, __pydantic_fields_set__ and model_dump output (including
exclude_unset, which the streaming path relies on) are byte-identical to the
previous behavior; a serialization-contract test locks that. A TYPE_CHECKING
block re-declares the extra attributes with their concrete types so type
checkers still see delta.content and friends.
Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
Staging now contains #33153, whose final rounds made _extract_upstream_auth_failure a thin delegate
to upstream_auth_challenge and introduced the response-level iterator this branch predates. The
resolution completes the consolidation both branches were converging on: iter_exception_tree
(faults/traversal.py) is the one tree walk, _iter_upstream_responses is rebuilt on top of it instead
of carrying a second copy of the traversal, the manager keeps the delegate, and the semantic filter
port from this branch stands. Test conflicts were append-append and both sides are kept
* fix(cache): make in-memory and disk increments atomic
* refactor(cache): narrow in-memory increment lock scope
* fix(cache): address follow-up review on increment tests/types
* fix(cache): refresh atomic increment coverage
* test(cache): widen increment race window with non-zero _SlowInt seed
The zero seed was falsy, so InMemoryCache.increment_cache's `get_cache(...) or 0`
and DiskCache.get_cache's truthiness guard both discarded the _SlowInt before
__add__ could run, leaving the sleep-based window-widening inert. Seed a non-zero
value and return _SlowInt from __add__ so the sleep fires on every read-modify-write
in both backends, making the concurrency regression deterministic.
* test(cache): cover InMemoryCache.async_increment delegation
Add a focused async test asserting async_increment accumulates through the
locked sync path, exercising the previously uncovered delegation line.
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Caller-supplied bedrock_tags land as AWS resource tags under the proxy's
AWS identity, letting an authenticated caller forge ownership or
cost-allocation labels. Add bedrock_tags to _BANNED_REQUEST_BODY_PARAMS
so per-request tags need general_settings.allow_client_side_credentials
or configurable_clientside_auth_params on the deployment, matching the
aws_bedrock_project_id precedent. Deployment-level bedrock_tags in
litellm_params are unaffected.
Also stop an explicit empty bedrock_tags list in litellm_params from
falling through to optional_params
The model_info / get_model_info_with_id endpoint tests drove refactored
endpoints with bare, unspec'd MagicMock routers and models. Because the
mocks were unspec'd, any attribute or method the (refactored) endpoints
newly read auto-materialized a child MagicMock, and whether that child
was reached depended on process-global state (premium_user, and the real
get_available_models_for_user chain reading litellm globals) that sibling
tests in the same xdist worker mutate. When reached, the MagicMock either
unpacked to empty (a, b = mock.method() -> 'not enough values to unpack
(expected 2, got 0)') or leaked into RouterModelInfo(**model_info) and
failed Pydantic str validation. Pass in isolation, fail under xdist.
The original TestModelInfoEndpoint failure (#33807 CI) was the same class
surfaced by merge skew: #33721 added a get_configured_token_limits unpack
to create_model_info_response, and CI's merge commit ran that against the
un-updated bare-mock test before the #33742 band-aid landed.
Fix (test-only, no product change):
- TestModelInfoEndpoint: mock the real seam (get_available_models_for_user),
configure the router methods the endpoint actually calls, return a real
Deployment, and drop the dead proxy_server.get_key_models/get_team_models/
get_complete_model_list patches the refactor had stranded.
- TestGetModelInfoWithIdBlocked: spec the model mock so unset enterprise
columns read as None instead of child MagicMocks.
- test_ProxyConfig_get_model_info_with_id_missing_model_id_raises: pin
premium_user so the asserted AttributeError no longer flips with the
ambient license global.
_load_persisted_master_key documents leniency on any unreadable prior
state but only caught parse errors; a permissions failure or non-UTF-8
bytes in config.yaml crashed configure instead of skipping the
carry-forward. Catch OSError and UnicodeDecodeError too and pin the
undecodable-file case with a regression test.
lite autoroute up minted a fresh master key and picked a fresh OS-ephemeral
port on every run, so any client configured against one session (an
already-open Claude Code session, a hand-configured script) broke on the
next run. The port is now a stable default (5483, overridable with --port)
that refuses loudly when busy or when 4000 is requested, since proxy_cli
silently rebinds a busy 4000 to a random port. The master key is minted
once, persisted in the generated config.yaml, reused by every later up,
and carried forward when configure regenerates the config.
_resolve_v2_auth dropped the resolved client_credentials auth when extra_headers
already carried Authorization (MCPJWTSigner, static_headers), so the upstream got
the injected header instead of the minted token and the one-shot 401 refetch was
lost. M2M now joins token_exchange and authorization_code in the authoritative
set; the conflicting header is dropped
The auth object is the httpx client's auth for the whole MCP session; after a
401 recovery it kept sending the rejected token first, burning a 401 round trip
and the single retry on every subsequent call
An expires_in of zero or below computes a ttl of 0; the entry could never be
served but still occupied a slot in the bounded backend, where it could evict
a live token. The mint still serves the current request and the next get
re-fetches under the per-server lock
Greptile P2s: the per-server lock dict now evicts its oldest entry past
max_locks so ephemeral server ids (REST tools preview) cannot grow it
unbounded, and the min-cache floor is capped at the token's actual lifetime so
an expires_in below the skew is never served past expiry
The graft test pinned the pre-migration contract (M2M defers to v1). Replaced
with two tests pinning the new one: a complete-config M2M server resolves via
the v2 arm into ClientCredentialsBearerAuth, and an incomplete-config server
fails closed with a 500 misconfigured naming the missing grant fields
Replaces the not_implemented stub with a live arm: ClientCredentialsTokenSource
mints and caches the M2M token (rotation-aware identity key, expires_in-driven
TTL, audience and token_endpoint_auth_method support) and
ClientCredentialsBearerAuth retries an upstream 401 exactly once with a freshly
minted token. to_server_spec owns oauth2_flow=client_credentials servers and
fails closed on incomplete grant config instead of connecting unauthenticated
Clients exporting large spend-log ranges were forced into 100-row pages,
which meant a bounded COUNT plus an increasingly deep OFFSET scan per
request. Larger pages reduce both the request count and the cumulative
OFFSET cost for the same result set.
The handler already excludes the heavy JSON columns (messages, response,
proxy_server_request) from the paginated SELECT and bounds the COUNT via
SPEND_LOGS_PAGINATION_COUNT_CAP, so per-row cost does not grow with page
size. 1000 matches the ceiling already used by the user and user-agent
analytics list endpoints.
* fix(fireworks_ai): set Content-Type application/json in validate_environment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(fireworks_ai): delegate chat validate_environment to OpenAIGPTConfig
Instead of re-adding the JSON Content-Type default inside FireworksAIMixin,
FireworksAIConfig now delegates header construction to OpenAIGPTConfig and only
layers the Fireworks-specific x-session-affinity header on top, so the
Content-Type default can no longer drift away from the OpenAI base and reintroduce
the 415.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(fireworks_ai): cover missing api key error path in chat validate_environment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
create_model_info_response cast cost-map max_input_tokens / max_output_tokens
with unguarded int(). The surrounding try/except covers only the get_model_info
lookup, so a deployment whose model_info carries a non-numeric limit (e.g.
"128,000" or an empty string) raised inside the per-model listing loop and
failed the entire GET /v1/models and /models response with a 500, taking healthy
deployments down with it. A deployment's model_info is registered into
litellm.model_cost verbatim, so the malformed value reaches the cost map and not
just the router index.
Router.get_configured_token_limits already coerced this safely for the
deployment path; the cost-map path was missed, so the two together still
regressed. Both now share coerce_token_limit in litellm_core_utils, which
returns None for a malformed value so the listing omits that one limit instead
of failing, matching the graceful degradation the endpoint had before the
cost-map switch.
The SERVER_ROOT_PATH fix for the per-server pass-through challenge belongs with its sibling
in exceptions.py (both fabricate a per-server resource_metadata URL and both omit the root
segment), and both are pre-existing paths unrelated to the aggregate discovery this PR adds.
Reverting the server.py change keeps this PR to the aggregate front door and avoids leaving
the two per-server challenge builders inconsistent; the per-server root-path fix lands as its
own change covering both sites.
tests/test_litellm/proxy/test_custom_proxy.py sets SERVER_ROOT_PATH at import time (its app
mounts under a custom path) and never restores it, so in a shared shard the value leaks into the
process. The discovery routes and the 401 challenges now read SERVER_ROOT_PATH to path-insert it
where they previously ignored it, so a leaked value rewrites every resource_metadata URL and the
exact-URL assertions in the delegate, pass-through, and aggregate challenge tests fail depending
on shard order
An autouse fixture clears SERVER_ROOT_PATH for the MCP discovery tests so they deterministically
exercise the default root-mounted deployment; the tests that assert a sub-path deployment set the
value explicitly within their own body. No assertion changed; the leak was invisible before only
because the code ignored the variable
Two RFC 9728 / 8414 discovery fixes on the aggregate front door, both raised by Bugbot on this PR
The aggregate authorization-server document at /.well-known/oauth-authorization-server/mcp used to
defer to a per-server row literally named "mcp", serving issuer {base} while the aggregate
protected-resource document advertises {base}/mcp as its authorization server. A spec client
following that chain fails the RFC 8414 issuer check and cannot sign in. The single segment /mcp is
now reserved for the aggregate so the issuer stays {base}/mcp and matches the protected-resource
document; a server named "mcp" keeps its standard two-segment discovery at
/.well-known/oauth-authorization-server/mcp/mcp
The 401 challenges built the resource_metadata URL as {base}/.well-known/oauth-protected-resource/mcp
with no SERVER_ROOT_PATH segment, but the routes are registered with the path-inserted root segment,
so a proxy mounted under a sub-path pointed DCR clients at a URL that 404s. Both the aggregate
challenge and the pre-existing per-server pass-through challenge now derive the path from one
well_known_root_suffix helper that the route registrations also use, so the advertised URL cannot
drift from the served route
The flag guarded no breaking change: the aggregate discovery lives at new /mcp-suffixed
routes, the challenge only fires at aggregate scope, and the authorize/token/register/admission
arms self-gate on the llm_dcrc_/llm_session_ prefixes. Bare-origin and per-server discovery are
left exactly as they were, and a server literally named mcp keeps its own discovery via
disambiguation, so turning it on for everyone changes nothing about existing flows.
