* fix(batches): price anthropic passthrough message batches correctly in batch cost job
Anthropic message batches created via the /anthropic passthrough were never
cost tracked. The CheckBatchCost job fetched batch results from the Files API
(POST /v1/files/msgbatch_.../content), which Anthropic rejects with "File id
must have file_ prefix"; the error response was silently wrapped as file
content, parsed as zero successful rows, logged as a $0 aretrieve_batch spend
row, and the job was marked batch_processed=true so the $0 was permanent.
Route msgbatch_ file ids to GET /v1/messages/batches/{id}/results in the
anthropic files transformation, raise on HTTP error status in
retrieve_file_content instead of returning the error body as content, parse
Anthropic's results JSONL shape (result.type == "succeeded",
result.message.usage with cache creation/read tokens) in batch_utils, price
cache creation tokens at cache_creation_input_token_cost in the batch cost
fallback (50% batch discount preserved for base input, cache reads, cache
writes, and output), and leave the managed object row unprocessed when cost
tracking fails so a later poll retries instead of permanently recording $0.
* fix(batches): carry cache token details into aggregated anthropic batch usage
The x-litellm-semantic-filter-tools response header was sliced mid-name at
MAX_MCP_SEMANTIC_FILTER_TOOLS_HEADER_LENGTH with a trailing "...", so the
admin UI test panel rendered the last selected tool name chopped. Truncate
the CSV at a tool name boundary instead so the header only ever carries
complete names, and note in the test panel how many selected tools did not
fit in the header
Streaming pass-through for native Anthropic /v1/messages and the /v1/responses
streaming iterator never set logging_obj.completion_start_time, so
_success_handler_helper_fn fell back to completion_start_time = end_time.
Downstream TTFT consumers (Prometheus, OTEL, Langfuse, Admin UI, spend logs
completionStartTime) then reported time-to-first-token equal to total request
duration.
Stamp completion_start_time on the first chunk in PassThroughStreamingHandler.
chunk_processor and BaseResponsesAPIStreamingIterator._process_chunk, mirroring
CustomStreamWrapper for /chat/completions.
Resolves LIT-4185
Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
* test(e2e): add management suite covering key/team/user/org lifecycle and route permissions
* test(e2e): decouple the enforcement-flip assertion from upstream health
Polling for a 200 on the newly-allowed model required it to be a routable,
healthy upstream, which is not the contract under test; poll until the
key_model_access_denied 403 lifts instead, excluding 401 so a revoked key
cannot read as success. Also document that the delete test's deferred teardown
firing on an already-deleted key is deliberate: cleanup must survive the test
failing before the in-body delete, and the repeat delete is a warn-free no-op
(the proxy answers 404 No keys found)
* test(e2e): inline the management suite's model and tpm literals
* test(e2e): drop the models_mgmt suite line from the folder list
* test(e2e): write the tpm limit as a plain integer literal
* feat(mcp): startup backfill stamping oauth2_flow on legacy null rows
Rows created before the write-side stamps carry a null oauth2_flow and rely on
read-time field-shape inference, which cannot tell a DCR-registered interactive
server (client creds + token_url, no persisted authorization_url) from an M2M
server unless endpoint discovery succeeds first; on a transient discovery failure
those servers flip to client_credentials for that registry load
The backfill classifies each null oauth2 row once, at rest, ordered by signal
strength: per-user token rows (only the interactive flow mints them, so this is
definitive and catches the DCR-trap cohort), then a persisted authorization_url,
then a persisted registration_url (DCR implies interactive; this covers
registered-but-never-signed-in rows), then the M2M credential shape mirroring the
legacy inference, else the interactive default that matches how
needs_user_oauth_token treats a null flow. Every stamp is logged with the rule
that fired and written with updated_by=oauth2_flow_backfill for auditability
Runs in _init_mcp_servers_in_db before the registry load so the first build of
the boot classifies from the column, is isolated so a failure cannot block server
loading, and is idempotent: a healed fleet exits after one indexed query. This
unblocks deleting the read-time inference for DB rows in the follow-up
Third step of the oauth2_flow persistence sequence, after #32283 and #32288
* fix(mcp): backfill leaves the ambiguous M2M shape unstamped instead of guessing client_credentials
The credential shape (client_id + client_secret + token_url, no interactive signal) is
shared by real M2M servers and DCR-registered interactive servers nobody has signed
into: the DCR persist writes creds and token_url but not authorization_url or
registration_url. Stamping client_credentials from that shape permanently mislabeled
the interactive cohort, and once explicit the value is authoritative, so per-user
traffic would run on the proxy's stored client credential with no discovery rescue
and no backstop (it only guards null rows)
The backfill now stamps only what it can prove. Interactive signals keep stamping
authorization_code; the ambiguous shape is left null with an actionable warning naming
the server and the fix (set oauth2_flow via the dashboard or PUT /v1/mcp/server). A
true M2M row keeps working per-request through the security backstop while the warning
nags; an interactive row keeps its Authorize button (null renders interactive), and one
completed sign-in creates the per-user token that stamps it authorization_code at the
next boot. Mirrors the config-level rule: M2M is asserted by a human, never guessed
Raised by review on the PR
* perf(mcp): batch the backfill stamps into one update_many per flow value
The per-row update loop issued one DB round-trip per legacy row at startup; rows
sharing a stamped value now go out as a single update_many, so the DB cost is
constant in fleet size. Per-row logging keeps the rule that fired for each server
Raised by review on the PR
* fix(mcp): backfill stamps only rows still null at write time and counts only real OAuth token rows as sign-in proof
Two review findings. The batched update_many matched on server_id alone, so an
explicit oauth2_flow set between the backfill's read and its write (an admin PUT or
a sign-in's DCR stamp landing in the boot window) would be overwritten with the
inferred value; the where clause now also requires oauth2_flow to still be null, so
an explicit value can never be clobbered under any interleaving
And the per_user_tokens rule counted any LiteLLM_MCPUserCredentials row as proof of
an interactive sign-in, but that table doubles as BYOK storage for user-supplied API
keys; a BYOK-flavored row would have stamped an M2M-shaped server authorization_code.
The rule now counts only rows whose payload decodes as a type oauth2 token via the
existing _decode_oauth_payload discriminator, so bare keys, undecodable rows, and
stale leftovers from a BYOK-to-oauth2 auth switch prove nothing
Raised by review on the PR
Drive the real parallel_request_limiter through _pre_call_with_fallbacks for
the LIT-3890 customer scenario: a key-level model_tpm_limit raises
ProxyRateLimitError from the pre-call hook and the configured gateway fallback
serves the request instead of returning a 429. Unlike the existing tests, this
exercises the actual limiter rather than a hand-built error.
Also switch the new _pre_call_with_fallbacks return annotation to builtin
tuple to stay within the ruff UP006 strict-rule budget.
* fix(llm_http_handler): send dict transcription request data as a JSON body
httpx form-encodes dicts passed via data= and silently ignores json=, so the
generic audio transcription path never actually sent a JSON body. No provider
hit this before; JSON-body speech APIs need it.
* feat(vertex_ai): add Google Cloud Speech-to-Text Chirp 3 transcription support
Adds a VertexAIAudioTranscriptionConfig wired through ProviderConfigManager so
vertex_ai/chirp_3 works on /v1/audio/transcriptions (sync and async) via the
Speech-to-Text v2 recognize API. Auth reuses the standard Vertex credential
resolution (vertex_project/vertex_location/vertex_credentials or ADC); the
location defaults to the us multi-region since chirp_3 is only served from the
us and eu multi-regions, and non-global locations use the regional
<location>-speech.googleapis.com host. Maps language to languageCodes (auto
language detection by default), joins all result alternatives into the
transcript, and tracks cost from totalBilledDuration with a
vertex_ai/chirp_3 price entry at Google's published $0.016/min.
* fix(vertex_ai): map bare ISO-639-1 language codes to BCP-47 for Speech-to-Text
OpenAI clients send language codes like "en", which Google rejects with 400
("not supported by the model chirp_3 in the location us"); Speech-to-Text
wants region-qualified BCP-47 like "en-US". Adds a shared
normalize_transcription_language_to_bcp47 helper in audio_utils (NVIDIA Riva's
transcription config already hand-rolled the same table privately) that maps
common bare codes and passes region-qualified ones through, and applies it in
the Vertex transcription request. Also narrows the response JSON parse guard
to ValueError.
* fix(vertex_ai): drop zero output_cost_per_second so chirp_3 cost tracking works
cost_per_second prefers output_cost_per_second whenever it is not None, so the
0.0 in the chirp_3 entry priced every transcription at $0.00 instead of using
input_cost_per_second. Remove it from both cost maps and pin the behavior with
a regression test computing 18s of chirp_3 audio to ~$0.0048.
* fix(vertex_ai): validate client-controllable location to prevent SSRF in Speech-to-Text
get_complete_url interpolated vertex_location straight into the request host,
and vertex_location is client-controllable on the proxy (it flows from the
request body and is not on the request-body blocklist). An authenticated caller
could send vertex_location="attacker.example/" to point the host at their own
server, so the proxy would POST the audio plus its admin-minted Google bearer
token and x-goog-user-project header to the attacker, exfiltrating a
cloud-platform-scoped OAuth token minted from the admin's credentials.
Factor the location validation the rest of vertex_ai already applied in
get_vertex_base_url (^[a-z][a-z0-9-]*$ plus the global allowance) into a shared
validate_vertex_location helper in common_utils and call it from both the chat
host builder and the new speech host builder. Invalid locations now raise a 400
VertexAIError instead of building a host. Also reject vertex_project values that
carry URL-structural characters, since it lands in the URL path.
Regression tests assert on the parsed netloc so the security property is pinned:
valid locations always resolve to a *speech.googleapis.com host and injection
inputs are rejected.
* fix(vertex_ai): reject unsupported transcription response_format values instead of silently ignoring
* feat(mcp): persist oauth2_flow explicitly on create instead of inferring it at read time
The UI create payload never carried oauth2_flow, so every UI-created oauth2 server
persisted a null flow and relied on _resolve_oauth2_flow's field-shape inference at
registry build. That inference cannot tell a DCR-registered interactive server
(client creds + token_url, no persisted authorization_url) from an M2M server unless
endpoint discovery succeeds first, and the dashboard cannot reproduce it at all
because credentials are redacted in responses
The create form now persists the selected flow for oauth2 servers: authorization_code
for Interactive (PKCE), client_credentials for M2M. The REST create endpoints stamp an
omitted oauth2_flow server-side with the same discriminator the legacy inference uses,
run at write time where the payload carries plaintext credentials, so the decision is
made once with full information and stored. Applied to the admin create, the BYOM
submission, and the temporary session-server endpoints
The edit form derives its flow display from oauth2_flow instead of token_url presence
(token_url is present on authorization_code servers too, so it cannot distinguish M2M)
and deliberately never writes oauth2_flow: it has no flow selector, so a write from
edit could only erase an explicit value, including the authorization_code stamp the
DCR flow persists. Regression tests pin all of this down
Second step of persisting oauth2_flow at every write site so the legacy inference can
eventually be deleted; the backfill for existing null rows lands next
* refactor(mcp): name the create-time flow stamp for its fallback-only contract
stamp_omitted_oauth2_flow with a dedicated explicit-value early return and the shape
check renamed to has_m2m_shape, so the precedence (caller's oauth2_flow always wins,
inference only fills an omitted field) reads directly off the code
* fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI
Convert Responses API custom tools to Chat Completions function tools and map
function_call responses back to custom_tool_call output items so Codex CLI gets
the apply_patch round-trip it expects. Preserve and validate allowed_callers
during the custom->function conversion so the Anthropic adapter's caller
allowlist is not silently dropped, which would let a tool meant to be callable
only by another tool be invoked directly by the model. Use modern type
annotations (list/dict/set/X | None) throughout to keep the ruff strict budget
within its ratcheted ceilings.
* fix(responses-bridge): address review feedback on custom tool bridge
Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam
and validate allowed_callers with a strict TypeAdapter so the two new cast()
calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only
tool types (computer_use, image_generation, namespace, shell) instead of
discarding them silently. Return output items as Pydantic models instead of
model_dump()ing every item to a dict, matching the declared return type. Apply
the same None-safe metadata pattern to the request_data paths that still used
setdefault, and drop the unused build_custom_tool_call_item helper.
* fix(responses-bridge): recover custom tool input when arguments is empty
* fix(auth): extract custom tool names for allowlist enforcement on responses route
The Responses guardrail translation handler only extracted function and mcp
tool names, so a key or team restricted by metadata.allowed_tools could invoke
a disallowed tool by declaring it with type custom now that the bridge converts
custom tools into callable Chat Completions function tools. Extract custom tool
names through the same path so check_tools_allowlist rejects them.
* fix(responses-bridge): scope input payload recovery to custom_tool_call items
Recovering tool arguments from the input field on any falsy arguments value
made plain function_call input items with empty arguments and a stray input
key get rewritten into a {"content": ...} envelope, corrupting multi-turn
replay for normal function tools. Gate the recovery on the item type so it
only applies to custom_tool_call items, which are the ones that store their
payload in input.
* fix(responses-bridge): default missing function_call arguments to empty string
With input recovery scoped to custom_tool_call items, a plain function_call
input item without an arguments key left raw_arguments as None and the
downstream str() turned it into the literal string None. Coerce to an empty
string instead, matching the pre-bridge behavior.
---------
Co-authored-by: duanhongyi <duanhongyi@doopai.com>
* feat(jwt): fall back to DB team memberships when JWT has no team claims
* style(jwt): use PEP 585/604 annotations in DB team fallback to clear strict gate
* fix(jwt): preserve DB teams on no-claim sync, model-gate DB fallback, stop team-id leak
When fallback_to_db_teams is enabled and a JWT carries no team claims,
sync_user_role_and_teams previously computed teams_to_remove as every existing
DB membership and wiped the user out of all their teams on each request, which
also left the DB fallback nothing to resolve. Skip team removal in that case so
memberships survive and the fallback can attribute usage.
Apply the same per-team model-access check the claim-based path enforces when
selecting a DB fallback team, so a team's models restriction is no longer
bypassed; a team that cannot serve the requested model is skipped in favor of
one that can.
Drop the user's team-id list from the x-litellm-team-id membership 403 detail so
a valid-JWT caller can no longer enumerate team IDs.
* fix(jwt): load team membership on DB fallback; scope header check to provisional teams
The DB-team fallback resolved a team but never loaded its team membership
row, so per-team membership budget limits were silently skipped on that
path. _resolve_db_team_fallback now fetches the resolved team's membership
when a user_id is known and returns it, matching the claim-based path so
downstream LiteLLM_TeamMembership budget enforcement works there too.
The provisional x-litellm-team-id validation also fired on any non-None
team_id, including an RBAC role-derived one, which 403'd RBAC team flows
when the asserted team was not also a DB membership. It now runs only when
team_id actually came from the header (team_id == header_team_id).
* fix(jwt): surface DB-fallback membership lookup failures at warning level
A transient get_team_membership failure on the DB team fallback path is
recoverable: the team is still resolved and the request proceeds, just
without per-team membership budget enforcement for that request. Logging
that at debug hid a silent budget-enforcement gap from operators, so it now
logs at warning and states that enforcement was skipped. Behavior is
otherwise unchanged: the resolved team is returned with a None membership
rather than failing the request, covered by
test_resolve_db_team_fallback_survives_membership_lookup_error.
* fix(jwt-auth): tighten db-team fallback gating and passthrough enforcement
Resolves four issues in the fallback_to_db_teams path:
- _resolve_db_team_fallback now surfaces a model-access denial when memberships
exist but none can access the requested model, instead of always returning
the no-membership message
- auth_builder gates the fallback on real JWT team claims via
get_all_jwt_team_ids so a configured team_id_default does not silently route
claimless tokens to the default team
- A team selected only via _resolve_db_team_fallback is re-validated against
the team's allowed_passthrough_routes; the earlier gate ran while team_id
was still None
- sync_user_role_and_teams considers both plural and singular team claim
shapes when reconciling DB memberships so singular-only tokens
(Okta/Auth0 defaults) no longer leave stale teams behind
* fix(jwt): don't upsert a provisional x-litellm-team-id before membership check
When fallback_to_db_teams is on and the JWT carries no team claims, an
x-litellm-team-id header is accepted provisionally and only validated against
the user's DB memberships later in auth_builder. With team_id_upsert also
enabled, get_team_object ran the upsert on that unvalidated header team first,
so an attacker-supplied header could create an orphaned team row before the
403 membership check. Suppress the upsert whenever the team is provisional
(db_team_fallback), since a genuine membership team already exists and an
invalid one must not be created. Regression:
test_auth_builder_provisional_header_team_is_not_upserted.
* fix(jwt): pin RBAC-asserted team against db-team-fallback header override
When a JWT carries an RBAC team role but no group claims, auth_builder already
sets team_id from the RBAC object_id. db_team_fallback still evaluated true
there, so the provisional x-litellm-team-id path accepted a header team and
silently overrode the RBAC-asserted team with any team the caller belonged to.
Gate db_team_fallback on team_id being unset, and drive the header's provisional
acceptance off db_team_fallback rather than the raw flag, so an RBAC token plus
a non-claim header team is rejected with 403 instead of substituting the team.
Regression: test_auth_builder_header_cannot_override_rbac_team_under_db_fallback.
* fix(jwt): scope dual-claim membership sync to fallback_to_db_teams
The membership sync read both plural and singular JWT team claims via
get_all_jwt_team_ids unconditionally, which silently changed reconciliation
for every deployment using sync_user_role_and_teams, not just those opting
into fallback_to_db_teams: a singular-only IdP token that previously stripped
all DB teams would now be recognized. Gate the dual-claim read on
fallback_to_db_teams so flag-off deployments keep the upstream plural-only
behavior, honoring the PR's contract that existing deployments are unchanged.
Regression: test_sync_user_role_and_teams_singular_claim_only_recognized_under_flag.
* fix(jwt): drop user team IDs from db-fallback model-access 403 detail
The model-access-denied 403 in _resolve_db_team_fallback echoed the user's
full DB team-id list in its detail. It is only the caller's own memberships,
but it is inconsistent with the membership-validation 403 in the same feature
that was deliberately scrubbed of team IDs. Replace the enumerated list with a
generic "no team you are a member of has access" message. Regression extends
test_resolve_db_team_fallback_distinguishes_no_membership_vs_model_denied to
assert the team id is absent from the detail.
* fix(jwt): keep db-team fallback off for alias-only tokens
* test(jwt): cover alias-only token skipping db-team fallback
The autofix in ed21199 added a get_team_alias clause to the db_team_fallback
gate so an alias-only JWT (team_alias_jwt_field set, no team-id claims)
resolves its alias via find_and_validate_specific_team_id instead of being
mis-attributed to the user's first DB team, but it shipped without a
regression test. This drives auth_builder with an alias-only token whose
alias resolves to a different team than the user's DB membership and asserts
the result is the alias-resolved team; reverting the get_team_alias clause
flips the result to the DB-membership team, so the test fails without the fix
* fix(jwt): prefer alias resolution over team_id_default
When the JWT only carries an alias claim and the operator configures
team_id_default, JWTHandler.get_team_id silently substitutes the
default into find_and_validate_specific_team_id. That made the helper
return the default team without ever attempting alias resolution, so
spend and access attached to the default team even though the token
identified a different team via its alias. Use get_all_jwt_team_ids
(which ignores team_id_default) to detect when the resolved team_id is
only the default and clear it so alias resolution runs first; the
default remains the fallback when no alias claim is present.
* fix(jwt): enforce team_allowed_routes in db-team fallback resolution
The claim-based path runs allowed_routes_check when selecting a team, but
_resolve_db_team_fallback selected a team purely on model access, so a
DB-resolved team could reach routes excluded by team_allowed_routes with no
downstream backstop. This mirrors the claim path's route gate in the fallback,
exempting auth-enforced passthrough routes that are gated separately by
allowed_passthrough_routes at the call site
* fix(jwt): enforce team_allowed_routes on header-team db fallback path
The auto-pick DB-team fallback already gates against team_allowed_routes, but a claimless JWT presenting x-litellm-team-id under fallback_to_db_teams set team_id directly from the header and only re-validated DB membership afterwards, skipping the route gate. A caller could reach management/info routes that the JWT config narrowed for team-role callers by supplying the header even though the auto-pick path on the same route returns no team.
* refactor(jwt): narrow db-team fallback except clauses to actual failure types
* fix(jwt): collapse provisional header team lookup failure into membership denial
A caller holding a valid claimless JWT under fallback_to_db_teams could
distinguish nonexistent teams (404 from get_team_object) from existing
teams they do not belong to (membership 403) by varying x-litellm-team-id,
giving an authenticated team-id existence oracle. The provisional header
path now rewrites the lookup failure into the exact 403 the membership
check raises, while claim-backed header teams keep the upstream 404.
Also drop the unreachable falsy-team guard in _resolve_db_team_fallback
(get_team_object returns a team or raises, never None) and stop codecov
carryforward for three dead flags whose stale sessions were measured
against old file revisions and sank patch coverage with phantom
executable lines
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
ModelArmorGuardrail.async_pre_call_hook and async_moderation_hook hardcoded
their inner should_run_guardrail event type to pre_call / during_call. The
central dispatcher already remaps call_mcp_tool -> pre_mcp_call/during_mcp_call
and passes the outer gate, but Model Armor's redundant inner gate then rejected
MCP calls for a guardrail configured with mode pre_mcp_call/during_mcp_call, so
tool-call content was silently skipped.
Remap call_mcp_tool -> pre_mcp_call/during_mcp_call in both hooks, matching the
existing behavior of the noma and cisco guardrails. Adds regression tests
covering both hooks (scan runs on MCP calls, still skipped for chat traffic).
Generated with AI
Co-Authored-By: Claude Code
Co-authored-by: eugene-yao-zocdoc <eugene.yao@zocdoc.com>
* fix(mcp): forward short OAuth state upstream, keep session in a cookie
Some upstream authorization servers reject the OAuth authorize request with
"state parameter too long" because LiteLLM replaced the client's short state
with its own long encrypted session blob (base_url, original state, PKCE, client
redirect_uri) and sent that upstream as state.
Forward a short random handle as the upstream state instead, and carry the
encrypted session in a per-flow HttpOnly, SameSite=lax cookie bound to that
handle. The browser replays the cookie on /callback, so the session is recovered
without any server-side store and the client still gets its own original state
back. /callback falls back to decoding state directly when no cookie is present,
so flows in flight across a deploy keep working.
Resolves LIT-4197
* test(mcp): cover /callback error path cookie read and clear
The happy-path regression test already asserts the short-handle -> cookie round
trip. Add a focused test for the IdP-error branch of /callback: it must recover
the client's original state from the per-flow cookie (not the short handle),
propagate the error to the client's redirect_uri, and expire the one-time
cookie. Fails if the error path stops reading or clearing the cookie.
* fix(logging): classify allm_passthrough_route as async to prevent duplicate success callbacks
Async passthrough requests set kwargs["allm_passthrough_route"]=True but that
flag is never propagated into litellm_params, and _is_sync_litellm_request only
checks acompletion/aresponses/aembedding/aimage_generation/atranscription.
Every async passthrough is misclassified as sync, which trips the CustomLogger
sync branch in success_handler and fires log_success_event in addition to the
async worker's async_log_success_event, causing 2-3 duplicate LangSmith runs
per Bedrock passthrough request
Propagate allm_passthrough_route through get_litellm_params and teach the
classifier about it. /chat/completions and other non-passthrough paths are
untouched
* test(passthrough): assert allm_passthrough_route flag propagates end-to-end
Integration-level guard on top of the unit tests in test_litellm_logging.py:
verifies that when kwargs["allm_passthrough_route"]=True enters
llm_passthrough_route, the flag survives get_litellm_params(**kwargs), lands
in the logging object's litellm_params, and _is_sync_litellm_request reads
the request as async
---------
Co-authored-by: yucheng <yucheng@yuchengs-MBP.localdomain>
Only the gateway-managed interactive flow reaches this persist (the public /register
routes never pass persist_credentials), so the row it writes is authorization_code by
definition. It was not recorded, which left the row as client creds + token_url with
no persisted authorization_url and a null oauth2_flow: exactly the shape the legacy
M2M inference in _resolve_oauth2_flow matches. The row normally survives because
endpoint discovery backfills authorization_url in memory before the inference runs,
but on any transient discovery failure at registry build the server flips to
client_credentials for that load, routing per-user traffic to the M2M path
Stamping the flow at the write site makes the classification explicit and permanent,
so a DCR-registered interactive server no longer depends on discovery succeeding to
classify correctly. First step of persisting oauth2_flow at every write site so the
legacy inference can eventually be deleted
* fix(e2e): route model management to the control plane and restore Gateway.create_model
The split-transport routing table listed only /model/info as a control-plane
prefix, so /model/new and /model/delete were sent to the data-plane gateway,
which does not serve management routes and 404s them. Every suite that
registers deployments at runtime (llm_translation, batches, access_control)
failed on the split stage deployment because of this. Widen the prefix to
/model/ so all model-management routes reach the control plane while /models
stays on the data plane.
Separately, batch_client.py and several llm_translation tests call
gateway.create_model, but Gateway never had that method, so all 17 batch tests
errored at fixture setup with AttributeError. Add create_model/delete_model to
Gateway (with the optional mode that batches needs) and make EndpointsClient
delegate to it instead of carrying its own copy.
Regression tests cover both: the routing predicate for management vs LLM paths
and the Gateway model-management surface via a typed fake Transport. Both fail
on the previous code
* test(e2e): make the fake transport payload depend on response_type
The recording fake always answered with {"model_id": ...} even when the
caller asked for NoBody, which only validated because pydantic ignores extra
fields by default. Return an empty payload for response types that carry no
fields so a future extra="forbid" on NoBody cannot turn the delete test into
a ValidationError inside the fake
* test(e2e): probe the full spend read surface including schema-hidden routes
The curated spend-route list missed twelve read endpoints, most of them
include_in_schema=False and therefore invisible to the schema-discovery test:
/spend/logs/v2, /spend/logs/session/ui, /global/all_end_users,
/global/activity/exceptions/deployment, and the per-entity daily activity
family (user, user aggregated, team, organization, customer, end_user, tag).
Add them all, verified responsive against the live split stage deployment.
/end_user was missing from CONTROL_PLANE_PREFIXES, so /end_user/daily/activity
would have been routed to the data plane and 404ed like /model/new used to;
add the prefix and pin it plus the daily-activity routes in the transport
routing test.
/provider/budgets stays excluded with a documented reason: it returns 500
whenever router_settings.provider_budget_config is absent, so probing it on a
proxy without provider budget routing configured can never be green
* fix(security): hash Bearer-prefixed API keys in spend logs
The safety-net hash in get_logging_payload only checked for keys
starting with 'sk-', missing keys that arrived as 'Bearer sk-...'.
This caused plaintext API keys to be stored in SpendLogs for failed
requests while successful requests correctly stored SHA256 hashes.
Adds _hash_api_key_for_spend_log that strips the Bearer prefix
before hashing, applied to both the api_key column and the
metadata.user_api_key field in spend log payloads.
* fix: strip Bearer prefix from non-sk keys in spend log fallback path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
CacheCodec.serialize dumped cached Pydantic models with model_dump(exclude_none=True), which drops any None-valued key, while deserialize does a strict model_validate. For a model with a required-but-nullable field (Optional[X] with no default), a None value is dropped on write and then fails model_validate on read with "Field required", so the entry can never be read back; that is a permanent cache miss, and in readers that rebuild the model from the raw cached dict an uncaught ValidationError that surfaces to the client as a 401
Removing exclude_none makes serialize and deserialize a lossless pair, so None fields are written as null and survive the round trip. LiteLLM_ManagedVectorStoresTable, the one cached model still carrying required-nullable fields and mis-caching on every read today, also gets the None defaults its peers already have
Round out the AWS auth-field coverage of _DB_LITELLM_PARAM_ENV_REF_KEYS
so every stringy aws_* field a deployment can pin in the DB resolves
os.environ/ refs at load time:
- aws_bedrock_project_id: Bedrock project/workspace association, banned
from request bodies via _BANNED_REQUEST_BODY_PARAMS
- aws_batch_role_arn: Bedrock batches role ARN (analog of aws_role_name)
- aws_workspace_id: Claude Platform workspace ID
Verified against three independent sources:
- BaseAWSLLM.aws_authentication_params (all 11)
- LiteLLM_Params-declared AWS fields (all 5)
- every aws_* string read from litellm_params/kwargs/optional_params
across litellm/ (all 14, excluding aws_bedrock_client which is a
boto3 client object, not a string, and aws_polly which is a provider
name)
The regression test now pins all 12 newly-allowlisted fields.
PR #30867 removed request-time os.environ/ expansion in
BaseAWSLLM.get_credentials to close LIT-3831. That relies on config-load
paths pre-resolving os.environ/ refs, but the DB-load path
(_resolve_db_litellm_param) only re-expands keys in
_DB_LITELLM_PARAM_ENV_REF_KEYS, which covered api_key,
aws_access_key_id, and aws_secret_access_key but not the other AWS auth
fields. A model stored in Postgres with e.g.
aws_role_name: os.environ/BEDROCK_ASSUME_ROLE_ARN
lands on the router with the literal string, get_credentials no longer
expands it, and STS returns
ValidationError: os.environ/BEDROCK_ASSUME_ROLE_ARN is invalid
Add the remaining AWS auth params to the allowlist so DB-sourced values
resolve at model-load time (trusted, server-side), matching the
YAML-config path. Team-scoped DB rows still get resolve_env_refs=False,
so the LIT-3831 defense-in-depth path is unchanged and request-body
injection is still blocked by _BANNED_REQUEST_BODY_PARAMS.
Regression tests pin every added field as an os.environ/ DB value and
assert it resolves on the router, plus a team-scoped pin that asserts
env refs remain literal.
Previously, every guardrail request forwarded the full conversation
history to CrowdStrike AIDR. In a multi-turn conversation this means
every prior message gets re-scanned on every new call, even though those
messages were already evaluated in earlier turns.
CrowdStrike AIDR internally has a conversation boundary optimization in
place for just this scenario (ref. <https://aidr-docs.crowdstrike.com/docs/aidr/apis#messages-array-optional---array-of-message-objects-containing-a-conversation-segment-with-the-ai-system>).
However, it is nevertheless wasteful to send so much data to the API
when only a subset of it will be processed. It also risks hitting the
documented 1 MiB request size limit.
So now we filter down to system messages plus either the messages after
the last assistant turn, or the last assistant message itself when that
is what is being guarded. We also preserve the original, full message
history within the guardrail in order to stitch back any
transformations.
Co-authored-by: Kenan Yildirim <kenan@kenany.me>
Port of #16162 by @dhruvyad onto litellm_internal_staging.
OpenRouter sends a usage chunk (including a provider-reported cost field)
after the finish_reason chunk. Previously the stream handler raised
StopIteration on the first post-finish chunk, so that usage/cost never
reached the assembled response and cost tracking fell back to token-based
estimates.
Carry usage.cost through chunk accumulation, preserve stripped usage in
_hidden_params, and propagate the provider cost into
_hidden_params["additional_headers"]["llm_provider-x-litellm-response-cost"]
so the cost calculator uses it.
* fix(streaming): surface in-body error payloads on OpenAI-compatible streams
vLLM and sglang return HTTP 200 streams whose SSE body carries the error,
e.g. data: {"error": {"message": "...", "code": 400}}. The OpenAI-compatible
chunk parser had no detection for this shape: since #23931 the payload parsed
into an empty chunk (choices=[]) and the stream ended silently with 200,
losing the provider's error and never attempting configured fallbacks.
Detect the payload in OpenAIChatCompletionStreamingHandler.chunk_parser and
raise OpenAIError with the upstream message and status code. The existing
mid-stream gate then applies: 4xx surface directly to the client, 5xx wrap
into MidStreamFallbackError so the router can run configured fallbacks.
Fixes#25492
* fix(streaming): serialize messageless error payloads as JSON
Address review feedback: an error dict without a message field now
serializes via json.dumps instead of Python dict repr
* feat(router): add separate ITPM/OTPM deployment rate limits
Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers.
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(router): keep ITPM/OTPM diff minimal in router.py
Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): make ITPM/OTPM limits separate and atomic
Address Greptile review on separate ITPM/OTPM deployment rate limits.
- OTPM is now reserved atomically pre-call with rollback, matching the ITPM
path, so concurrent requests can no longer overshoot the configured output
limit before reconciliation
- ITPM counts input tokens only; it no longer accumulates completion tokens,
so the input-token limit and x-ratelimit-limit-input-tokens header describe
input usage as their names imply
- _read_reservation_from_kwargs only falls back to litellm_params.metadata when
the top-level metadata channel is absent, so production requests carrying a
litellm_params.metadata dict still reconcile and refund their reservation
Adds regression tests for OTPM atomicity under concurrency, input-only ITPM
enforcement, and reservation lookup when litellm_params.metadata is present.
* fix(router): subtract input tokens only from remaining-input-tokens header
The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total
tokens (input + output) instead of input tokens only, so clients saw remaining
input quota understated by the completion token count on every response. Now
consistent with the input-only ITPM counter.
* fix(router): make itpm/otpm vs tpm/rpm precedence explicit
When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path
takes over and the tpm/rpm limits are not enforced. Log a warning the first
time such a conflicting deployment is seen so the supersession is not silent,
and document the mutual exclusivity.
Post-call reconciliation now only trues up a counter that was actually
reserved against, so the itpm/otpm keys are no longer incremented for
deployments that never configured that limit.
* fix(router): track actual io-token usage on the reservation-minute key
Post-call reconciliation now keys off the exact cache key stashed at pre-call
time rather than one recomputed from the response-time minute. This fixes two
issues: a request whose pre-call estimate was 0 now still writes its actual
billable input to the ITPM counter (previously it was skipped, leaving the
limit unenforceable for that request), and a call that finishes in a later
minute reconciles against the minute it reserved against instead of pushing a
negative delta into the next minute. Counters are only touched when their
limit is configured.
* fix(router): run io-token reconciliation before the model_id guard
async_log_success_event gated IO reconciliation behind the model_id guard that
only the TPM tracking path needs. Since reconciliation works entirely from the
cache keys stashed in kwargs, a success event whose standard_logging_object
lacks model_id would skip reconciliation and leave the reservation on the
counter until the TTL expired, wasting quota. Route the IO path first.
* fix(router): don't replay in-flight delta for itpm/otpm headers
For ITPM/OTPM model groups the counter is incremented at reservation time
(pre-call), so the remaining values returned by get_remaining_model_group_usage
already account for the current request. Replaying the in-flight delta on top
double-counted it and understated x-ratelimit-remaining-input/output-tokens by
up to max_tokens on every response. Skip the delta for io-token groups; the
legacy TPM/RPM replay path is unchanged.
* fix(router): clear io-token reservation after reconcile/refund
async_io_token_refund_failure and async_io_token_reconcile_success now clear
the stashed reservation keys from the request metadata once done. Otherwise, on
a model group mixing IO-limited and non-IO deployments, a failed IO call that
retries on a non-IO fallback left the stale sentinel in the shared request
metadata; the fallback's success handler would divert into IO reconciliation
against the already-refunded key, driving the ITPM counter negative and
skipping the non-IO deployment's TPM tracking.
* fix(router): tidy reservation channel lookup and header guard
Consolidate the reservation channel lookup into a single ordered helper shared
by read and clear, so top-level metadata always wins over litellm_params
metadata without the tangled per-iteration fallback.
Also stop gating the router rate-limit header block on the presence of
x-ratelimit-remaining-input/output-tokens. That block only emits those headers
for ITPM/OTPM groups; for a non-IO group backed by a provider that natively
returns input/output token headers, the extra conditions suppressed the
router's own remaining-tokens/requests headers.
* fix(router): strip client-supplied io-token reservation keys
The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key,
and the otpm equivalents) are server-only, but metadata is caller-controlled on
proxy requests. An authenticated caller could forge these fields with an
arbitrary cache key so the post-call reconcile/refund path would decrement any
deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip
the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs,
which runs before the router stashes its own reservation, so only a genuine
server-side reservation is ever read post-call.
* fix(router): track TPM routing load for io-limited deployments
deployment_callback_on_success early-returned for any deployment with itpm/otpm
set, so its total-token usage never landed in the router's TPM routing counter.
TPM-aware routing strategies then saw 0 load for IO deployments and over-routed
to them in mixed model groups. Only skip tracking when neither tpm/rpm nor
itpm/otpm are configured; itpm/otpm enforcement still runs separately in
ModelRateLimitingCheck, so the routing counter and the enforcement counters
stay independent.
* fix(router): expose standard tpm/rpm headers for io-limited groups
get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group
that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests;
clients and prometheus gauges reading those saw no data. Build both header sets
instead of returning early.
Also simplify the in-flight header replay: only the tpm/rpm counters are
incremented post-response, so the delta now adjusts just those. The itpm/otpm
counters are incremented at reservation time (pre-call), so the input/output
token headers already reflect the request and are left untouched - which
removes the need for the separate io-group special case.
* fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance
Two follow-ups from review. The pre-call OTPM reservation only rolled back the
ITPM reservation on a RateLimitError, so a transient cache error while reserving
OTPM left the ITPM counter inflated until the TTL expired; catch any exception,
release the ITPM reservation, then re-raise.
Replace the module-level lru_cache warn-once (caching a logging side effect,
which never re-warns in a long-lived process) with an instance-scoped set of
already-warned deployment ids on ModelRateLimitingCheck.
* fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup
Clear the reservation in a finally block so a mid-reconciliation cache error
still removes the stash and a duplicate success event can't re-process it.
Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a
deployment with no id no longer collapses every id-less deployment onto the
str(None) key (which would suppress all but the first warning).
* fix(router): skip io reservation when deployment can't be keyed
_get_cache_keys returned a shared 'global_router:None:None:...' key when a
deployment was missing model_info.id or litellm_params.model, so misconfigured
deployments could share one rate-limit bucket. Return None in that case and
skip io reservation for the request.
* fix(router): honor explicit max_tokens=0 in io reservation
_resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit
max_tokens=0 fell through to the model default. Only fall back to
max_completion_tokens when max_tokens is absent.
* fix(ci): satisfy lint budget, router coverage, and dashboard schema sync
- Modernize the new itpm/otpm module's type hints to PEP 585 lowercase
generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006
violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match.
- Replace three try/except Exception blocks that must stay broad by design
(token_counter and litellm.get_model_info raise untyped exceptions, and an
io-token refund failure must never break the logging pipeline) with
contextlib.suppress(Exception), matching the codebase's existing resolution
for this exact BLE001 pattern.
- Add direct unit tests for get_model_group_io_token_usage (multi-deployment
aggregation and the empty-model-list case) in test_router_helper_utils.py,
satisfying the router function-coverage check.
- Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on
GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types.
* fix: enforce io token rate limits consistently
* fix: honor zero max tokens in otpm reservation
* fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base
Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10
floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span
alias.
The previously committed ruff-strict-budget.json ratcheted UP006 down from a
stale base; litellm_internal_staging has since tightened that same ceiling
further on its own. Reset the file to the current base's committed values and
re-ratchet from there so the budget only ever moves down relative to the
actual merge-base, never against a stale snapshot.
* fix(router): attach ITPM/OTPM headers on dict responses and harden reservation
Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM
estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit
headers through /v1/messages dict responses via _hidden_params.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses
Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so
set_response_headers can attach rate-limit headers to streaming Anthropic
messages responses that lack a _hidden_params slot.
Co-authored-by: Cursor <cursoragent@cursor.com>
* style: ruff format add_retry_fallback_headers.py
Fix CI ruff format check failure on get_hidden_params_dict call site.
Co-authored-by: Cursor <cursoragent@cursor.com>
* refactor(router): extract set_response_headers helpers to fix C901 budget
Move header-attachment logic into add_retry_fallback_headers helpers so
set_response_headers stays under the strict complexity ceiling.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: keep IO token reservation when response usage is missing
Missing usage was reconciled as zero and fully refunded the pre-call
reservation, allowing limit bypass on repeated successful calls. Only
adjust counters when usage is resolved from the response or standard
logging fields; otherwise keep the reservation until TTL expires.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: enforce RPM/TPM alongside IO-token limits on mixed deployments
Deployments with both itpm/otpm and tpm/rpm previously returned after the
IO reservation and skipped RPM/TPM checks. Run both paths and refund the
IO reservation only when RPM/TPM rejects after a successful reservation.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: track TPM usage on success for mixed IO+TPM deployments
The early return after IO-token reconciliation in log_success_event and
async_log_success_event skipped the TPM counter increment, so the tpm_key
the pre-call check reads was never written and tpm_limit was never
actually enforced on deployments that also configure itpm/otpm.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: treat total-only usage as unresolved in IO-token reconcile
usage/standard_logging_object entries carrying only total_tokens (no
prompt/completion or input/output breakdown) were treated as resolved
usage, resolving to (0, 0) and refunding the full reservation. Both
_usage_is_present and the standard_logging_object fallback now require an
actual input/output breakdown before reconciling, keeping the reservation
otherwise.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: reserve minimal token when input/output estimation fails
_reservation_value(0, limit) reserved the entire limit whenever token
estimation failed (empty/unsupported input, tokenizer error), letting one
such request claim the whole bucket and 429 every concurrent request to
the deployment until it completed. Reserve 1 token instead so estimation
failures no longer serialize traffic.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix: refund IO reservation synchronously before retry deployment pick
On retry, set_io_token_rate_limit_request_kwargs clears reservation
sentinels from the shared kwargs dict before a background failure handler
can refund them, stranding the counter until TTL. Refund and clear any
stale reservation in _update_kwargs_with_deployment before stripping
sentinels for the next attempt.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling
Pass the deployment litellm_params.model to token_counter so it uses the
model's native tokenizer instead of the generic fallback, narrowing the
reservation over/under-estimate window between pre-call and post-call
reconcile.
Add a ponytail: comment to refund_stale_reservation_before_retry explaining
the known ceiling: the synchronous DualCache.increment_cache issues a
blocking Redis INCR when a Redis backend is configured. This only fires on
streaming mid-stream retries (non-streaming failures await their failure
handler before the retry picks a new deployment, leaving no sentinels to
refund). Upgrade path: make _update_kwargs_with_deployment async.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
The group_by=team branch queried spend for every team in the date range
regardless of team_id; team_id was only honored in the separate branch that
also required a customer_id. Route the team report through a new
get_spend_by_team helper (sibling of get_spend_by_team_and_customer) that binds
team_id as an optional $3 predicate, so a provided team_id narrows the result
to that team and a null team_id still returns every team.
* ci: gate CircleCI jobs on changed paths
Every CircleCI job used to run on every PR. Now each job starts with a
lightweight `skip_if_unrelated_changes` step that inspects the PR diff and
halts the job as successful when nothing relevant changed. Docs-only PRs
(*.md, *.mdx, docs/) run nothing, UI-only PRs (ui/) run just the frontend
jobs, and any backend change still runs both the backend and frontend jobs.
The decision logic lives in .circleci/scripts/classify_changes.sh (pure,
reads the changed-file list on stdin) so it can be unit tested, while
path_filter.sh handles the git plumbing and fails open (runs the job) on
any uncertainty such as a missing merge base or a non-PR pipeline. Halting
via `circleci-agent step halt` keeps the job green, so required status
checks are never left pending. The Windows smoke job is intentionally left
ungated to avoid cross-platform shell fragility
* fix(ci): keep path filter fail-open when classifier errors
Guard the classify_changes.sh invocation with `|| run_full` so a broken or
non-zero classifier runs the job instead of falling through to a silent
halt, and mark the advisory logging pipe best-effort with `|| true`. Add
path_filter.sh regression tests covering the docs-only halt, backend run,
non-PR fail-open, and classifier-failure fail-open paths
The test posted /config/update, slept a fixed 20s, then fired a single chat request with no readiness check or retry. When the single-process proxy was momentarily not accepting connections in that window, the request failed with a bare openai.APIConnectionError and took the whole job down, since the suite runs against one shared container with pytest -x
Gate the chat request behind a /health/liveliness poll, retry it on connection errors only so real HTTP errors and the Langfuse assertion still fail the test, close the previously leaked aiohttp session, and target 127.0.0.1 instead of the 0.0.0.0 bind address. In CI, give the proxy container --restart on-failure so an intermittent crash recovers instead of leaving the port dead for the rest of the run
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects
spend_tracking/spend_e2e_client.py imported SpendTagsResponse and
TagSpend from models, but neither was ever defined, so importing the
client raised ImportError and pytest aborted collection for the whole
e2e session. The tag-spend tests had never run.
Model /spend/tags as it actually answers: a bare array of per-tag
aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the
existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field
that also wouldn't match the array shape; it now reads .root, matching
how spend_logs consumes its RootModel.
* test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction
Adds regression nets and gap-surfacing tests:
A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the
DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document
that reasoning_effort='none' and thinking type='disabled' are silently dropped
(LIT-3686 / GH #27453)
A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py):
parametrized regression net asserting real completion content, not just a 200,
across the configured providers for /chat/completions and /responses (GH #28991)
A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is
honored and prompt-cache read tokens grow on a repeated cacheable prefix
A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call
rate limiter runs, then asserts no unattributed spend row is left behind by the
internal input-file retrieval (LIT-3266)
A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct
key_alias and asserts each alias gets its own labeled series on /metrics
A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict)
regression proving eviction must not close an httpx client still held by an
in-flight caller (LIT-3221 / GH #13034)
Extends tests/e2e/models.py with the typed request and response fields these
tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage
fields, spend-log api_key)
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): drop unused litellm-regression-tests submodule
The e2e suite migrated the regression cases into this repo; nothing
imports the submodule at runtime (only a provenance comment references
it), so the .gitmodules entry and gitlink pointing at a personal repo
would just make upstream CI init a submodule it never uses. Remove both
to keep the change test-only.
* test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage
The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client
close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread
cleanup vs shared client teardown) and belongs in its own PR, not this e2e
coverage PR, so revert the file to its base state.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): migrate access-control and inference-endpoint regression tests
Move the access-control and non-chat inference-endpoint cases from litellm-regression-tests onto the shared e2e harness so a regression in either fails here first
access_control/ asserts the gateway's authorization and error-shape contract: a key limited to one model is denied 403 (key_model_access_denied) when it calls another, a key scoped to allowed_routes=["llm_api_routes"] is forbidden 403 from a management route, and an unknown model is rejected 400 before any provider is called. The source asserted 401 for the disallowed-model case against an older proxy; the live contract is now a 403, so the guard tracks current behavior
llm_translation/ gains one file per non-chat inference endpoint (/v1/responses, /v1/messages, /embeddings, /v1/rerank, /v1/audio/speech, /v1/images/generations). Each test registers the deployment it needs through /model/new, drives real provider traffic, asserts the parsed body carries real content instead of just a 200, then deletes the model on teardown, so nothing is hardcoded into the gateway config
* Update endpoints_client.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(bedrock): emit SSE error event when invoke Messages stream ends without message_stop
* fix(bedrock): tighten stream-terminal detection to avoid false positives and double errors
The bytes branch of _is_message_stop_chunk used a plain substring match,
so a content_block_delta whose partial_json contained the literal text
message_stop would look like a real terminal event and suppress the
synthetic incomplete-stream error. Match the SSE event header line
instead.
Also treat a provider-emitted error event as terminal so a stream that
ends with an upstream error is not followed by a second, contradictory
synthetic incomplete-stream error.
* test(bedrock): lock in that the synthetic truncation error event is excluded from logged chunks
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(anthropic_messages): forward provider response headers on streaming /v1/messages responses
* fix(anthropic_messages): forward aclose to inner streaming iterator
* fix(anthropic_messages): forward aclose through the streaming response wrapper
The proxy's streaming cleanup closes the handler's return value via
hasattr(response, "aclose"); the new wrapper hid the upstream
generator's aclose, so provider connections could linger on client
disconnect. The wrapper now delegates aclose to the wrapped stream and
AgenticAnthropicStreamingIterator closes its inner and follow-up
streams. Also adds test coverage for the agentic streaming branch
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix: zero out crash-class basedpyright rules across litellm/
* feat(lint): add LIT009 banning inert type: ignore comments
* docs: require bracketed rule and reason on every suppression
* chore(lint): ratchet budgets down and zero crash-class pyright limits
* fix: narrow auto router routelayer through a local before calling
* test: add regression tests for crash-class fixes
* fix: drop dead AZURE_AD_TOKEN lookups and word-bound the type-ignore regex
* feat(mcp): add entra_obo profile to the token_exchange (OBO) arm
Microsoft Entra On-Behalf-Of uses the RFC 7523 jwt-bearer grant rather than RFC 8693, so the existing token_exchange arm cannot mint tokens against Entra. This adds an entra_obo profile on TokenExchangeConfig that switches the request form to Entra's jwt-bearer OBO dialect (the inbound token as assertion, the target resource carried in scope, and requested_token_use=on_behalf_of), while reusing the shared caching, single-flight, TTL, and fail-closed machinery. The exchanger is renamed from Rfc8693TokenExchanger to OboTokenExchanger since it now serves both dialects
The profile is threaded through the config-load path, the DB credentials blob, and the MCPCredentials request schema, so an operator can select it from YAML or the management API. It rides in the existing credentials JSON blob, so there is no new auth_type and no DB migration
Resolves LIT-4163
* feat(mcp): propagate the Entra Conditional Access step-up challenge on the OBO 401
An entra_obo exchange that the IdP rejects for Conditional Access returns a 4xx with
error=interaction_required and a claims blob the client must satisfy to step up. The arm
dropped both and emitted a static RFC 9728 challenge, so a CA-protected Entra upstream was
unreachable through the gateway. The provider now reads the RFC 6749 error code and the claims
string off the rejection body (error_description is still never carried; it can leak IdP
internals), threads them through SubjectTokenRejected -> CredError.unauthorized, and the
challenge builder folds them into WWW-Authenticate: the machine error only when it is a plain
OAuth token (guards against header injection from a hostile body) and the claims base64-encoded
in a claims parameter, the convention MSAL-family clients decode. With neither field the header
is byte-identical to the static challenge. The multi-server aggregate still absorbs a
step-up 401 to an empty listing; only single-server routes surface it
* fix(mcp): use error=insufficient_claims for the Entra step-up challenge
Per Microsoft's claims-challenge format, a WWW-Authenticate carrying a claims challenge must set
error=insufficient_claims (the value MSAL-family clients key on to recognize the challenge and
replay the claims), not the raw token-endpoint code. The challenge now sets insufficient_claims
whenever a claims blob is present and keeps invalid_token otherwise, with a step-up-accurate
error_description in the claims case. The presence of claims now drives the error value, so the
raw oauth_error no longer needs threading from the provider through CredError to the edge; that
plumbing is removed (the provider still reads the error code for its gateway-fault classification).
Both the error value and the base64 claims are fixed-alphabet, so nothing from the IdP body reaches
the header unescaped. Cross-checked field-by-field against Microsoft Learn; a bogus jwt-bearer OBO
POST to the real login.microsoftonline.com/common endpoint confirmed Entra recognizes the grant and
returns the error shape the parser reads. Follow-up: also emit authorization_uri alongside
resource_metadata for strict non-MCP MSAL clients (RFC 9728 resource_metadata already serves MCP)
* fix(mcp): filter blank scopes on the config-load path so entra_obo fails closed
The DB-build path already runs YAML/DB scopes through _extract_scopes (which drops blanks), but the
config-load path read server_config["scopes"] raw. A YAML `scopes: [""]` therefore reached the
exchanger as a ("",) tuple: non-empty, so the entra_obo `not config.scopes` precondition skipped its
fail-closed misconfigured path and the form builder POSTed an empty scope to the IdP instead of
failing before any network call. Config-load now filters blanks the same way, so an all-blank list
normalizes to None and the precondition fails closed. Regression test loads a blank-scope entra_obo
server and asserts the exchange returns misconfigured without POSTing
* feat(prometheus): add api_provider label to token, latency, request and cache metrics
The token (input/output/total), latency (llm_api, time_to_first_token,
request_total, request_queue_time), proxy request (total/failed) and cache
metrics were emitted from the same call sites as litellm_spend_metric and
litellm_requests_metric, which already carry api_provider, yet these were
missing it. That left no way to break tokens, latency, request counts or cache
hits down by upstream provider even though the provider is already on the
payload as custom_llm_provider.
Add api_provider to each metric's label allow-list. The success path already
populates enum_values.api_provider from standard_logging_payload, so those
metrics emit it with no further plumbing. The cache label is added to the
shared _cache_metric_labels list, so alongside litellm_cache_hits_metric and
litellm_cache_misses_metric it also covers litellm_cached_tokens_metric and the
provider prompt-cache read/creation token metrics; the label-presence test
asserts all of them. For the client-side failure path, where a deployment may
not have been resolved, derive it best-effort from
litellm_params.custom_llm_provider, a partial standard_logging_object, or
inference from the requested model name via litellm.get_llm_provider, falling
back to empty rather than guessing.
Resolves LIT-4178
* fix(prometheus): satisfy ruff BLE001 budget and update enterprise label assertions
- Suppress the strict-rule BLE001 budget breach with a justified noqa;
the broad except in the failure-path provider extraction is
intentional defense-in-depth (covered by
test_extract_api_provider_swallows_unknown_model_but_logs_unexpected_errors),
not dead code to delete
- Update tests/enterprise assertions for litellm_tokens_metric,
litellm_input_tokens_metric, litellm_output_tokens_metric, the three
latency metrics, and the proxy request counters to expect the new
api_provider label, matching what litellm_mapped_enterprise_tests
caught in CI
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
