is_error_str_rate_limit treats any standalone 429 in the stringified exception as
a rate limit, and for openai-compatible providers that check runs before the
status-code branch. Providers echo the request back in validation errors, so a
400 whose body happens to contain a 429 comes out as RateLimitError.
Tokenised prompts hit this routinely, since 429 is an ordinary token id (" that"
in several tokenisers) and an echoed prompt_token_ids array is enough:
{"error":{"message":"`tools` must not be an empty array",
"type":"invalid_request_error","code":400},
"prompt_token_ids":[9906,429,1234]}
The mislabel is not cosmetic. RateLimitError tells callers and routers to retry,
so a request that cannot succeed gets replayed, and the failure is booked against
provider throttling rather than the caller. Against DeepInfra, one recurring 400
("`tools` must not be an empty array") came back as a rate limit in 77 of 198
occurrences, the split depending only on whether the echoed prompt contained 429.
16482 narrowed '"429" in error_str' to \b429\b after a false positive on
'asbjdad429addad'. Word boundaries cannot separate a real 429 from a token id, so
the same class of false positive survives.
is_error_str_rate_limit now takes an optional status_code, and the bare-number
branch fires only when no explicit status contradicts it. The status is read off
an arbitrary exception, so a non-integer is treated as unknown and left to the
existing behaviour. The repo has a single call site.
The phrase branches are untouched, so a provider reporting a real rate limit in
the message text under a non-429 status still maps to RateLimitError (11455).
This is not "status code wins".
Tests cover the matcher (suppressed under a 400; still detected with no status,
None, 429, or a non-integer status; phrase honoured under a 400) and
exception_type end to end (400 with 429 in the echoed body -> BadRequestError,
real 429 -> RateLimitError). Reverting the source change fails the latter.
* fix(main): an explicit provider outranks a known OpenAI model name
completion() picks the OpenAI handler whenever `model in
litellm.open_ai_chat_completion_models`, and that clause is evaluated before the
gemini and vertex_ai branches. get_llm_provider() already resolves those names
to "openai", so the clause only adds anything when the provider is something
else, and then it silently overrides it: the config built for the requested
provider is handed to the OpenAI handler.
For gemini that is fatal. VertexGeminiConfig.transform_request raises
NotImplementedError by design, since Vertex builds its request in its own
handler, so `gemini/gpt-4o` dies in async_transform_request before anything is
sent. register_model() reaches the same state without an odd model id: an entry
claiming litellm_provider "openai" adds its name to
open_ai_chat_completion_models, so one mislabelled pricing entry reroutes every
later call to that model in the process.
The name clause now applies only when no other provider was resolved.
* test(main): move the routing regression into the mapped test file
CLAUDE.md asks bug fixes to extend the mapped test file, so these belong in
tests/test_litellm/test_main.py rather than a module of their own.
They also no longer swap out the provider handler objects. Both Gemini cases
inject an HTTPHandler whose post() answers like generativelanguage does, then
assert the URL the request went to and read the reply back; the OpenAI case
injects an OpenAI client and patches its own raw-response create. That asserts
the endpoint the call reaches instead of which attribute the test replaced, and
matches the neighbouring tests in the file.
* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery
* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils
* fix(proxy): make model_list request param optional for direct callers
* style: apply ruff format to changed lines
* style: satisfy ruff strict-rule budget (UP006, I001)
* style: satisfy type-discipline budget (LIT002 mutable-ok, LIT009 pyright ignore)
* style: satisfy LIT001/LIT010 and drop explanatory comment per contributor rules
* fix(proxy): translate team model names in the Anthropic /v1/models response
* ci: trigger buildkite status report
* feat(proxy): carry token limits into the Anthropic-native /v1/models entries
* fix(proxy): cast the injected request so the anthropic-version guard is a real comparison
* fix(proxy): explain the model listing casts so the type-discipline gate passes
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Rebuilds the thirteen form-free components under common_components on
the in-repo shadcn layer, so they inherit the dashboard's design tokens
instead of styling themselves through Ant Design and Tremor.
SearchSelect and the three dropdowns that wrap it now forward an
optional input id, so an antd Form.Item label still resolves to its
control. The e2e steps that reached into antd's Select and Modal
internals now go through the test id, role and data-slot.
Rebuilds public_model_hub, MakeSkillPublicForm, ModelSelect and the
guardrail LogViewer on the in-repo shadcn layer, so they inherit the
dashboard's design tokens instead of styling themselves through Ant
Design and Tremor.
Public prop signatures are unchanged, so no caller moves. The two
teams e2e steps that reached into antd's Select internals now drive
the combobox through its test id, role and data-slot instead.
Request metadata carries the whole UserAPIKeyAuth object, whose team_metadata
holds the customer's own langfuse callback_vars. The only filter on the emitted
blob was a four key deny list written as a circular reference crash guard, so
those credentials reached the customer's own langfuse traces.
The emitted blob is now the StandardLoggingPayload allowlist plus the litellm
computed enrichments, and nothing is copied across from raw request metadata.
That makes the credential exclusion structural rather than a filter someone has
to keep correct. Steering keys keep reading raw metadata, matching literal_ai.
Proxy callers are unaffected: their request metadata already rides under the
allowlisted requester_metadata key, nesting intact.
debug_langfuse dumped raw request metadata into the trace as a second copy of
the same leak. It now emits caller scalars only.
When StandardLoggingPayload is absent the trace is still emitted with the
existing trace_id fallback, so failure traces survive.
* fix(proxy): track spend for OpenAI passthrough /v1/embeddings
OpenAI passthrough embeddings returned 200 but wrote no spend because the
route was unsupported and Cohere's /v1/embed prefix stole the match.
* fix(proxy): clear embeddings lint and Greptile comment nits
Inline embeddings cost tracking to avoid new LIT001/002 hits, trim
redundant doc comments, and cover the Cohere /v1/embeddings collision.
* fix(proxy): drop unreachable embeddings TypeError guard
convert_to_model_response_object with response_type=embedding already
returns EmbeddingResponse; the isinstance check was dead patch coverage.
A deployment with PTU flat-cost attribution also billed every request per
token, so a team paid for reserved capacity and again for the traffic that
capacity serves. Nothing set the per-token price and an unset price falls
back to the public cost map, which made the double charge the default.
/model/new and /model/{id}/update now store zero for every pricing field the
cost map could otherwise fill, refuse a price the caller supplies alongside
PTU config with a 400 naming the field, zero a price already on the row
rather than rejecting later edits of unrelated fields, and drop the zeros
again when the PTU config goes.
A PTU deployment is no longer read as a free model by the budget checks,
which would have waived every budget for it.
* fix(mcp): expose client HTTP headers to logging callbacks and hooks
MCP protocol tool calls built a synthetic Request with only content-type, so metadata.headers reaching logging callbacks and guardrails was empty while /mcp-rest/tools/call exposed the full set. Rebuild the synthetic request from the connection's raw headers (shared with the sampling path), and pass sanitized headers to the pre-call hook, the MCP to LLM guardrail bridge and the Responses API MCP bridge. Credential headers stay masked and proxy key headers stripped.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and upstream MCP credential headers from logging copies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): make client side auth header name accessor public
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): strip custom proxy key and client redaction opt-out from mcp headers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): drop custom proxy key header in the synthetic request builder
Strips general_settings.litellm_key_header_name in build_synthetic_mcp_request so every caller, including sampling, is covered, and reverts passing general_settings into add_litellm_data_to_request on the tool call path since that also switches on enforced_params.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
The Admin UI's Authorize & Fetch Token flow stored its pending server in a
module-level dict, so /register, /authorize and /token only succeeded when
every leg happened to land on the process that served /session. On a proxy
with NUM_WORKERS greater than 1, or more than one replica, each click was an
independent draw and failed with a bare 404, which reads as intermittent.
Persist the pending server as a short-lived draft row instead, so any worker
resolves it. The in-memory cache is kept as the fallback for proxies with no
database configured, which keeps single-process deployments working as before.
A session runs under a caller-supplied id only when that id names a server
that really exists, which is the edit form re-authorizing a saved server.
Anything else gets a fresh id, so two concurrent sessions can never share one
draft and silently adopt each other's URL or client credentials. Drafts past
their lifetime are swept on each write so abandoned sessions do not
accumulate, and a lost create race adopts the winner rather than failing a
caller whose session is ready.
Drafts are excluded from listings and never enter the runtime registry. The
exclusion keeps rows whose approval status is NULL, which both short spellings
of the filter drop, silently hiding every server predating the approval
workflow.
Measured on a two-worker proxy against the live GitHub MCP server, 120
concurrent authorize calls per leg: staging 56/120 failures, this branch
0/120, staging again 65/120 as a positive control.
* fix(team): sweep dangling team references and cache on team delete
delete_team drove all of its cleanup off the team's members_with_roles roster, so any
user row referencing the team by another route kept a dangling team id forever and the
deleted team stayed visible on /user/info. Nothing swept LiteLLM_UserTable.teams or
LiteLLM_TeamMembership by team id, schema.prisma declares no relation between the
membership table and the team table so there is no cascade to fall back on, and the
cached team object was never invalidated on delete.
Adds a sweep that runs before the team rows are dropped: it strips the deleted ids from
every user row that still lists them and removes every membership row for those teams.
Adds _delete_cache_team_object in auth_checks and calls it per deleted team so the
team_id:{team_id} entry cannot outlive the team.
The sweep is targeted, not indiscriminate: only the deleted ids are removed and the
other teams on a user record are left intact.
* fix(team): fail member_add when the team is deleted under the row lock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(team): correct the post-delete sweep note for the member_add lock path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(azure_ai): add Fireworks FW model pricing on Azure AI Foundry
* fix(azure_ai): drop incorrect FW-Kimi-K2.6-Code alias
* test(azure-ai): assert FW max token metadata
* feat(azure_ai): add Inkling and Nemotron 3 Ultra pricing
* fix(cli): hide codex and opencode from the lite command listings
They stay registered and invokable, so existing `lite codex` users keep
working; they just no longer show up in `lite --help` or the interactive
shell's command list.
* feat(cli): make the hidden lite command list configurable
codex and opencode are supported, so hardcoding them as hidden was wrong. Let deployments curate their own listing with `lite config set hidden_commands codex,opencode` instead; nothing is hidden by default and hidden commands stay invokable.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
os.exec* has no process-replacement semantics on Windows, so `lite claude`
printed its routing line and returned to the prompt while Claude Code was left
detached without a usable console. Windows now spawns the agent, waits for it,
and exits with the child's status. Batch shims such as the npm-installed
claude.cmd go through cmd.exe because CreateProcess cannot run them directly,
and that command line is emitted verbatim with every token quoted so a spaced
path or an argument holding a shell metacharacter cannot be re-parsed by the
command processor. POSIX keeps using os.execvpe unchanged.
/team/member_delete dropped the roster entry by matching user_email against
members_with_roles, then built its user-row lookup from that same raw email
instead of from the user_id the roster entry already carries. An email the user
row does not literally hold matched nothing, so the team id stayed in the user's
teams array and the team-membership row was left orphaned while the call still
returned 200.
/team/member_add resolves an email to a user case-insensitively but stores the
caller's casing on the roster, so inviting "Alice@Example.com" for a row holding
"alice@example.com" and removing by that same string is enough to reach it.
_cleanup_members_with_roles now returns the roster entries it removed, and both
the user-row update and the membership delete run against their user ids.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
CLI session tokens minted by /sso/cli/poll set team_id and team_alias but
never team_models or team_model_aliases, so the token carried a team with
none of that team's grants. /v1/models bails out to "unrestricted" when both
key_models and team_models are empty and listed the whole proxy, and team
model aliases never resolved because both can_team_access_model and the
pre-call rewrite read team_model_aliases off the token.
The team data was not close at hand: _fetch_cli_sso_team_details projected
full team rows down to team_id and team_alias before they reached the mint.
Widen that projection to include the team's models and its joined alias
table, and populate both fields at mint time.
Also stop writing the user's personal allowlist into the key models slot
when a team is bound, matching virtual-key semantics where a team-bound
credential is governed by the team grant.
Because an empty team grant is itself a real value meaning unrestricted, a
team whose grants cannot be resolved must not be minted as empty: that is
the same "unrestricted" bail-out this fix exists to close. The poll now
refuses to mint when the selected team has no complete cached detail.
That refusal is only safe because a login can no longer be pinned to a team
whose grants will never resolve. Deleting an organization drops its team
rows but leaves the memberships behind, so the login now offers only teams
whose rows still exist, and a lookup that fails outright fails the login
rather than caching a session that silently drops every team.
Against a real Postgres the previous commit still died on MonthlyGlobalSpend:
only 2 of the 8 creation sites went through the tolerant helper, so the losing
replica re-raised on the first unguarded one and skipped the rest.
The regression test now makes every CREATE lose the race and asserts all 8 are
still attempted, which fails on the partial fix.
Every replica booting against the same fresh database sees each view as
absent and issues the CREATE. Postgres fails all but one with a
duplicate-object error, and that exception propagated out of
create_missing_views, so every view after the first was never created and
/global/spend* 500'd for the life of the deployment.
Losing that race reaches the desired end state, so treat it as success.
Genuine DDL errors still propagate.
Every case in TestAccessControl asserted that something was refused. A gateway
that denied the allow-listed model too would have passed all of them, so the
suite could not tell "denied correctly" from "broken outright".
Adds the positive half: a key allow-listed for gemini-2.5-flash can call it and
gets back a real completion rather than a 200-wrapped error.
Also tightens the unknown-model case. It accepted any valid JSON, so a bare
"{}" or even "null" satisfied it. It now requires the OpenAI-shaped error
envelope with a message a client can actually surface, parsed through a typed
model instead of json.loads.
* fix(proxy/batches): stop forwarding custom_llm_provider twice in list and cancel
The model-routing branches of list_batches and cancel_batch passed
custom_llm_provider as an explicit kwarg while also leaving it inside the dict
they splat, so every such call raised "got multiple values for keyword argument
'custom_llm_provider'" and returned a 500.
list_batches SCENARIO 2 called data.update(credentials) but never removed
custom_llm_provider before litellm.alist_batches(custom_llm_provider=..., **data);
it now uses prepare_data_with_credentials, the same helper the create and
retrieve branches already use, which pops it out.
cancel_batch SCENARIO 3 resolved the provider with
`provider or data.pop("custom_llm_provider", None) or ...`, so when the path
param provider was set the pop short-circuited and a body custom_llm_provider
stayed in data and collided with the explicit kwarg. The body value is now
popped unconditionally before the fallback chain, so the path param wins cleanly
and data no longer carries a duplicate.
Both paths already had strict-xfail regression tests documented "remove when
fixed"; those markers are dropped so the tests now guard the fix.
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
* fix(proxy/files): avoid duplicate custom_llm_provider in list
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
---------
Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
