Merge deployment model_info into a copy of the lru_cache'd get_model_info() dict and drop unset Nones, so Deployment's mirrored pricing defaults no longer overwrite built-in prices process-wide.
Fixes#36980
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two small pod-spec passthroughs the componentized chart was missing, both
additive and empty by default so existing renders are unchanged:
- gateway/backend/ui deployments gain a `startupProbe` knob (same
`{{- with }}` toYaml pattern as liveness/readiness), to gate liveness during
a slow cold start without a kill loop.
- gateway/backend/ui HPAs gain an `hpa.behavior` passthrough rendered verbatim
under spec.behavior (scaleUp/scaleDown policies + stabilization windows).
Tests: extend probe_tests.yaml (startupProbe absent by default / renders
verbatim) and add hpa_behavior_tests.yaml. Full chart suite: 76 tests pass.
Signed-off-by: Louis Vauterin <louis.vauterin@doctolib.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Anthropic's Models API declares max_input_tokens and max_tokens as nullable, not
optional, and the live vendor endpoint returns both keys on every entry. The
merged Anthropic-native listing dropped either key whenever LiteLLM could not
resolve a limit, so a client validating against a nullable-but-required schema
saw a malformed entry for any model the cost map does not know.
OpenAI and Azure GPT-5.x answer a chat request whose output budget cannot fit a
single visible token with a 400, while the same models return a length-truncated
200 one or two tokens higher. Agents that probe a model with a hardcoded
max_tokens of 1 read that 400 as "model unavailable".
The four chat request helpers now recognise the provider's own sentence and hand
back the length-truncated response the provider gives at a slightly larger
budget: finish_reason "length", empty content, zero completion tokens. Any other
400 still raises. Streaming is covered by the same seam, and the caller's budget
is never raised on their behalf.
The provider bills the prompt it processed but sends no usage object with the
400, so the prompt tokens are estimated with the same token_counter every other
usage-less path uses. Reporting zero would let a caller send an arbitrarily
large prompt with max_tokens 1 and be charged nothing.
The Object Permissions section rendered the MCP Servers badge with shadcn's
default variant (solid bg-primary), so a plain count showed up as a black pill
next to the light Vector Stores and Agents counts. Counts now use secondary
everywhere, and destructive stays reserved for the blocked state.
update_trace_keys lets a caller name which request metadata entries get copied
onto an existing trace, and the name is unrestricted. Sending
update_trace_keys: ["user_api_key_auth"] with existing_trace_id serializes the
resolved auth object, including the team callback credentials it carries, onto
the trace through Langfuse.trace(**trace_params). TraceBody is Extra.allow, so
an unexpected key ships rather than being dropped.
Any holder of a team key can do this and read the result in the destination the
team already logs to, so the feature is now inert unless an operator turns it on
with langfuse_enable_update_trace_keys.
This reverts commit ab2333b6c4.
Every Admin UI login mints its session key against the sentinel team_id
`litellm-dashboard`, and no LiteLLM_TeamTable row is ever created for it.
That lookup is therefore a provably-absent row on every UI request, which
#36837 turned into a hard refusal with no override, so the whole dashboard
404s.
Reverting restores the token-derived fallback. The model-access widening
#36837 closed is reopened and needs a re-land that exempts the UI sentinel
team.
Keeps the spend buffer restore path free of mutable-collection construction so the type discipline gate stays within its LIT002 ceiling.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
cost_per_second treated a declared-but-zero output_cost_per_second as a real
rate, so the output branch claimed the call and the elif locked out
input_cost_per_second. Every transcription model shipping
output_cost_per_second 0.0 next to a real input rate billed $0, which covers
43 of the 55 per-second entries in the cost map: all 36 deepgram models, both
assemblyai, both elevenlabs scribe, both groq whisper and azure-stt. Custom
deployments pairing the two fields the same way billed $0 as well
Take the output branch only when that rate is actually billable, so a zero
falls through to the input rate. Entries that duplicate one rate into both
fields, whisper-1 among them, keep billing exactly what they bill today
`_query_first_with_cached_plan_fallback` recovers from Postgres's "cached
plan must not change result type" by recreating the Prisma client, which
drops both the server-side plans and the engine's client-side statement-name
cache. Since #30183 the shared reconnect path probes the writer with
`SELECT 1` first and skips the recreate when it answers, which is right for
the IAM token refresh it was added for and wrong here: the connection is
healthy, it is the session's prepared statements that are stale, so the probe
always passes and always vetoes the recreate. Callers now pass
`force_recreate` to skip that probe, and only the cached-plan fallback does.
Getting past the probe is not enough on its own. Both cooldown checks would
still skip the recreate for 15 seconds after any earlier reconnect, which
outlives the 10 second auth retry window, so a migration landing in that
window kept 503ing. `force=True` would fix that but would also let every
concurrent caller of the same burst kill the engine the first one just built.
The caller instead names the engine it observed before the query, and the
cooldown is waived only while that engine is still the live one, so the first
caller repairs the pool and the rest fall back to the normal cooldown.
That engine has to be the one the query actually ran on. `query_first` is a
top-level read, so with a read replica configured it is dispatched to the
reader and it is the reader's prepared statements that go stale, while
`writer_db` names a different engine with its own counter. The observation
and the cooldown comparison both go through `read_db`, added alongside
`writer_db` and backed by a `read_target` property on the routing wrapper
that `__getattr__` now dispatches through so the two cannot drift.
The observation carries the wrapper, not just its generation. `read_db`
resolves to the reader while it is available and to the writer once it is
not, and those counters are independent and both start at zero, so comparing
a bare number across that switch pits one engine's counter against another's.
Equal by coincidence waives the cooldown for an engine already replaced;
unequal gates a caller that needs the recreate. Identity settles it, and is
sound because the engine object is never re-pointed without the generation
also moving.
Three smaller holes on the way out. The waiver is withdrawn once a repair of
that same engine has been tried and failed, so a burst collapses onto one
attempt instead of each caller running its own recreate serially; the record
is keyed per engine rather than counted globally, so an unrelated reconnect
failure cannot suppress a stale reader's recovery and a writer failure cannot
evict the reader's record. And a forced recreate that the optimistic-lock
guard declines is no longer reported as a success on either the direct or the
heavy path, since the routing wrapper leaves the reader untouched in that
case; a decline is deliberately not counted as a failure, so the caller's own
backoff still gets its waiver on the next attempt.
A decline on the heavy path clears the dead-engine flag before raising. The
clear after the cycle is skipped by any raise, which is right for a failure
and wrong here, and the non-forced path already clears it on a decline, so
this restores that policy rather than inventing one. Stranding the flag would
route the next cycle back down the probe-free heavy branch, where the
refreshed generation matches and the recreate kills the healthy engine a
refresh just spawned, which is #29176.
Clearing that flag is necessary and not sufficient. The escalation check
re-arms it whenever the consecutive-failure count sits at the threshold, so a
decline that left the count alone sent the very next attempt back down the
same path. A decline is raised only at the generation guard, and the
generation moves only after a replacement connects, so a decline is proof
that a replacement succeeded and the count is reset on it.
Fixes#36418
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The two new `translate_tools_to_responses_api` calls carried
`# type: ignore[arg-type]`, which CLAUDE.md bans as LIT009: pyrightconfig.json
sets enableTypeIgnoreComments to false, so the comment silently does nothing and
the reportArgumentType error stands. Annotating the fixtures as
list[AllAnthropicToolsValues] makes both calls check clean with no suppression
at all.
MCP tool calls run their guardrails against a throwaway LLM-shaped dict
built by `ProxyLogging._convert_mcp_to_llm_format`, not against the dict
the tool call is logged from. `@log_guardrail_information` therefore
appended `standard_logging_guardrail_information` to that throwaway
dict's metadata bucket, where `get_standard_logging_object_payload`
never saw it, so the Guardrails Monitor reported zero evaluations and
zero blocks for all MCP traffic.
Thread the request's `litellm_logging_obj` into `pre_call_tool_check`
and `_create_during_hook_task` and bridge the guardrail records onto it:
- Seed `data["litellm_logging_obj"]`, which unified guardrails read and
pass into `apply_guardrail`.
- Call `_sync_guardrail_info_to_logging_obj` in a `finally`, which is
what native guardrails need and what makes the block path work: a
blocked call raises straight out of `pre_call_tool_check`, so the
record has to be attached before the exception leaves the frame.
Only the guardrail evaluation records are copied. The synthetic
request's messages and tool arguments are deliberately left behind --
they can carry end-user data and nothing in the monitor needs them.
In `call_mcp_tool`, flush the failure handlers before
`post_call_failure_hook` so the `status="failure"` standard logging
object exists when `_ProxyDBLogger.async_post_call_failure_hook` writes
the spend-log row the monitor's "Total Blocked" counts. Both handlers
gate on `should_run_logging("sync_failure")` / `("async_failure")` and
then mark it, so the `@client` wrapper's own post-raise logging is a
no-op and nothing is double-counted -- the same pattern
`_fire_mcp_tool_call_logging` already uses for `isError=True`.
Threaded through every MCP tool entry point: the managed-server path,
the local-OpenAPI registry path, the legacy registry fallback, and the
Responses API's `_execute_tool_calls`.
Both charts run schema migrations from a Job that is a pre-install and
pre-upgrade hook, and neither set activeDeadlineSeconds. A migration that
blocks on the database therefore never fails: backoffLimit is not reached
because the pod never terminates, so the Job stays active indefinitely and
the release waits on the hook forever. `helm upgrade` and any GitOps
controller driving it stop reconciling the whole chart until someone
deletes the Job by hand, which means unrelated changes to the gateway, the
backend and the UI silently stop shipping.
Give the field a 1800s default, guarded by `with` so setting it to null
restores the old unbounded behaviour. A migration that has exhausted its
retries is not going to succeed on the next one, so failing is strictly
better than hanging: a failed sync is visible and retryable, a hung one is
neither.
Chart.yaml is deliberately untouched. Recent template-only changes to
litellm-helm did not bump it either.
Translating Anthropic tools left the outbound function-tool `strict` unset,
which the Responses API does not read as non-strict. OpenAI's function-calling
docs say strict mode requires every field in `properties` to be marked
required, and with `strict` omitted the schema gets normalized to satisfy that
instead of being rejected. What users see is a tool whose `required` lists
every property, so models fill optional Anthropic tool arguments with empty
values. Send `strict` explicitly so an unset value stays non-strict and an
explicit `strict: true` still reaches the provider
On the Chat Completions adapter, `strict` was also missing from
`mapped_tool_params`, so a tool-level `strict` was merged into the OpenAI
function `parameters` schema (mutating the caller's `input_schema` along the
way) instead of being set on the function. Map it to `function.strict` and
leave it unset when the caller omits it, since Chat Completions already
defaults to non-strict
is_error_str_rate_limit treats any standalone 429 in the stringified exception as
a rate limit, and for openai-compatible providers that check runs before the
status-code branch. Providers echo the request back in validation errors, so a
400 whose body happens to contain a 429 comes out as RateLimitError.
Tokenised prompts hit this routinely, since 429 is an ordinary token id (" that"
in several tokenisers) and an echoed prompt_token_ids array is enough:
{"error":{"message":"`tools` must not be an empty array",
"type":"invalid_request_error","code":400},
"prompt_token_ids":[9906,429,1234]}
The mislabel is not cosmetic. RateLimitError tells callers and routers to retry,
so a request that cannot succeed gets replayed, and the failure is booked against
provider throttling rather than the caller. Against DeepInfra, one recurring 400
("`tools` must not be an empty array") came back as a rate limit in 77 of 198
occurrences, the split depending only on whether the echoed prompt contained 429.
16482 narrowed '"429" in error_str' to \b429\b after a false positive on
'asbjdad429addad'. Word boundaries cannot separate a real 429 from a token id, so
the same class of false positive survives.
is_error_str_rate_limit now takes an optional status_code, and the bare-number
branch fires only when no explicit status contradicts it. The status is read off
an arbitrary exception, so a non-integer is treated as unknown and left to the
existing behaviour. The repo has a single call site.
The phrase branches are untouched, so a provider reporting a real rate limit in
the message text under a non-429 status still maps to RateLimitError (11455).
This is not "status code wins".
Tests cover the matcher (suppressed under a 400; still detected with no status,
None, 429, or a non-integer status; phrase honoured under a 400) and
exception_type end to end (400 with 429 in the echoed body -> BadRequestError,
real 429 -> RateLimitError). Reverting the source change fails the latter.
* fix(main): an explicit provider outranks a known OpenAI model name
completion() picks the OpenAI handler whenever `model in
litellm.open_ai_chat_completion_models`, and that clause is evaluated before the
gemini and vertex_ai branches. get_llm_provider() already resolves those names
to "openai", so the clause only adds anything when the provider is something
else, and then it silently overrides it: the config built for the requested
provider is handed to the OpenAI handler.
For gemini that is fatal. VertexGeminiConfig.transform_request raises
NotImplementedError by design, since Vertex builds its request in its own
handler, so `gemini/gpt-4o` dies in async_transform_request before anything is
sent. register_model() reaches the same state without an odd model id: an entry
claiming litellm_provider "openai" adds its name to
open_ai_chat_completion_models, so one mislabelled pricing entry reroutes every
later call to that model in the process.
The name clause now applies only when no other provider was resolved.
* test(main): move the routing regression into the mapped test file
CLAUDE.md asks bug fixes to extend the mapped test file, so these belong in
tests/test_litellm/test_main.py rather than a module of their own.
They also no longer swap out the provider handler objects. Both Gemini cases
inject an HTTPHandler whose post() answers like generativelanguage does, then
assert the URL the request went to and read the reply back; the OpenAI case
injects an OpenAI client and patches its own raw-response create. That asserts
the endpoint the call reaches instead of which attribute the test replaced, and
matches the neighbouring tests in the file.
Moves the onboarding views, router settings inputs, tag rate limit editor,
fallback buttons, created-key display and the shared numerical input onto the
in-repo shadcn layer. Each control has a direct equivalent, so this is a
like-for-like swap with no layout changes and no new styling.
Router settings saves by reading input values straight off the DOM with
document.querySelector('input[name="..."]'), a path no test covered. Adds a
regression test that types into a field and asserts the typed value reaches
the payload, so the name attribute contract stays enforced.
Also adds tests for TagRateLimitEditor, which had none and whose RPM cell
switched from antd InputNumber to a native number input.
- Extract input_cost, output_cost, cache_read_cost, cache_creation_cost, reasoning_cost, and tool_usage_cost from logging object cost breakdown
- Populate x-litellm-response-cost-* component headers in ProxyBaseLLMRequestProcessing.get_custom_headers
- Ensure headers are omitted when cost breakdown is absent or values are None
- Add comprehensive test suite covering component headers, math invariants, caching, reasoning, and discounts/margins
Replaces antd Radio, Checkbox and Tooltip, plus Tremor Text and Badge,
with the in-repo shadcn equivalents across the three MCP permission
panels, and drops the no-restricted-imports suppressions they no longer
need. Also removes the stale suppression on settings.test.tsx, which
imports neither library.
The tool rows keep their existing click-to-toggle behaviour: the row
owns the toggle and the checkbox no longer carries its own change
handler, since Base UI replays the click through a hidden input that
reaches the row on its own.
Adds payload-level tests for the risk-group view covering group clear,
mixed-state re-arm, single-tool toggles from both the box and the row,
and a controlled round trip proving each control re-renders from the
permissions it emitted.
DateRangePickerValue is a plain object shape, not a component, so the
twelve files that used it were each carrying a no-restricted-imports
suppression for a type that tremor declares as
{ from?: Date; to?: Date; selectValue?: string }.
Declare that shape in components/shared/date_picker_types.ts and point
every consumer at it, which drops ten suppressions from the baseline.
advanced_date_picker and usage_date_picker keep their tremor imports:
they still render tremor Button, Text and DateRangePicker, and moving
DateRangePicker itself needs react-day-picker.
The pending-state test seeded its deferred promise by declaring the resolver
with let and reassigning it inside the executor. Promise.withResolvers is the
standard way to get the same handle without the reassignment, and the
assertions are unchanged.
Resolve the usage-filter conflicts by reusing upstream's PaginatedSearchSelect for User Usage while preserving server-side search, pagination, loading, and no-results behavior.
Generated with AI
Co-Authored-By: Codex
The discount and margin removal confirmation used AlertDialogAction, which
renders AlertDialogPrimitive.Close and dismisses the dialog on click. The
dialog therefore disappeared while the removal request was still in flight,
leaving the admin with no sign that anything happened and free to fire a
duplicate removal.
Swap the confirm control for a plain destructive Button, track an isRemoving
pending state that disables Cancel and relabels Remove to "Removing...", and
clear the pending removal in a finally block once the request settles.
