* feat(guardrails): map each guardrail scan id to its guardrail, stage and provider
Adds the x-litellm-guardrail-scan-metadata response header, a JSON list of
{guardrail, stage, provider, scan_id} entries, next to the existing
comma-separated x-litellm-guardrail-scan-id header. Prisma AIRS records the
execution stage for every scan and OpenAI Moderation now records its
moderation id too. The new metadata key is internal: client-supplied values
are stripped and it is exposed through the UI CORS allow list.
Resolves LIT-6018
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): cap the scan metadata response header at a configurable length
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): hardcode the scan metadata header cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model-management): honor an explicit null as a clear on model update
PATCH /model/{model_id}/update merged the patch with exclude_none and then
popped explicit nulls only for the mirrored pricing fields, so a null sent for
max_input_tokens, mode, supports_vision or any other key was dropped and a value
pinned by an earlier save could never be removed.
The route now follows JSON Merge Patch over both blobs: a key absent from the
body is unchanged, a key sent as null is removed from the stored row, and a key
sent with a value is set. Ownership and identity keys keep ignoring a null, as
do the fields the stored models require, since clearing one writes a row no
reload can rebuild. Mirrored pricing keys still clear from both blobs.
Clearing a price also needed the router to stop merging a deployment's cost-map
entry onto its previous registration, which left the old rate in place and kept
billing at a price the deployment no longer carried.
Adds a create, read, partial-update, clear, enforce, delete lifecycle e2e that
reads back on every replica, and a harness helper for that read-back.
* fix(router): keep a deployment id that names a real model from evicting its catalog entry
Deployments are keyed into litellm.model_cost alongside the built-in catalog, so
evicting a deployment's stale entry by id could take a real model's entry with it:
registering a deployment whose model_info.id is "gpt-4o" stripped that model's
pricing, context window and capability flags process-wide, for every other
deployment of it, until the next price-map reload.
Only evict an entry this registration owns. A colliding id keeps the previous
merge, which pollutes the catalog entry rather than emptying it.
Also pins the Admin UI round trip: the model edit form echoes the whole /model/info
row back on save, and that read reports every key the deployment never stored as an
explicit null, so the clear path has to leave those keys alone.
* fix(router): decide cost-map eviction by what this registrar created
The previous guard read a catalog entry off `litellm_provider`, so a deployment
that declares its own provider in model_info was treated as one and kept billing
at a price it no longer carried. It also only held for a single registration: a
second one under a colliding id saw the id the first merge left behind and
evicted the catalog entry anyway.
Track the cost-map keys this registrar creates instead. A key it created is
evicted before re-registration; one it did not is left to merge, which is what a
deployment id colliding with a catalog model name needs.
Also folds the required-fields comment into the docstring that already gives the
reason.
* fix(router): release a deployment's cost-map key when it is deleted
The ownership ledger only grew. A deleted deployment kept its claim, so if a
later catalog refresh started publishing a model under that same name, the next
registration would treat the catalog entry as the deployment's own and evict it.
Deleting a deployment now gives the key back, which also stops the ledger
growing for the life of the process.
* fix(router): hold a cost-map key while another live router still serves it
The claim is process-wide but the release was per-deletion, so with two routers
serving one deployment id, the first deletion put the survivor back on merging
and the price it had just cleared would keep billing.
Release the key only once no live router still serves that id.
* fix(router): register a router in the live set when it gains a deployment
_live_routers was only joined when a router was constructed with a model_list,
but a router built empty is populated through add_deployment, and the empty
branch exists for exactly that. Such a router was invisible to the live-router
scan, so deleting the deployment from another router released the shared
cost-map key while it was still serving that id.
Joining the set where a deployment enters the list covers every path, and it
also lets a price reload rebuild what a dynamically built router serves.
* fix(e2e): read the stored model row from the control plane, not each gateway
The lifecycle suite polled /model/info on every URL in PROXY_REPLICA_URLS. Those
URLs are the stack's gateways, and gateway/routes/allowlist.py trims them to the
LLM data-plane surface, so /model/info answers only on the backend and 404s on
every replica. All five tests failed at their first read-back in CI while passing
against a monolith, where one process serves both planes.
The stored row has one answer behind it, so it is read through the shared
transport, which routes control-plane paths to the backend. What every gateway
must agree on is which models it serves, so the create and delete steps poll
/v1/models per replica instead, a route the gateway does serve.
read_back_everywhere now rejects a control-plane path outright rather than
timing out on it.
Two things surfaced behind that. /public/ was missing from the transport's
control-plane prefixes, so model_cost_map() was routed to a gateway and 404'd,
and the billing steps needed a data-plane wait: a PATCH lands on the backend and
each gateway picks it up on its own config reload, measured here at 12-24s, so
they now drive calls until the new rate reaches the spend row and let the
deadline fail them.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C1S92J8gSxxKVe1JBzxWBF
* test(models): keep polling outcomes immutable and document shared ownership
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* fix(mcp): honor an explicit null on toolset update, cover MCP lifecycle e2e
PUT /v1/mcp/toolset dumped its payload with exclude_none, so a field sent as
null looked exactly like one the caller left out and the stored value
survived. An admin could not clear a toolset's description: the save reported
success and the old text came straight back. It now dumps with exclude_unset,
so absent keeps and null clears, which is what PUT /v1/mcp/server already did.
A null tools list clears the selection to empty, and a null toolset_name is
ignored because a toolset always has a name.
Adds create, read, partial-update, clear and delete e2e coverage for MCP
servers and toolsets, with every read-back polled on every replica so an edit
that lands on one replica and not another fails the test, plus an enforcement
test proving a key granted a toolset lists exactly that toolset's tools
against the real Datadog upstream.
* fix(e2e): refuse a read-back that no replica serves
A read-back over an empty replica mapping satisfied every predicate and
returned as if it had converged, so it would have asserted nothing and
passed. No wiring can produce that today, since the replica list always
falls back to at least one URL, but a helper whose whole job is proving a
write reached every replica should not have a shape that passes vacuously.
* fix(mcp): keep a null tools list a no-op on toolset update
Treating a null tools list as a clear meant an existing client that sends
tools=null during a partial update, meaning "leave the selection alone",
silently lost every tool the toolset grants. That is a permission surface,
so the quiet version of it is the worst version.
A toolset always has a tool list, the same way it always has a name, so a
null on either is now a no-op. Emptying the selection is an explicit [],
which cannot be confused with a field the caller left out, and which is
what the dashboard already sends.
* fix(e2e): keep MCP admin routes on the data plane
/v1/mcp/* is a lazily mounted feature, so a gateway registers it on the first
matching request, which happens after the startup route trim that drops
management endpoints. Routing it to the control plane therefore sent every MCP
call to the one backend process: the new lifecycle read-backs proved a single
process rather than every replica, and mcp_client's await_registered barrier
waited on a registry that does not serve the tools/list call it guards, so the
existing MCP suites polled a gateway that had not synced yet until poll_timeout
Verified against a two-gateway split stack (backend on 4001, gateways on 4010
and 4011, one postgres): both gateways answer /v1/mcp/server and /v1/mcp/toolset,
and each served 6 server reads and 7 toolset reads over the run
* fix(e2e): grant the toolset by the tool's own name, not the wire name
tools/list serves a tool as <prefix><tool_name>, but a toolset grants by the
tool's own name: resolve_toolset_permissions reads toolset.tools[].tool_name
straight through, and the prefix is added on the way out. The test built the
toolset from the names tools/list reported, so the grant matched nothing, the
scoped key listed no tools, and await_tools ran out its whole poll_timeout
before failing
Measure the prefix off search_datadog_logs, whose own name is known, rather than
guessing it from the alias, since the proxy can be configured to prefix with a
short server id instead. The expectation compared against tools/list stays in
wire names; only what the toolset stores crosses back
* test(mcp): build immutable lifecycle updates and replica results
* test: validate opaque stream IDs and hide log-reader credentials
* test: isolate auto-router scenarios and clean partial setup
* test: honor Datadog search rate-limit reset headers
* test: share the Datadog read-back deadline across retries
* test: preserve captured MCP toolset update fields
* test(e2e/ui): cover key budget window, non-admin model scope edit, and key blocking
Three Playwright specs for the Virtual Keys flows customers hit most, each
reading its result back through /key/info and /v1/chat/completions rather
than trusting the toast:
- a monthly spend cap and reset window set through Edit Settings, surviving
a reload, with clearing the window leaving the cap in place
- a team member narrowing their own team key's models, and the proxy
refusing the model they dropped
- blocking a key from its detail page, then unblocking it
Each test owns the key it edits and deletes it on teardown, so retries and
--repeat-each never run out of fixtures.
* test(e2e/ui): tighten virtual key specs from review feedback
Replace the mutable suite-level key state with a Playwright fixture, so the
alias and token are never reassigned and cleanup stays tied to the test.
Assert /key/delete succeeded instead of discarding the response, so a failed
cleanup surfaces rather than leaving rows behind.
Drop the explanatory JSDoc the repo's comment policy disallows, keeping only
the one line explaining why Date.now() alone is not unique enough.
Type the master-key POST helper against a real guard instead of casting to
Record<string, any>.
Assert the unblocked key is served with a 200, not just the response text,
and that clearing the reset window also clears budget_reset_at.
* test(ui): assert the team response through Playwright
* test(e2e/ui): cover team-scoped model visibility, re-editing litellm params, and model health checks
Three Models and Endpoints flows had no end-to-end coverage, and all three
keep coming back as bug reports.
modelsByTeam walks an internal user through the Current team control and
asserts the table lists exactly what each team grants. It creates one
deployment that belongs to no team, proves that deployment is visible under
Personal, then proves it is absent under both seeded teams, so an empty
table cannot pass the same assertions.
editLitellmParams adds a temperature and a custom pair to a deployment,
saves, then re-edits the temperature and drops the custom pair. It checks
both update request bodies, polls the stored deployment until the new
temperature is there, reloads the page to confirm the second save is what
renders, and sends one chat completion to prove the deployment still serves.
modelHealthStatus runs the health check on a reachable deployment and on one
pointed at a dead port, asserts the healthy and unhealthy cells and the two
detail dialogs, and reloads to confirm both statuses are stored.
Every deployment these specs create carries a unique name and is deleted in
afterEach, including on the failure path.
* test(e2e/ui): find health rows across every page of the health table
The health table pages server-side at 50 rows with no search box, so on a
proxy carrying more deployments than that the two deployments the spec
creates can land on a later page and the lookup finds nothing.
Row lookups now walk the pages, using the table's own page indicator to
know when to advance and when to wrap back to the first page.
* test(e2e/ui): build the created deployment ids without mutating the array
* test(ui): scope model deployments to Playwright fixtures
The streaming pipeline step only takes a post-call hook on routes whose
translation assembles the streamed response (chat completions, Responses,
Messages). On /v1/completions, the Gemini streamGenerateContent route, and
A2A streams the pipeline is skipped with the merge-base warning and the hook
runs on its own afterwards, instead of getting a None response while the
header says the guardrail ran. A guardrail that overrides
async_post_call_streaming_iterator_hook next to its post-call hook keeps its
native per-chunk path rather than running buffered through the adapter
A client that named a bare gateway alias logged the session as "unknown" and billed nothing,
because the model was read off the raw setup frame and the extractor only yields a name when the
string already contains "/models/". The rewriter qualifies that same model a few lines later for
the upstream, so the supported client form, an alias, was the one that went unbilled.
Resolving through the rewriter first means the real model reaches the logging object, and from
there the cost map. A route with no rewriter, which is every non-Live passthrough, hands the frame
over untouched.
(cherry picked from commit 573982803df612fd94144e2e06dd647f8530d4e8)
Live reports grounding in the server frames and never in usageMetadata, so nothing
set the counter the cost path reads and the per-query charge was missing from every
grounded session. Google bills a grounded Live prompt on top of its tokens, and that
fee dwarfs the token cost, so a non-zero spend check could never catch it.
Both Live surfaces now read serverContent.groundingMetadata where they build usage,
and reuse the chat path's own classifier so web search and Maps keep their separate
SKUs rather than being counted together.
Separately, a client sending turn_detection: null reached a membership test against
None and took the session down with no traceback, while the branch immediately above
already guards for it. Live emits grounding and usage on the same frame, verified
against Vertex directly, so the realtime counter is set where usage is built.
(cherry picked from commit c997436be34beb2e84f8468286b8016c195eca92)
(cherry picked from commit 26c8d4822fc1c5c44fe8f72f4f57473a2ce1acbf)
The four session helpers this PR added were unannotated. Typing them needs a
name for the (text, audio) pair each turn carries, so _LiveTurn is a TypedDict
rather than a Mapping union that would leave sum() over a prompt pair
ill-typed, and AUDIO_SESSION is declared with it. The message list reuses
list[dict[str, object]], the annotation the passthrough already uses where it
collects those messages
toolUsePromptTokenCount was the one prompt-side total not named in
_AGGREGATED_FIELDS, so it rode the unknown-key pass-through and took the first
frame's value while promptTokenCount, candidatesTokenCount and totalTokenCount
beside it were summed. Live's frames grow over a session, so the first frame is
the smallest number in the series and a grounded session under-reported its
tool-use tokens by everything after turn one. It is now summed like its three
neighbours.
This is reporting only, and pricing these tokens is deliberately left out. Google
charges tool-use prompt tokens at the input token rate, but generic_cost_per_token
reads the input bill out of prompt_tokens_details and only falls back to
prompt_tokens when the details carry no text or a cache hit overlaps them.
Measured on the native-audio entry with 500 tool-use tokens: adding them to
prompt_tokens moves an ordinary Live turn's bill by $0.0000000000, and on a turn
with a cache hit it moves it by $0.0002650000 where the tokens are worth
$0.0002500000, because it perturbs the cache-overlap correction. Pricing them
belongs beside the modality terms in the shared input-cost path, in its own
change that fixes the same latent no-op on the ordinary Gemini path.
Not verified against a live capture: no Vertex Live session we have captured
reported toolUsePromptTokenCount at all, so the summing convention is inferred
from the three prompt-side totals that accumulate the same way.
The two helpers this branch adds took Sequence[Mapping[str, Any]], which the
repo forbids, and only typechecked because Any is compatible with everything.
Both now take Mapping[str, object] and the raw *TokensDetails value is narrowed
to its mapping entries at each of the three call sites.
TypedDicts are the wrong tool here: _merged_modality_totals reads count_key and
details_key as runtime strings, and the aggregation deliberately passes unknown
keys straight through, so both need a mapping whose keys are not literals.
The narrowing is not cosmetic. The handler's only failure path returns no result
at all, so a *TokensDetails value that was not a list of objects used to raise
while being read and cost the whole session its bill.
The Live passthrough builds Usage from the TEXT-modality counts alone, so audio,
image and video tokens never reach the cost calculator and bill as nothing. A
one-turn audio session reported 13 text and 127 audio input tokens and billed the
13; a camera session reported 1043 prompt tokens and billed 11.
Reporting the full per-modality breakdown fixes it, because the shared Gemini
input and output cost path already prices audio, image and video from
prompt_tokens_details and completion_tokens_details. On the native-audio entry
that is a 6x difference per token in both directions, which is the whole gap.
Aggregation across turns is unchanged. Google charges per turn for every token in
the Live session context window, current turn plus all accumulated tokens from
previous turns, so the existing summing is what Vertex bills and it stays as it
is. That is worth stating because the cumulative promptTokensDetails looks like a
restatement of one running total, and treating it that way would under-bill a
multi-turn session. See the LiveAPI context-window note on
https://cloud.google.com/vertex-ai/generative-ai/pricing.
Live can also name the modality carrying the rest of a turn and omit its
tokenCount. Reading that absent key as zero left the tokens inside
candidatesTokenCount but outside the breakdown, so real speech was charged at the
text output rate. A lone unpriced entry now takes whatever the turn's declared
count leaves over. Two or more cannot be told apart, so they are still left to the
calculator's text remainder.
Server-side toolUsePromptTokenCount is now reported in prompt_tokens_details. It
is deliberately kept out of prompt_tokens: no Gemini route prices tool-use tokens,
and adding them there instead suppresses the cache-overlap correction and raises
the bill for no extra work.
Removes _calculate_live_api_cost, whose result never reached the bill. It set
kwargs["response_cost"], which the standard logging path recomputes from the
ModelResponse, and on a measured audio session it returned $0.000487 against a
$0.0000425 row. Now that the modality counts reach the standard calculator,
keeping a second hand-rolled pricing path would only ever double-charge.
The rewrite of the aggregator is arithmetically identical to what it replaced. It
sums the same three counts and the same per-modality details, still takes the
remaining fields from the first turn, and drops nine LIT010, one C901 and 42
basedpyright findings in the process.
Reasoning items the Responses API returned for a /v1/messages turn were rebuilt
from their summary text on every replay, so the prompt the model saw changed
between turns and the prompt cache never matched. The bridge now asks for
reasoning.encrypted_content, carries it in the thinking signature (or as a
redacted_thinking block when there is no summary), and replays it verbatim as
the reasoning item's encrypted_content. Anthropic replay paths drop those
tagged blocks so a cross-model resume never forwards OpenAI bytes to Anthropic
Replace the thread semaphore around HuggingFace encodes with an anyio CapacityLimiter
applied at every offloaded count site through offload_token_count, so waiting counts no
longer hold slots in the shared 40-thread pool and inline counts on the event loop never
block on the bound. Rename TOKEN_COUNTER_MAX_CONCURRENT_HF_ENCODES to
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS
get_optional_params_image_gen only forwarded the per-call drop_params flag
to provider configs, so litellm_settings drop_params: true never dropped
the n the MAI generations endpoint ignores. The MAI edits config also
advertised and forwarded size, which that endpoint ignores. Non-numeric
and non-positive n now surface as a 400 instead of a 500 or a pass-through.
Every test this PR adds now annotates its fixture and parametrize
parameters. The submit-time warning for a tag-matched deferred policy
names only the policies, since a wildcard attachment pattern would let
caller-provided tag text reach the log.
A guardrail that removes or adds a tool call on an ended stream used to be
silently ignored: every handler substitutes the original list on a count
mismatch and the executor skipped its observer once the translation could
deliver rewrites. The executor now tracks the count change on the observer
and releases the original chunks with the discard warning on every
translation, matching what the merge base did for any tool call rewrite
