* feat(proxy): name the successor model for deprecating deployments
* chore(ui): add successor_model to the generated deprecation api types
* fix(proxy): keep the cost map successor when a deployment overrides only the date
* fix(proxy): read the cost map successor from metadata.successor
The cost map schema rejects a top-level successor_model key, and curated
entries already carry the successor under metadata.successor. Read that key,
and look it up even when the cost map entry has no deprecation date, so a
deployment that sets only its own date still names the successor.
* fix(proxy): keep the first deployment when deduping, let a blank successor hide the cost map one, and name a served successor by its model group
* fix(proxy): name only the deployment's own successor_model, since cost map successors are LiteLLM ids rather than served model names
* fix(proxy): leave out a successor_model the deployment's callers cannot call, with a warning
* fix(proxy): report successor_model as the admin set it and only warn when no model matches it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs
* fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off
* fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off
* test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it
With modify_params on, a trailing tool search turn whose call has a kept result
got a placeholder answer from the factory and went out as a marked tool_use,
but the cache control hook counted marks before that placeholder existed and
skipped the mark, so a request at the cap reached Anthropic with 5 marks and
failed with a 400. The hook now finds client answers on the sanitized messages,
the same list the factory reads.
Also make the replay checks cheaper (prefix check before validation, early
return without provider_specific_fields, and look up client answers only when
a tool call carries cache_control), drop the two audit cells that cover old
gaps outside the tool search fix, and derive the pass-through test timeout
from the graceful stop budget.
* fix(anthropic): keep the duplicate tool result warning out of the cache control hook
The hook finds client answers with only the step of the modify_params sanitize
pass that pairs tool calls with their results, so a request with a duplicate
tool result logs the warning once again instead of once per mark count.
Also restore the Snowflake non-stream and cached-stream tool search cells as
known bugs with pytest.skip instead of deleting them, and put the Anthropic
pass-through test timeout back to 240 seconds.
* fix(anthropic): count cache marks next to a message the pairing step cannot read
With modify_params on and a marked tool call, the cache control hook paired tool
calls with their results before counting. A message without a get method (a
stray string or null in extra_body.messages, or an openai SDK
ChatCompletionMessage) made that pairing raise AttributeError before any
provider call, where main counted the marks and let the request through.
The hook now pairs only when every message can be read the way the pairing
step reads it, and otherwise counts on the messages as sent, like main did.
* fix(anthropic): count cache marks after placing a later system message the way the request builder does
* test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census
* test(anthropic): align the tool search replay cells with main and cover a system message after the search turn
The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882.
The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool
reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell
expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its
mark, and a new cell covers a system message after the trailing search turn in both stream modes
* fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request
* test(anthropic): cover marked tool calls next to a malformed tool id and a later system message
* perf(anthropic): look up client tool answers only when a marked tool call is a server tool call
Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked
tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count
marks
* test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell
* perf(anthropic): place mid-conversation system messages in linear time for models that keep them
* fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time
* test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn
* fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag
* test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router
* fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure
* test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement
* fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests
Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route.
* test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse
* fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow
* test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt
* refactor(test): avoid local list mutation and satisfy type discipline gate
* style(prompt_templates): keep line length under 120 chars for ruff format
* style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401
* style(prompt_templates): eliminate basedpyright errors in function_call_prompt
* fix(prompt_templates): keep tuple and None system content when adding the function prompt
Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged
* fix(prompt_templates): reject unreadable system content without echoing it
Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt
* fix(prompt_templates): leave non-list system content failing as before
None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated
* fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params
* test(prompt_templates): pin the function prompt on every system message
* test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): intercept the bare web_search tool type on chat completions
* fix(websearch_interception): match the bare web_search type only on chat completions
/v1/messages keeps its existing detector, so a non-Anthropic model there no
longer short-circuits a bare web_search tool into raw search results
* test(websearch_interception): cover the bare web_search type on chat and through the messages adapter
* test(integration): wait on the right signal in five flaky integration contracts
The Prometheus shared-label tests waited for the latency metric and then read the
request counter, which the logger writes after an await. They now wait for the counter
The auto-router session poll returned on the first 200 while the caller expected two
turns, so it now waits for the expected turn count
The S3 queue cap test let its failing sink recover on a 15s clock that could expire
mid-flush. The sink now stays down until a flush fails after every event is queued
K3 normally takes 168-175s against a 180s limit, so the limit is now 300s
The MCP env var browser spec reused an idle keep-alive socket that the proxy closes
after 5s, so its API calls now send Connection: close
* test(integration): add until_asserted so a wait polls the exact asserts it guards
eventually() takes a predicate separate from the asserts that follow it, which lets a
test wait on one value and then assert on another written later. until_asserted reruns
a block of asserts until they all pass and re-raises the last failure on timeout
The Prometheus shared-label tests and the auto-router session check now use it
* fix(vector_stores): stop sharing one list across default-constructed registries
VectorStoreIndexRegistry and VectorStoreRegistry took a mutable list as their
default argument. Python evaluates that once at class definition, so every
registry built without arguments shared one list. The proxy builds one that way
during startup, so an upsert into it leaked into every registry created after,
and into the class default itself.
Default to None and build a fresh list per instance. A caller-supplied list is
still used as given, including an empty one, so nothing that passes its own list
changes behaviour.
Fixes#38874
* test(vector_stores): annotate locals Final and mutate registries through their public API
The leak tests now go through add_vector_store_to_registry and upsert_vector_store_index rather than appending to the registry's list directly, which is closer to how the proxy hits the bug.
* fix(vector_stores): drop the last mutable list default and identity-only tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): out-of-process mock OpenAI, Anthropic, Bedrock and guardrail servers for the perf cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): coding-agent request corpus and real provider captures for the perf mocks
Request bodies come from a deterministic coding-agent session (system prompt, Read/Edit/Write/Bash/Grep/Glob tools, tool_use and tool_result turns carrying real source files picked by the request marker) in OpenAI chat, Responses and Anthropic formats. captures/ holds one real response per provider, route, shape and stream mode (gpt-6-luna, claude-sonnet-5, us.anthropic.claude-sonnet-5), cold prompt cache, body only, raw stream chunks with time offsets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): replay module for the perf mocks, captured provider responses picked by route, mode and request size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mocks replay the captured provider responses, knobs shrink to flake and speed
The OpenAI, Anthropic and Bedrock mocks answer with the captured body or stream chunks for their route, mode and nearest request size, ids swapped per request, on the captured clock scaled by the p knob. The synthetic reply builders, the per-token stream pacing and the hand-rolled Bedrock event-stream encoder are gone
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mock request log names its provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf guardrail mock remembers the request marker by call id for the response hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mock captures retaken cold, one marker and system-prompt prefix per cell
Run 2 shared one marker across cells, so every stream capture was served from the provider's prompt cache (cache_read_input_tokens up to 523946) and the json cells had small prefix hits. Every cell now carries its own marker, also prepended to the system prompt, and all 30 captures report zero cached tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf corpus reads tests/e2e files so the lane's runner image has them
* test(e2e): perf long_context json captures for Claude retaken with a 16k output cap, no longer cut mid-thinking
* test(e2e): perf mocks keep only what the replay path and the test read
* test(e2e): perf mocks interpolate the provider's time to first byte from the captures around the request size
* test(e2e): perf captures at 100k (short) and 500k (long) tokens with tools, the fixed-size set removed
* test(e2e): PerformanceTestCase schema, deterministic request plan and wire records for the perf cases
RequestShape names a corpus session (input_tokens, tools, output_tokens) and wire.request_body builds it with setup/corpus/session.py, so a 300k request is 85 file reads of real source in the endpoint's own format. The plan no longer predicts reply text, the mocks replay captured provider responses and the runner joins what the client saw against the mock's log by marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf Metric gains time_to_first_token_seconds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf User loses the unused role field, every JWT user is an internal_user
* test(e2e): perf RequestShape sizes requests from an input-token range, short and long stock shapes
* test(e2e): perf case runner (provision, Locust plan replay, observe) and the 50 rps mixed-provider case
provision.py checks the deployed config against the case and builds the callers, run.py replays the plan from a separate Locust process (locustfile.py) recording status, header, first-byte, first-token and total time per request, observe.py joins the records with what the mocks received and the workers' RSS to score every Limit, and compares each served body with the capture the mock replayed for it as a diagnostic. The case sends 15,000 requests at 50 rps over three model groups, two guardrails and the three request shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf case asserts time_to_first_token_seconds at percentile 50 and 99
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf provision creates every JWT user as internal_user
* test(e2e): perf observe reads only the mock log fields that remain
* test(e2e): 50 rps case sends short and long coding-agent sessions sized per request
* test(e2e): perf run keeps the per-request records as perf-records.jsonl under E2E_ARTIFACT_DIR
* test(e2e): the mixed small-gateway case runs at 25 rps (50 saturated one 4-worker pod on main and on v1.105.1)
* test(e2e): small_bodies_high_rate perf case, the github issue 46276 shape (11 to 17 KB chat bodies, keys only, 200 rps)
* test(e2e): the mixed small-gateway case runs at 10 rps for 6,000 requests (25 rps still saturated one pod on v1.105.1)
* ci: python lint job gets 20 minutes, the e2e harness phase (625 tests plus the replay harness suite) pushes a run past 15 when tests/e2e changes
* test(e2e): small-bodies perf case runs at 5 rps so gateway_added_seconds is per-request overhead, not queue wait
* test(e2e): mixed-provider perf case runs at 5 rps x 3,000 requests from 100 users
* test(e2e): mixed-provider perf case sends happy-path traffic only (no flaky deployment, no guardrail interventions or blocks)
* test(e2e): perf gateway model_list is three happy-path deployments, one per provider
* test(e2e): Postgres and Redis work per request as perf case limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mocks wait a case-level latency with jitter instead of replaying captured timings, cpu_seconds_per_request limit from metrics-server
* test(e2e): perf suite PR carries the small_bodies_5rps case only, the mixed-provider case moves to its own PR
* test(e2e): small_bodies_5rps limits from the ten v1.105.1 baselines (1.5x on datastore counts, 2x on latency and cpu, absolute on errors and memory)
* test(e2e): perf mocks keep only the short captures the small case reads, captured timings and the Bedrock invoke route removed
* test(e2e): perf mocks generate each reply from the request (provider schema, usage, ids, event order, chunk sizes) instead of replaying captures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(agents): support Databricks Agent Platform agents in the agent gateway
Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire
(Responses-shaped input, output_text items, output_text.delta SSE) to Model
Serving endpoints (/invocations or the unified /serving-endpoints/responses)
and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials
Register it through the agents config block or the Admin UI agent catalog and
call it through /a2a/<agent> (message/send, message/stream) or model:
a2a/<agent> chat completions. The chat route now runs every completion-bridge
agent through its own provider, and attaches the minted Databricks OAuth token
for both bridge and URL agents. Consumed OAuth settings never reach the
provider call
* fix(agents): harden databricks_agent input, output, and OAuth handling
Non-text content parts answer 400 unless drop_params is set, a reply without
an output list answers 502, flat OAuth fields resolve an os.environ api_base,
the bridge debug log no longer echoes the request model, routing tests stub
only the HTTP boundary, and the catalog description fits the dropdown
* fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block
* fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App
* refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name
* fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty
* test(agents): drive the flat OAuth a2a send through the real bridge
* fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change
* fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents
* fix(databricks_agent): address review caveats on chat bridging, params, and auth
Map max_tokens and max_completion_tokens to max_output_tokens and pass
temperature, top_p, and user, so /v1/messages works without drop_params.
Send A2A message metadata only with string values, answering 400 or
dropping the rest under drop_params. Join message items with a blank line
in both stream and non-stream replies. Cache the env service principal
token until shortly before it expires. Keep databricks_agent replies out
of the response cache.
On the chat bridge, let request fields win over the agent's stored params
while pinning the agent's api_base and api_key, and apply the agent's
static_headers between client and agent extra_headers, as /a2a does.
Resolve os.environ/ only for databricks_agent api_base and api_key instead
of every bridge provider. Ignore an unusable databricks_oauth block on URL
agents on chat, as at the merge base, instead of answering 400.
* test(internal_context): list the databricks agent token cache as process-local
* fix(databricks_agent): keep the agent's endpoint, input, and target out of client control
extra_body model and input answer 400, or are dropped under drop_params.
A client api_base or base_url never replaces the agent's card URL on the
bridge. Chat to a URL agent keeps its merge-base call, and an incomplete
databricks_oauth block on a URL agent keeps its merge-base ValueError.
* test(databricks): expect the merge-base 400 for a client api_base on agent chat
* fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach
* fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(decisions): read a deployment's own supports_box_questions flag before the cost map
The box-question check only read the shared cost-map row and required a decisions mode there, so an OpenAI-format deployment whose model_info sets supports_box_questions was refused with a 400, which kept the wire test from #46633 red on main
* fix(decisions): keep the box flag on its deployment and out of request bodies
A model_info that is not a plain dict no longer fails the call, the proxy
decisions endpoint drops a caller-sent model_info, and the router stops
copying supports_box_questions onto the shared cost-map key.
* fix(decisions): keep the box flag shared with same-model deployments like main
Scoping supports_box_questions to one deployment turned box questions on an unflagged sibling of the same backend model from reaching the provider into a 400. Siblings inherit the flag through the shared cost-map row again, as they do on main. Also formats two new tests.
* test(decisions): pin box flag inheritance by an unflagged same-model deployment
* test(decisions): audit box-question deployment flag across providers, routes, and chaos cases
* fix(cost): price bridged /v1/messages reasoning tokens from the chat completion, not the served Anthropic body
* fix(cost): keep the type gate flat for the bridged /v1/messages billing path
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): treat adaptive thinking as enabled when mapping response_format
is_thinking_enabled only recognised thinking.type=enabled, so a request that
paired thinking.type=adaptive with response_format took the synthetic JSON
tool path with a forced tool_choice, which Anthropic rejects alongside
thinking. The same check also gates the Bedrock Converse and Databricks
forced tool_choice downgrades, so they now see adaptive thinking too.
* fix(anthropic): keep signed thinking blocks with empty text on chat completions
Models that default to display: omitted return signed thinking blocks whose
text is empty, and Anthropic rejects the next turn when they are not replayed
unchanged. is_unsignable_thinking_block dropped them along with unsigned
blocks, and the inline content path required non-empty text, so a tool-use
turn on Opus 4.7+ or the 5 families lost its thinking blocks and 400ed on the
tool-result turn. Only a missing signature now drops a block, and the
mid-conversation system placement mirrors that rule. The /v1/messages strip
helper is left to #41362.
* fix(anthropic): skip the thinking-strip retry when block_binding asks for an error
The /v1/messages retry loop strips thinking blocks and resends on any
invalid-thinking 400, including the binding mismatch a caller opted into
with thinking.block_binding.prefix_mismatch_behavior: error. The retry
decision now sees the request body and surfaces that 400 instead of
overriding the caller's choice. #42888 covers the litellm_params side.
* fix(bedrock): carry thinking.block_binding through the Invoke thinking rebuilds
Bedrock Invoke rebuilt the thinking param from scratch when display: updates
was requested, on both the /v1/messages and chat completions paths, and the
legacy enabled-to-adaptive translation plus the clear_thinking injection did
the same, so thinking.block_binding never reached Bedrock. Every rebuild now
spreads the caller's thinking and only overrides the keys it owns, and
AnthropicThinkingParam types block_binding. The thinking-binding-controls
beta was already on the Bedrock allowlist.
* fix(anthropic): keep forcing the response_format tool under adaptive thinking
This reverts ad9e861ae7. Anthropic, Bedrock Invoke and Vertex all accept a
forced tool_choice next to thinking.type=adaptive and only reject it with
type=enabled, so counting adaptive as enabled fixed no live failure. It did
drop the forced JSON tool, so Claude answered response_format requests in
prose, and it downgraded a required tool_choice to auto on Bedrock Converse
* test(e2e): drop the adaptive thinking tool round trip case
With adaptive thinking the model often skips thinking on the weather prompt, so
turn 1 carries no signed empty block and the case fails before the replay it is
meant to cover. Anthropic also answers the replay with a 200 whether or not the
block is forwarded, so the case could not tell the fix from main. The factory
unit tests pin the outgoing shape instead
* fix(anthropic): only skip the thinking-strip retry for a binding mismatch 400
prefix_mismatch_behavior: error asks Anthropic to reject a block bound to
another conversation. It says nothing about unrelated bad blocks such as a
null signature, so those still get the strip-and-retry self-heal. Also trims
the empty-thinking docstrings to what the code guarantees and types the new
test parameters
* refactor(anthropic): narrow thinking fields with the mapping validator instead of typing.cast
typing.cast is banned by the strict ruff gate (TID251), so the three new casts pushed that rule one over its merge-base count. The existing pydantic mapping validator gives the same typed view without an unchecked cast, and it now has a generic name since it is no longer only used for Claude Code payloads
* refactor(anthropic): read thinking settings through the validated mapping so basedpyright stays at its merge-base count
The Bedrock clear_thinking rewrite still narrowed with a cast and an untyped dict fallback, which added two reportUnknownVariableType errors. It now uses the same pydantic mapping validator, made public for that reuse, and the retry hook validates the request body itself so the handler passes a typed dict instead of an untyped one
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* feat(proxy): split Azure PTU deployments across teams by share
A PTU deployment's model_info takes ptu_shares (team_id -> whole PTUs adding up to
ptu_count) in place of a single team_id. With LITELLM_ENABLE_PTU_COST_ATTRIBUTION on,
routing hides the deployment from teams holding no share, the v3 limiter enforces each
share as a per-team TPM ceiling from Azure's per-model TPM-per-PTU sizing, the daily
flat-cost rollup splits the hourly cost by share, and /team/daily/activity plus the Team
Usage page report PTU-hours next to tokens.
* fix(proxy): weight cached input in PTU ceilings and size team-scoped deployments by public name
The PTU flag is read from the process environment on every check instead of
through the secret-manager path, so a proxy with a hosted secret manager does
not pay a round trip per request. A team's ceiling now carries the model's
cached-input ratio and settlement charges cache reads at that ratio, matching
how Azure sizes a PTU. A deployment registered for one team is found by its
team_public_model_name for PTU-hours and the ceiling, since its model_name is
rewritten to a unique internal name. ptu_shares are StrictInt on ModelInfo and
parsed_ptu_shares refuses booleans, strings, and non-string team ids, a
fractional ptu_count reserves nothing, and models_processed counts distinct
deployments rather than holdings.
* fix(proxy): reserve PTU ceilings in normalized tokens, keep share routing flag-independent, and warn only shared or Azure reservations
* fix(proxy): size the uncapped output floor against the PTU share in raw output tokens and escape the exported team id
* fix(ptu): refuse ptu_shares naming a team that does not exist
* refactor(ptu): keep Azure detection under llms/azure and check share teams with one query
* fix(proxy): drop the reverted TeamIdSearchFilter import after merging main
Main reverted the team key search change that added TeamIdSearchFilter, so the PTU share team check types its where clause locally
* fix(proxy): chunk the ptu_shares team lookup under the IN-list bound
* fix(proxy): count a request naming a shared PTU deployment directly against its group's ceiling
* fix(proxy): resolve every name a shared PTU deployment is served under to its group's ceiling
* fix(ptu): refuse ptu_shares declared without ptu_count and cost_per_ptu_per_hour
* fix(ui): seed total_ptu_hours in the empty daily activity metadata
* fix(ptu): size usage rows by the name the request used
* fix(ptu): size a team's usage rows by the deployment it is served from
* fix(ptu): never size a team's usage row by another team's reservation
* fix(ptu): reserve an uncapped request against the cap the proxy writes, not its prompt length
* fix(ptu): read a share map only through the terms registration requires
A ptu_shares map stored without ptu_count, cost_per_ptu_per_hour and
ptu_effective_from, or one that does not add up to ptu_count, is refused at
registration but was enforced once loaded: every other team got the 400 and
the share holder a ceiling while no flat cost accrued. _deployment_shares now
reads the shares through ptu_terms, so such a row reserves nothing, the same
as before the map was read.
* fix(ptu): keep a declared split as the access rule and check a config split at boot with the flag off
6fb24d1d read ptu_shares through the pricing terms so a termless map reserved
nothing, but with the flag off the router skips ptu_config_error for config.yaml
entries, so such an entry loaded and served every team, against the rule that a
declared split restricts access whether or not cost attribution is on.
_deployment_shares reads the declared map again, and the router now raises
ptu_config_error for a config-sourced entry declaring ptu_shares regardless of
the flag, so a malformed split is refused at boot instead of being honoured or
ignored silently.
* fix(ptu): charge wildcard-served names, pass-through usage, and team-less settlements against the team's PTU ceiling
A name served by a wildcard route now resolves to the shared deployment behind it, so the
limiter's ceiling and the team daily activity sizing both find it. Pass-through responses
settle the PTU counter from combined_usage_object in Azure-normalized tokens, and a
reservation taken at admission is settled even when the success metadata lost the team.
* fix(ptu): resolve a wildcard-served name to the one pattern the router picks, and a deployment id before any wildcard
A team's wildcard reservation behind an open proxy-wide * route was charged and sized for traffic the * route served, and naming a shared deployment by id next to a catch-all wildcard escaped its per-team ceiling. Router.wildcard_route_deployments now owns the most-specific-proxy-wide-else-team precedence, and both the limiter and the consumption page resolve through it after listed rows and deployment ids.
* test(router): pin the wildcard route precedence that PTU sizing and the router share
* fix(proxy): accept whole-number PTU shares written as floats, as ptu_count already does
A share of 2.0 now reads as 2 on POST /model/new and in config.yaml, where it was
refused with a 422 and dropped at load. Fractional, infinite, boolean and string
shares stay refused. Adds unit tests for the malformed-rate, provider-over-prefix,
zero-token Responses usage, fractional-count and sizing-warning paths.
* fix(proxy): settle team PTU reservations in their own minute and keep billed PTU days when shares move
* fix(proxy): write each PTU deployment day's team charges in one transaction
The backfill counts a deployment day as priced once any team's row exists for it,
so a rollup that wrote team A's share and lost team C's left team C unbilled for
good. Each deployment day's team rows now go through one Prisma batch with the
retry around the whole batch, so a day is fully written or not written at all and
the next backfill prices it for every team.
* fix(proxy): read the PTU flag from a secret manager, serve share holders from their share, and refuse unfittable calls and share-holding team deletes
* test(router): list the cached PTU ceiling helpers in the router coverage check
* fix(proxy): read the PTU flag only for share holders, guard org and SCIM team deletes, keep share holders on their shared deployment through retries and the scheduler, and date PTU-hours by the terms in force
* fix(proxy): bill a rerun PTU day to the current share holders alone
* fix(proxy): replace a PTU day's rows by day and deployment, not a negated team list
* fix(proxy): read the PTU flag from the secret manager first, like get_secret
* fix(router): scope retry health lookups to the caller only for PTU share holders
* fix(router): move the share-holder retry check into ptu_shares
Adds ptu_hours and total_ptu_hours to the daily activity integration goldens and fixtures, since every route now reports them
* test(integration): cover PTU shares across routing, delete guards, cost attribution and config regex timing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(guardrails): resolve decision and judge models in caller team scope
* fix(guardrails): scope guardrail model calls to the caller's team only where it owns the deployment
Read the team only from the auth object the proxy attached to the request, never from caller JSON,
and use it only when the router's route to the guardrail model for that team reaches a deployment a
teamless caller cannot. Shared names, aliases, legacy team tags and wildcards keep resolving and
billing as before. Decision calls pass the team in litellm_metadata only when it is used, so the
logged model group stays intact.
Adds integration cells for chat, responses and messages, forged team ids, a team deployment that
shares a name with a shared model or wildcard, and billing, plus unit cases for legacy team tags
and aliases.
* fix(guardrails): route team guardrail calls to the team deployment without charging the team
Guardrail calls on a team's own model now go to the deployment the router picks for that
team, by its id, with no team on the call. Like every other guardrail call, they write no
spend row and add nothing to the team's or the key's spend
* fix(guardrails): pin the team deployment only for the judge guardrail, not shadow eval judge calls
* fix(guardrails): keep the teamless route when only a team wildcard covers the guardrail model
* refactor(guardrails): scope the team model fix to the decision model guardrail only
* fix(guardrails): send decision calls on a team's own model with the team, and keep team wildcards out
The decision call on a team's own model now carries the caller's team instead of being pinned to a
deployment id, so it is charged to that team and goes through the team's normal routing: an access
window that reserves the deployment for the team lets the call through, and pre-call checks see the
input and skip a deployment too small for it.
A team deployment listed under a wildcard public name, such as team_public_model_name openai/*, no
longer captures a guardrail model the team never named, whatever model it points at.
Adds unit and integration cells for the reserved deployment, mixed context windows under
enable_pre_call_checks, a team wildcard name over a fixed model, and the owner team's spend.
* fix(guardrails): log the owner team's name on decision calls charged to it
The decision call on a team's own model now carries the team alias beside the team id, so the
spend row and the Logs page name the team the call was charged to.
* fix(decision_model): use the caller's team only for names nothing else serves
A team model named like a shared deployment, a shared wildcard or an SDK
public name took over the guardrail's decision calls for that team, so a
team admin could point a proxy-wide guardrail at their own endpoint. The
team is now used only when a caller without a team reaches no model under
the name.
* refactor(decision_model): drop the team route subset check the served check covers
Once nothing serves the name without a team, every deployment a caller
without a team can reach is a wildcard, so the subset check only ever
rejected an empty team route. Pin team wildcards over a name nothing else
serves.
* fix(decision_model): use the caller's team only when judge_target finds nothing without a team
The served check rebuilt the router's rules and missed a shared wildcard whose
provider comes from custom_llm_provider, so a team model under the guardrail's
name took over that team's decision calls. Ask judge_target, the same lookup the
teamless dispatch makes, instead
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(batches): emit per-line JSONL batch records to callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): only emit line items for final batches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(batches): satisfy ruff format for line item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover line-item edge shapes and general settings propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep deployment credentials out of callback-visible params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(batches): declare credential transport attribute on Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): integration contract for per-line batch callback events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert per-line events beside the aggregate and create events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert line item events carry no credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset batch line-item flag when the setting is absent
Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): map anthropic batch lines through the anthropic message transformer
Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: tag the anthropic result cast with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: shorten cast-ok reason so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip line-item callbacks for providers without a line reconstruction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap widened provider literals so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): pair bedrock lines by recordId and type them from the reconstructed response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep line item events out of proxy limiters and strip credentials from child params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip failed line item events in parallel request limiter failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): tolerate lifecycle spend rows when asserting no line item rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): place mutable-ok tags on the lines the discipline gate flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip router budget accounting for batch line item events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep anthropic line errors and skip native vertex batch rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): restore test-quality tags on line item tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move batch line-item unit test to tests/unit after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): emit batch line-item callbacks once per batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when fan-out emitted nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): make the line-item claim release ownership-safe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep bodyless failed batch lines reaching callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): seed batch line hidden params in get_hidden_params defaults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(batches): reuse fetched batch result files for line-item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when nothing was emitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): type batch line-item callback sites to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap batch hidden-params cast for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reason suppressions for batch line-item lint deltas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop stale mutable-ok suppressions after LIT002 removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486)
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters
Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id)
bypassed the call-type-only batch guards:
- ClickHouseSpendLogger wrote one spend row per line on top of the aggregate
aretrieve_batch row, billing every batch twice with the flag on.
- Router deployment callbacks and the routing-strategy usage handlers charged the
historical batch's tokens/rpm to the live TPM/RPM counters, so after
Router.aretrieve_batch the next chat raised RouterRateLimitError.
Skip these events wherever the aggregate batch retrieve is already skipped.
* fix(logging): keep internal litellm_params authoritative in model_call_details
A request body key named litellm_params lands in optional_params, and
update_environment_variables merged **optional_params after the internal
entry, so a caller could overwrite the logging kwargs' litellm_params and
forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose
body carried {"litellm_params": {"batch_parent_id": ...}} returned 200
with no ClickHouse spend row. Re-assert the internal litellm_params after
the merge so every sink (ClickHouse, Postgres, router quota counters)
reads the same trusted value, and assert the aggregate aretrieve_batch
row still writes in the ClickHouse line-item test (review finding).
* fix(clickhouse): skip failed batch line items in the spend sink too
A failed JSONL line is part of the aggregate aretrieve_batch row's
request counts; writing it its own spend_logs row inflated request
counts with the line-item flag on. Extend the batch line-item guard to
the failure handler (review finding) and pin both directions: a failed
line item writes no row, a genuine live failure still does.
* fix(batches): declare batch_line_items service family on claim calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(integrations): keep batch line items out of the built-in metering sinks
With store_batch_line_items_in_callbacks on, per-line batch events carry a
real response_cost and reached six metering sinks that lacked the
is_batch_line_item_event guard, so a completed batch metered aggregate +
per-line (roughly 2x true spend): prometheus (litellm_spend_metric and
request counters), openmeter, lago, datadog cost management (FOCUS
BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost
metrics. Skip line-item events in each sink's success/failure metering
entry points, matching the guards already on the Postgres and ClickHouse
spend sinks. OTel spans for line items are still emitted; only their
cost/token metrics are skipped. Tests drive real sink instances on the
real dispatch lists through a completed 2-line batch and assert each
sink meters exactly the aggregate.
* style: format metering-sinks test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test
Follow-ups from review on the sink guards: record_failure() now skips
batch line items too (a failed line's interval measures retrieval and
emission, not its model call), and the prometheus guard test snapshots
and restores the process-wide registry collectors instead of leaving
them unregistered. The OTel v2 test is now a flag on/off A/B over the
whole metric snapshot: cost and usage values must match exactly and no
metric may gain datapoints with the flag on.
* chore: finish main merge conflict resolution for batch line items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort imports in litellm_logging after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): use the public hook class names main now exports
* fix(batches): only the batch writer can mark a callback event as a line item
amoderation, transcription and speech build litellm_params from every request kwarg,
so a body field named batch_parent_id reached Logging.litellm_params and every
spend, budget and rate-limit hook skipped the request. update_environment_variables
now drops an incoming batch_parent_id; the line-item writer sets it through
Logging.mark_batch_line_item. Batch line hidden params are emitted only when set.
* chore(batches): keep the lint gates within main's ceilings after the merge
Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the
banned cast with a literal match, dropped isinstance checks main made redundant,
and moved test-quality reasons onto the lines the gate reads.
* fix(anthropic): read the message id off the validated message
The anthropic_messages logging path can hand over an already parsed message object,
which has no .get; main read its id through getattr, so take it from the validated
AnthropicResponse instead.
* fix(batches): an unreadable error file no longer drops the output line events
Costing counts an error file it could not fetch as no failed lines; line logging
refetched it and the raise aborted the whole fan-out, so the successful lines were
never emitted. The error file now reads as empty with a warning.
* test(batches): pin billing for batch_parent_id on every endpoint and value shape
Chat completions, embeddings, responses and messages join moderations, speech and
transcriptions, and speech is driven with an empty, int, list, object and 5 KB value.
* test(batches): bill every request of a concurrent burst mixing forged and plain bodies
* revert: keep the line-item fan-out all-or-nothing on an unreadable error file
85fc4498cc emitted the output lines and kept the batch claim when the error file
could not be read, so its failed lines were never retried. Raising releases the
claim and the next retrieve retries the whole fan-out.
* test(batches): pin the all-or-nothing retry and warm the burst model first
An unreadable error file emits no lines and the next retrieve emits all of them.
The burst cell drives one speech model after a warm-up request so a cold worker
cannot fail it, and waits for the batched key spend.
* fix(batches): run the line fan-out outside the aggregate event's deadline
The logging worker cancels a callback coroutine after its per-callback
deadline, and the line fan-out ran inside the aggregate aretrieve_batch event
before its callbacks, so a large batch or a slow sink could cancel the
aggregate billing event. The fan-out now runs in its own task, built from a
copy of the parent's litellm_params taken before the aggregate callbacks
write their request-local markers.
Anthropic keeps no input file and reports input_file_id "None", which the
fan-out fetched and failed on, so no Anthropic line ever arrived. That id is
now treated as no input file.
* fix(batches): let running line fan-outs finish before proxy shutdown closes their clients
A started fan-out keeps its batch claim, so a shutdown that cut it short
skipped the unsent lines until the claim expired. proxy_shutdown_event now
waits up to MAX_TIME_TO_CLEAR_QUEUE for them first. Also replace a test's
type: ignore with an isinstance check.
* fix(batches): keep batch line fields off callback kwargs and skip fan-outs at shutdown
Line bodies were spread into the callback kwargs, so a line's metadata could settle router rate limit reservations; they now stay in optional_params, and the router rate limit and prompt caching hooks skip line events.
A retrieve logged after the shutdown drain, or by the logging worker's exit flush, no longer starts a fan-out its loop would destroy while it holds the batch claim.
The output file is fetched before the error file again.
* fix(batches): log batch lines under the retrieve's callback settings and keep them out of alerting and shadow eval
Line events inherit the retrieve request's key and team callback settings, so turn_off_message_logging and per-team logging credentials apply to them, and they keep the system prompt in the logged messages.
Shadow eval, Datadog metrics, Slack daily reports and PagerDuty skip line events, since the batch API served them and the aggregate event already counts the batch.
* test(batches): line events follow the key's message logging opt-out
* test(batches): line events follow team logging settings, keep the line system prompt and survive a restart once
* fix(batches): keep in-memory line claims for 10,000 batches and pin managed batch emission across poller and retrieves
* Keep batch line unit tests in-process and drop repeated per-sink comments
* Share one batch provider alias across batch_utils and line item logging
* Report Bedrock line status codes, skip unknown Bedrock lines without raising, fetch the input file after the native Vertex check
* Read a swallowed Redis claim failure as unavailable, price unpriceable lines at zero, keep the OTel failure docstring
* Call handle_completed_batch when line items are off, keep a cache test from leaking Redis into the line claim cache, pin an unpriceable line's zero cost on the callback kwargs
* Hand the cost poller's credential snapshot to line item file reads, so a managed Bedrock batch's lines read its input from the deployment bucket
* Log batch lines through the sync callbacks a direct async call runs, keep failed lines out of router cooldown counts, and emit an error line whose body is text
* fix(batches): log failed Bedrock embedding lines as embeddings with their input
A Bedrock batch input record carries no url, so a failed Titan v2 line fell back to acompletion and its
inputText never reached the event messages. The failed line now takes aembedding from the deployment
model and the record's inputText is the line input
* fix(batches): keep a Bedrock line's system prompt in its logged messages
Bedrock batch input records hold the system prompt as content blocks, which logging only reads off the kwargs as text, so Claude and Nova lines dropped it
* feat(batches): store per-line batch requests and responses in S3/GCS via batch_line_item_callbacks
Replace the per-line logging path with a dedicated hook. When the hourly batch cost
poller finishes a managed batch, it pairs every output and error line with its input
line, prices each line, and hands one payload to every logger listed in
litellm_settings.batch_line_item_callbacks. s3_v2 and gcs_bucket write it as one
NDJSON object per batch under batch_line_items/<day>/<batch id>.jsonl.
Bodies are dropped and the header kept when the creator key or team, the global
setting, or the logger itself turns off message logging, and when the creator can
no longer be found. A failed upload is logged and never blocks spend tracking.
* fix(batches): store batch line items after the batch is billed and retired
Send a billed batch's lines only after its row is marked complete, so a slow or
failing sink never holds a claimed batch open. Skip reading the input file when
the bodies are dropped anyway, still send the error lines of a batch whose output
file is gone, and end every NDJSON record with a newline.
* test(batches): cover team-level and deleted-key redaction of batch line items
* fix(batches): store batch line items in the background so a hung bucket never holds up billing
The poller now retires a batch first and hands its line items to a per-worker background store that runs one batch at a time. A slow, hung or failing bucket no longer delays billing or retiring the other batches in the poll cycle. While a poll cycle's worth of batches is still waiting to store, the next batch's lines are dropped with an error.
* fix(batches): store an unbilled batch's line items only once it is retired
When retiring an expired, cancelled or all-failed batch fails, the batch stays on the poll page, so its lines are no longer read and stored again on every poll cycle until it is retired. The poller also releases a billed batch's output bytes before reading the next batch's output, as it did before line items existed.
* fix(batches): price and write batch line items off the event loop
Pricing every line of a large batch and serializing its object ran on the
proxy's event loop, so a 50k-line batch stalled the worker for seconds.
Both now run in a worker thread. An object the bucket refuses for good is
no longer kept in s3_v2's drop marks, which only a log-queue flush clears.
Vertex AI input rows carry the custom_id in their key or request labels, so
their lines now get the request too.
* fix(batches): pair no request for a shared custom_id and wait for an unclaimed batch's retry
A Vertex AI embeddings entry whose input is a list is split into one row per
item, all with the same custom_id, so its line now gets no request instead of
the last row. On a database without batch_processed nothing claims a billed
batch, so when marking it complete fails it stays in the poll page and its
lines now wait for the next poll instead of being written on every cycle.
* fix(batches): store batch line items only once the poll has billed every batch
Pricing a large batch's lines on a worker thread still competes with the poll for the GIL, so the next batches
in the same poll were billed and retired seconds later. The background stores now wait until the poll cycle that
queued them has finished.
* fix(batches): hash line-item objects off the event loop and keep lines with a lone surrogate
A batch's whole line-items object was hashed for its S3 PUT on the event loop, and a line
holding a lone surrogate from a provider's JSON escapes made the object impossible to
encode as UTF-8, so the batch's object was never written. The object is now prepared in a
worker thread, and only the line that cannot be encoded is written with escaped characters.
* fix(batches): keep request and response for batches created without a virtual key
A batch created with a JWT has no API key on its row, and its line items were stored without
request and response as if its creator had been deleted. Such a batch now follows its team's
and the proxy's message logging settings, the same as a master key batch. A deleted key or team
or a failed lookup still drops the bodies.
* fix(batches): pair line items with the right request on every deployment and skip empty objects
A batch retired with nothing billable now reads its input file by the id the provider reported, so a batch that
ran on the second deployment of a model group keeps each line's request. A Vertex AI line read as custom_id
"unknown" pairs with no input row, and a batch whose files hold no readable line writes no object
* fix(batches): let OAuth2 batches follow team and proxy message logging settings
The line-item redaction check looked up every saved creator credential other than the master key or a CLI session as a virtual key. An OAuth2 batch saves the raw token, so it read like a deleted key and lost its request and response
It now looks up only a saved value shaped like a virtual key's hash, reusing is_valid_sha256_hash, so OAuth2 batches follow their team's logging settings and the proxy-wide setting while deleted keys still fail closed
* fix(batches): hash the creator credential and write batch line items in parts
The creator credential reaches line metadata only as a sha256 hash, so an
OAuth2 token or the master key alias is never written to the bucket. A queued
store keeps only whether the batch was billed and downloads the output file
again at its turn, logging an error naming the batch if that fails. Lines are
built and uploaded one part at a time, each closing once its raw request and
result bytes reach BATCH_LINE_ITEMS_MAX_PART_BYTES (16 MiB by default), with
the input file indexed by byte span instead of parsed up front.
* docs(batches): say async_log_batch_line_items is called once per part
* fix(batches): log an error when a batch's deployment is gone before its line items are stored
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
A chat completions request to gpt-4.1 (or any OpenAI model older than 5.6) with drop_params on still sent the marker inside messages and tools. It is now removed there, while drop_params off keeps sending it through and models that read it keep it either way
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Responses route now resolves openrouter alongside openai. The
OpenRouter config reuses the OpenAI request and response transforms and
owns only its bearer credential (OPENROUTER_API_KEY, then OR_API_KEY) and
base URL (OPENROUTER_API_BASE). Wire model ids follow Python's
get_llm_provider, including the openrouter/auto native id with an
explicit provider. The Python route host no longer hands litellm.openai_key
to OpenRouter.
The OpenRouter live test lives in tests/live/openrouter.rs and returns
early when the provider key is not exported, so cargo test stays offline
by default instead of hiding it behind --ignored.
Co-authored-by: nate-berri <nate@berri.ai>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(responses): bill a background response once, from the poll job
Reading a completed background response with GET /v1/responses/{id} wrote a
full-cost spend row on every read, on top of the row the background cost poll
writes. Reads of a Responses object (GET, DELETE, cancel) now never bill, and
the poll job is the single biller.
The poll job now bills with the creating request's attribution (key hash, team,
org, user, aliases, tags) read off the managed object row, through a helper it
shares with the batch cost poll. The background create persists the creating
key and request tags so that attribution survives until the poll runs.
* test(proxy): a background response read reports zero cost
* fix(responses): bill a background response once, on the first read or poll that sees it finished
A conditional update moves the stored row out of queued or in_progress so only one worker or pod bills it, through one re-read stamped with the creator's attribution. A failed billing read puts the row back for the next poll
* fix(responses): keep the caller's response id out of the billing failure warning
* fix(responses): keep the creating client's User-Agent tags on the one background charge
* fix(responses): keep managed background reads free and bill each response once
The managed files hook marks reads and cancels of proxy-issued response ids, so only those stay free. SDK and database-less reads of a background response still bill, extra_body background creates are tracked, and the poll's status read no longer cools down a deployment on a 404.
* fix(responses): bill finished background responses from the fetched body and store rows Postgres accepts
* fix(responses): give the background billing event its own spend log row and bill responses the create left untracked
* fix(responses): narrow the managed files hook with a protocol instead of an enterprise import
* fix(responses): leave a read the caller retargets with client-side credentials to the cost poll
* fix(responses): bill background responses on the creator's end user and report the charge on the settling read
The read that settles the bill returns the charge in x-litellm-response-cost, the billed event
carries the creator's end user and the key, team, and user budget fields, and its spend row is
named after the id the client got. Override reads, cancels, and deletes settle from the
configured deployment, a streamed create marks its row billed, stored rows keep no model
output, and model insights counts the billed row's spend without a request.
* test(responses): cover background response billing across proxies, reads and chaos
* fix(responses): settle background billing only for callers allowed to address the response
A delete now settles the bill only after the response id access check passes, a streamed
create is marked billed when it sets background inside extra_body or its completed event
arrives after the last chunk, the billing event keeps the deployment's litellm_params
base_model, a key with no user and no team bills no user instead of its owner scope, and the
enterprise cost check import is optional
* fix(responses): record the bound on the pending status IN filters
* fix(responses): send the billed cost split on the settling read and keep the batch cost event as it was
The read that settles a background response now sends the whole cost header
family from the billed event, so the breakdown headers add up to the charge.
Batch cost events go back to main's attribution (creator id as the user, no
budget fields); only background responses get the end user and budgets.
Integration tests expect the charge on the settling read (or none when the
poll billed first), one charge for a streamed create, and SDK flows that poll
with the id each read returns.
* fix(responses): bill each background response once across upgrades, streams, and failed settles
Rows stored before this billing model are found by the issued id and rekeyed by the
provider id, and the cost poll retires one a read already billed. A streamed background
create is free on its own and bills the finished response through the same managed row
before the client sees response.completed, so a racing read cannot bill it again. A
settle that fails retries in the background, a retargeted read with no row tracks and
bills from the configured deployment, and a delete keeps its billed cost header.
Reverts the unrelated json.dumps narrowing in the completion transformation.
* fix(responses): bill cache-polled background streams and match poll rows by the issued id
* test(responses): cover deletes, failed settles, budgets, base models, and cache-polled background billing
Adds integration cells for the background response billing paths the audit
inventory lists: a DELETE that settles the bill and refuses a stranger, a
settle the database refuses that retries once it recovers, the Prometheus
remaining budget, base_model pricing from model_info and litellm_params, the
daily model usage rollup, the end user, the settling read's cost split, stored
rows without model output, a forged managed marker on the create, streamed
creates with background in extra_body, and the cache-polled create
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): allow model instances in LiteLLM_EndUserTable validation
* fix(proxy): preserve property-backed attributes when coercing spend
* refactor(proxy): default null end-user spend with a field validator
* fix(proxy): drop model-wide from_attributes from the end-user table
* test(proxy): unskip the /customer/block integration contracts
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(scim): write a group's roster in one locked transaction instead of one member at a time
POST, PUT and PATCH /scim/v2/Groups resolve every member in one query and write the roster, the
membership rows and the users' team tags in one transaction under the team's advisory lock, so a
500-member push runs a fixed number of statements instead of thousands. PUT now answers with the
roster it just wrote
* test(scim): assert the roster the IdP gets back and boot the bulk roster cell with two workers
* fix(scim): skip the team lock for a rename-only PATCH and lock member rows in id order before the roster write
A PATCH naming no members opened the roster transaction and took the team advisory lock for nothing. A SIGKILLed worker then left its orphaned query engine holding that lock until the 60 s transaction timeout, and /team/delete, which takes the same lock, timed out behind it. An empty delta is now answered from a plain team read, with no transaction and no lock
Two groups trading members in parallel locked user rows in opposite orders, and one request died with a Postgres deadlock. Every user row a push touches is now locked with SELECT ... ORDER BY user_id FOR UPDATE before the first write, so the two requests serialize
The bulk roster cell gets a 120 s client through a new owned_proxy_process client_timeout, so an unfixed push fails on the statement count instead of the harness clock, requires positive counts on both pushes, and gains a parallel roster-trade test that answered 500 before the lock
* fix(scim): evict the cached team on a group rename and read a no-op roster from the writer
A rename-only PUT or PATCH wrote the new alias and left the team cached under
its id and old alias on every worker, so keys kept logging spend against the
old team alias until the TTL expired. The group write now evicts the cached
team (and broadcasts the eviction) right after the alias or metadata update.
sync_team_roster reads the team from the writer when the plan names nobody,
so a replica cannot hand back the pre-rename row; a plan already matching the
roster returns inside the lock without writing, evicting, or emitting the
members metric; the response is built from the written row, so lastModified
reflects the roster write; and detaching a removed member bumps the user's
updated_at like every other user write.
* fix(scim): bump detached members' updated_at in UTC
The detach UPDATE wrote updated_at = CURRENT_TIMESTAMP, which Postgres casts
into Prisma's naive timestamp column in the session timezone, so on a database
whose session is not UTC the removed member's row landed hours away from the
UTC instants Prisma itself writes. Write NOW() AT TIME ZONE 'UTC' instead, and
have the bulk roster cell assert the bumped row sits between two reads of the
database's own UTC clock taken around the replace.
* fix(scim): chunk the group write's member lookups under the bind cap
Every Prisma "in" filter the group write issues bound one parameter per
member with nothing bounding the list, which check_unbounded_in_lists
reports as 11 new findings. The single-field lookups now go through
litellm.repositories.chunked_in (caller_transaction inside the roster
transaction, per_chunk_ok for the role recompute), and the one
cross-field OR read slices its values at IN_LIST_CHUNK_SIZE and records
the bound. A push of IN_LIST_CHUNK_SIZE + 1 members now reads in two
slices, and a member named only in the last slice still resolves.
* fix(proxy): send bulk cache evictions in slices so the broadcaster drops none
Veria pointed out that the invalidation publisher caps its pending tasks at
1,024 and drops the excess with a warning, and that a group push now detaches
every removed member's keys in one call instead of one member at a time, so a
removal of more than roughly 340 members (three cache keys each) silently left
stale key objects on the other replicas. The merge base never hit the cap
because each member was a separate request.
Bulk evictions (team, user, organization, and session key deletes, plus the
roster settle) now publish in slices of 256 and wait, bounded at two seconds,
for the backlog already handed to the event loop before each slice. A Redis
that never answers costs a bulk caller at most that wait per slice instead of
wedging the request, and single-key callers are untouched. Regression tests
push 1,500 keys and a 500-member removal through a slow publisher and assert
every eviction reaches Redis; both fail against the previous tip.
* fix(proxy): clear this worker's cache before pacing bulk eviction broadcasts
Greptile pointed out that pacing the broadcasts in slices also paced the
local eviction: a bulk delete waited for the publish backlog before dropping
the next slice from this worker's own memory, so on a Redis that accepts
connections but never answers a key deleted from the database kept
authenticating on the handling worker for up to two seconds per slice. The
merge base cleared every key locally at once.
Bulk evictions now drop every affected entry from this worker's memory first,
through the cache's own partition lookup since key objects live apart from
the other auth objects, and only then publish in paced slices. The roster
settle does the same for the deleted tokens, the JWT mapping keys and every
changed member's user and membership entries before any step that can wait.
Regression tests run both paths against a publisher that never answers and
check that nothing is still cached on this worker before the first bound
elapses; both fail with the local eviction removed.
* fix(proxy): drop the Redis copies of bulk-evicted auth objects before pacing the broadcasts
* docs(proxy): drop the module docstring that repeated the PR description
* fix(scim): bound bulk cache evictions to one Redis round trip and close lock-order gaps
Bulk key evictions now drop entries from memory, delete them from Redis in one
chunked DEL, and broadcast every key in one pipelined publish, so a large roster
removal no longer overflows the publisher's pending cap or costs a Redis round
trip per member. A push waits for in-flight membership loads before evicting.
User rows are locked in byte order to match the spend writer, a bulk user delete
takes every team lock before its first user write, and a POST whose roster write
fails deletes the empty team so the IdP retry creates it again.
* test(team): observe delete broadcasts at the Redis layer
* fix(scim): lock user rows in id order on every multi-row writer and create unprovisioned members concurrently
* fix(team): chunk the email lookup behind the member row lock
* fix(scim): create a member deleted mid-push again instead of failing the group write
* fix(scim): refuse strict-mode mid-push deletes with 400 and check license seats once per batch
A member deleted mid-push is created again only when scim_upsert_user allows it,
under the verb's own created_via. New group members are checked against the
license seat limit as one batch before any is created. Auth cache publish drops
log the key count instead of the key.
* fix(scim): lock member rows while recomputing admin roles after a group push
* fix(users): re-read teams under the row locks in bulk user delete so a racing group push keeps no deleted member
* fix(users): take /user/delete and SCIM DELETE /Users users off their current teams under the row locks so a racing group push keeps no deleted member
* fix(scim): keep retrying a push while deletes race it, lock rows in member bulk delete, evict removed keys from Redis first
A group push gave up after one retry when SCIM user deletes kept landing between attempts, answering 500 in 9 of 10 race rounds. It now retries while each attempt finds newly deleted members, and gives up only when a member it created again goes missing a second time.
The team member bulk delete now locks the removed members' user rows in the shared order and takes the team off their teams arrays in one array_remove statement, so it no longer writes back a teams array read before a concurrent push added another team.
A push that removes members now drops their deleted team keys from memory and Redis together with the members' other cache entries before the team eviction, so a removed key cannot refill from Redis mid-cleanup.
* fix(user-delete): keep the team alias label on the team members gauge after removals
* test(integration): count relay statements over plaintext so an SSL-enabled Postgres does not hide them
* fix(user-delete): skip a team deleted while the user delete waited on its lock instead of answering 404
* fix(scim): answer 404 and roll back when the team is deleted during a roster write
* test(integration): audit cells for SCIM bulk roster writes, user delete locking and the eviction broadcast
* feat(proxy): report when each per-model budget window resets
model_max_budget_usage on /key/info, /user/info and /team/info now carries budget_reset_at per model, the open window's start plus its period, or null when no window is open. The window start is read in the same batched lookup as the spend counter
* fix(proxy): count a monthly budget window from its own start when reporting its reset
duration_in_seconds measures Nmo from now, so a window opened on January 15 and read on February 10 reported February 12, then null while the counter still refused requests until February 15. duration_in_seconds_from measures the month from the window start, and duration_in_seconds now shares that month arithmetic
* fix(proxy): measure months from timezone-aware datetimes and assert whole usage values
datetime.fromtimestamp without a tz tripped DTZ006. The month arithmetic now runs on an aware local datetime, which keeps the same wall-clock result. The duration tests move to parametrized functions, since subTest payloads carrying datetimes cannot cross the xdist worker boundary, and the new usage tests compare the whole entry
* fix(budgets): end a monthly budget window a calendar month after it opens
Each charge recounted the month from the charge date, so a monthly window
opened in a month of another length expired days before or after the reset
/key/info reports. The window is now measured from its start, by the same
helper the usage report uses. A period ending after year 9999 reports no
reset instead of failing the read.
* fix(budgets): count monthly budget windows in UTC for every router budget
Months were counted on the host's local calendar, so replicas with different TZ
settings disagreed on when a monthly window ends. Provider, deployment, tag and
per-model budgets now all measure a window from its start in UTC, and a test
covers the router budgets. A reset past year 9999 is documented as null.
* test(budgets): cover monthly tag budgets in the router window test
* test(budgets): add integration cells for budget_reset_at and monthly budget windows
* test(budgets): find a monthly window in another span on every calendar day
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): list models from team access groups
* 修复 PR #32599 的类型检查并补齐模型列表回退测试
* fix(proxy): keep a key's own model list narrow when adding team access group models
A key with its own model list was shown every model from its team's access
groups on /v1/models, /v1/model/info and /model_group/info, though auth
denies those calls. Team access group models now widen a key's list only when
the key inherits the team's models. /v1/model/info reuses the shared listing
helper without a catch-all, and route-level tests replace the mock-only ones
* fix(proxy): skip the list build in the unrestricted model info check and test every listing route
The unrestricted check for /v1/model/info only needs to know whether the key
and team lists are empty, so it no longer builds and expands the model list a
second time. The route test now covers /v1/models and /model_group/info next
to both model info routes
* fix(proxy): scope model info for personal and agent keys with no model list
* fix(proxy): keep config keys' team access groups and own-team team_id scope in model listing
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(cost): add tier 1 claude messages stream twins and recapture drifted usage shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): freeze the login throttle clock so the retry-after assertion is deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): honor end_user_id_jwt_field on the JWT-to-virtual-key mapped path
A JWT mapped to a virtual key dropped the end-user claim, so spend rows had
no end user and customer budgets, limits and region never applied. The mapped
token now carries the claim, and it takes precedence over the body user and
the x-litellm-end-user-id header, which remain the fallback when the claim is
missing or empty.
* test(jwt): budget-check the body or header end user when a mapped JWT has no end-user claim
* fix(jwt): replace the annotation offsets with a targeted pyright suppression to keep LIT001 flat
* test(cost): add Azure AI Claude Haiku 5.5 cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): declare service tier cache price fields on CostMapEntry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): drop CostMapEntry reorder now declared on main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): open faked stream items empty so tool arguments and text arrive once
* fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty
* fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once
* test(integration): cover the empty-open Responses stream items on every fake, replay and block path
Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL
* feat(rust-bridge): scope catalog rules with a contextvar
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): select native integration routes with scoped rules
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): run the Rust extension suite without LITELLM_RUST
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* test(rust-bridge): use scoped catalog rules in dispatch tests
Co-Authored-By: Yujong Lee <yujong@berri.ai>
* refactor(rust-bridge): keep catalog rules a static global and override them once per test
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(providers): add Inference APIs as OpenAI-compatible provider
Registers https://api.inferenceapis.com/v1 as a JSON-configured provider (chat completions and Responses API).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* fix(providers): list Inference APIs in Add Model and the provider docs index, add request-level tests
- provider_create_fields.json entry so the Add Model dropdown test passes
- provider_endpoints_support.json (+ backup) entry so the provider docs check passes
- tests now capture the outgoing HTTP request (URL, auth header, body) instead of only inspecting registrations
- drop a redundant comment
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(providers): inject a recording HTTP session instead of patching httpx.Client
Request-level tests now set litellm.client_session to an httpx.Client with a MockTransport, so nothing is
monkeypatched on a class.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* chore(providers): drop redundant comment on the inferenceapis entry
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011qUBY2P8qTTuJCUeMzTCD4
* feat(inferenceapis): add cost map rows for the 12 chat models and test the responses route
Prices and context windows come from https://inferenceapis.com/pricing and GET /v1/models.
Moves the provider test to tests/unit and adds the interactions flag the endpoint matrix now carries.
* feat(inferenceapis): price the vendor's chat model aliases and mark /v1/messages supported
* fix(inferenceapis): serve /v1/messages natively and keep audio routes off the provider
* fix(json providers): send provider-specific params in extra_body and report context overflow as ContextWindowExceededError
* fix(exception mapping): match the Inference APIs overflow phrase instead of any context_length_exceeded substring
* fix(exception mapping): map context overflow to ContextWindowExceededError for Inference APIs only
Providers without their own mapper keep reporting an overflow as BadRequestError, since the router raises a ContextWindowExceededError instead of trying generic fallbacks when context_window_fallbacks has no entry for the group. Adds the integration cells for the provider, its fallbacks, and the extra_body and overflow families.
* test(inferenceapis): hold upstream replies across the worker kill in the chaos cell
A reply the killed worker sent before SIGKILL can lose its queued spend row, which the
product never promised to keep. Holding the upstream until after the kill means only the
surviving worker answers, so every 200 must log exactly once.
* test(inferenceapis): make the chaos cell kill the quieter worker and always release held replies
The survivor now must hold at least one in-flight request, so the burst check cannot pass as 0 == 0, and the held vendor is released on exit so a failed assertion cannot stall the wire server teardown
---------
Co-authored-by: TaylorHawkes <857858+TaylorHawkes@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(streaming): keep provider headers on MidStreamFallbackError
A 429 raised while opening a stream is wrapped in MidStreamFallbackError,
which dropped the litellm_response_headers exception_type() attaches to
the wrapped error. retry-after and x-request-id reached callers on the
non-streaming path but not on the streaming one, so retry backoff,
provider request id logging and proxy error headers lost them.
* refactor(streaming): drop explanatory comment from MidStreamFallbackError header copy
* test(streaming): cover sync and async header carry on MidStreamFallbackError
Parametrizes the provider-header test over the sync and async stream
wrappers, and drops the hasattr test, which pinned how the attribute is
stored rather than what any caller observes
* test(streaming): add integration cells for stream-open provider headers
Cover the SDK stream openers (gemini, vertex_ai, codestral, predibase and the
eager providers), the Router stream retry and fallback paths, and the proxy on
both the router and the config-less serving paths, including a mixed burst, a
worker SIGKILL and a SIGTERM restart. Add a helper that boots a config-less
proxy owned by the test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): route Model Garden models through the generateContent adapter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vertex_ai): annotate generateContent route test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): type litellm_params in get_vertex_ai_model_route
* fix(vertex_ai): bill and log Model Garden generateContent calls like chat completions
Streamed Model Garden calls through the generateContent adapter now request
stream usage where the deployment supports it, keep cached-token details, and
look up the provider-relative cost key, so they bill like the non-stream call.
The adapter forwards router metadata, passes upstream errors through with their
status, countTokens falls back to local counting for non-Gemini routes, and the
native Gemini config is limited to the Gemini routes.
* fix(google_genai): keep adapter stream usage, admin opt-out, single failure count, and base_model routing
Streamed generateContent through the chat adapter now ends with the upstream usage, honors
always_include_stream_usage false and the caller's stream_options, counts one failure per
failed call, and routes Vertex deployments by their base_model like chat does
* fix(google_genai): ask for stream usage on every adapter stream so Claude streams end with usage
* test(google_genai): expect the trailing usage chunk on bridged OpenAI streams
The OpenAI bridge test now sends the usage chunk OpenAI returns for include_usage and expects it translated into usageMetadata. The response_format support helper goes back to its single-purpose form since its only other caller is gone.
* fix(google_genai): end adapter streams with one STOP event carrying usage, and keep upstream usage after the finish chunk
* fix(google_genai): send the held STOP event when no usage arrives, keep a caller's stream_options, and route countTokens like generateContent
* test(integration): cover the moved Vertex routes, adapter stream usage, the shared iterator and usage settings over the wire
* test(integration): add cache-hit twins for the stream usage setting and non-stream cells for every iterator member
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
