Stamping model_group on a batch retrieve routed the whole batch job's token usage
into the per-model-group counters that usage-based, latency-based, cost-based and
least-busy routing read, so polling a finished batch could exhaust a group's TPM or
RPM window and lock live chat traffic out with RouterRateLimitError. Polling also
drove the least-busy in-flight counts negative once per poll per deployment, which
pinned chat to whichever deployment had been polled most.
The strategy callbacks now skip batch retrieve call types, so a retrieve still lands
in spend logs under its model group while the numbers that pick a deployment for the
next chat request stay driven by live traffic only.
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
A fake-streamed provider hands the adapter one chunk carrying both the
delta payload and the finish_reason, which is exactly what the combined
chunk splitter exists for, but its content check never listed the refusal.
The translation short-circuits on finish_reason, so that refusal text was
dropped and the client got `stop_reason: refusal` over an empty content
array, the symptom this PR set out to fix.
Both refusal accumulators also drop their `mutable-ok` lists for a plain
string attribute
The first-delta guard read `delta.refusal` directly, while the translation
three lines later goes through `openai_chat_refusal_text`, which also reads
the `provider_specific_fields` LiteLLM parks unrecognized fields in. A
provider that sends the refusal that way had its only refusal delta skipped
as blank, so the client got `stop_reason: refusal` over an empty content
array, which is the symptom this PR set out to fix
The /v1/messages adapter lowers a tier the entry does not accept, so dropping max from the astra
rows moves that path from Foundry's 400 to a request at xhigh. Nothing pinned that, and the guard
test's docstring named gpt-6-astra as the only gpt-5 name with an azure_ai row, which 11 rows
contradict.
Stamping model_group let both router deployment callbacks past their
`model_group is None` early return for batch retrieves. A batch reports the
whole job's token total on retrieve and reports it again on every poll of the
finished batch, so those tokens are not load in the current minute: three polls
of one completed 1,200 token batch pushed a tpm:1000 deployment to 3,600. The
fan-out also probed unrelated deployments, adding an rpm tick to each.
`vertex_ai/lyria-3-clip-preview` and `vertex_ai/lyria-3-pro-preview` were
registered with `supports_vision`, `supports_image_input`, and an `image`
modality, which contradicts their `gemini/lyria-3-*` siblings and makes
/model/info advertise image input on text-to-music models.
The new Lyria passthrough branch runs before the image-generation branch
and keys on the same `predictions[0].bytesBase64Encoded` shape imagen
returns, so only the cost-map lookup separates them. Cover an imagen
predict response end to end so a future change that drops that lookup
fails here instead of misbilling images as audio.
The takeover of a $0 row an older proxy left behind used to charge the batch when
the update could not reach the database. That leaves the row still reading $0, so
every later retrieve finds the same row and charges the batch again, which is the
repeat charging this PR exists to stop. The retrieve that does take the row over
is the one that charges, and a batch nobody retrieves again after that failure is
never charged, the same as one whose proxy died inside the write window.
Google prices Lyria per generated clip, so every Vertex Lyria entry in the
price map now carries a single output_cost_per_image and both the speech
and the passthrough cost paths read that one field. The old
output_cost_per_second and audio_seconds_per_prediction pair assumed a
30 second clip, which does not match the 32.768 second WAV Vertex returns,
and no other model in the map priced audio that way
Drops max_audio_length_hours and max_audio_per_prompt from the price map,
its schema, the generator, and ModelInfo, since nothing reads them, and
drops the audio_mime_type hidden param for the same reason: the response
already carries the resolved content type on its own header
Folds the per-model bundled catalog lookups into one cached parse of the
local cost map, validated with a TypeAdapter over a ReadOnly TypedDict
A proxy without this fix wrote the batch's cost row on every poll while the batch
was still running, so that row reads $0 and the insert that claims the charge has
nowhere to land. The retrieve that charges the batch now writes its own payload
over that row under a where clause that still names spend 0.0, so exactly one
retrieve takes it over and every later one reads the charge and charges nothing
gpt-6-astra is the only gpt-5-family name with an azure_ai row. Prefixing the rest
cost them every effort flag, since get_llm_provider sends an azure_ai name down the
azure provider when a global AZURE_AI_API_BASE points at an openai.azure.com host and
azure/<model> is not a key either, which turned temperature, top_p and logprobs on
azure_ai/gpt-5.1-chat-latest from accepted into an UnsupportedParamsError.
Both Azure routes refuse it. A live call to the same deployment through
openai/deployments/gpt-6-astra/chat/completions on api-version 2025-04-01-preview
answers reasoning_effort max with a 400 unsupported_value naming none, low, medium,
high and xhigh as the values it takes, and xhigh returns 200, so azure/gpt-6-astra
and azure/us/gpt-6-astra now match the azure_ai row.
A proxy running the old code wrote <batch id>_batch_cost at $0 every time it polled a batch that was still running, so after an upgrade the claim found that row and read it as proof the batch had already been charged. Only a row that recorded a charge counts now, which leaves those $0 rows, and any row a client planted under the batch id, to be charged over
disable_spend_logs skipped the claim entirely, so under that setting every retrieve of a finished batch charged again. The claim now runs either way and writes the one row per batch that makes the charge exactly once, while the per-request logs stay off
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
Both bridges opened an empty text block on a refused streaming turn and
closed it without a single delta, so a client replaying that assistant
turn got HTTP 400 "text content blocks must be non-empty" from Anthropic.
The safeguard-refusal fallback that motivated withholding the text only
runs on the awaited non-streaming response, so nothing needed it withheld
Move the refusal readers into the shared messages/utils helpers so the
adapters stop reaching into each other's private statics, which is also
what put reportPrivateUsage over its budget
Foundry rejects reasoning_effort max on the gpt-6-astra deployment with a 400 that
names none, low, medium, high, and xhigh as the supported values, so the card no
longer lists max. The request path never gated max (only xhigh is opt-in), so this
only changes /model_group/info and router capability gating. The azure/ twin stays
as is because it was not verified on an Azure OpenAI host
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card
for the reasoning_effort none gates, so temperature and top_p were refused while the
azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series
params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/
prefix the way the azure route does
Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a
flex service tier cost test that fails at the merge base, and keeps the wildcard test
from stripping azure_ai/gpt-6-astra out of the provider set
The cost callback used to look for an existing `<batch id>_batch_cost` row before charging a
completed batch, which left a window where concurrent retrieves on any instance all charged the
key, and it would honor a row any request had written under that id. The spend update writer now
inserts the batch cost row itself with `create_many(skip_duplicates=True)` and only the retrieve
whose insert lands charges the key, team, and user. An existing row only takes the charge when it
is a successful `aretrieve_batch` row, so a client-chosen `x-litellm-call-id` on another endpoint
cannot suppress billing. Batch cost rows no longer get their own immediate flush path
`batch_cost_is_final` now treats the proxy's normalized `complete` status like `completed`, which
the enterprise batch cost poller relies on when it decides whether a completed batch is safe to
retire. Tests build that status with `model_copy` since the OpenAI `Batch` model rejects it
The `test-quality-ok` markers sit on the `patch(` lines the gate keys on, and the logging tests no
longer wrap the priced retrieve in `contextlib.suppress`
The first canary run failed pass 1 because the stage-mirror config had no
openai-text-embedding-3-small while test_llm_api_routes_group_grants_every_llm_endpoint
calls /embeddings with it; the public log named the test, which is the
behavior the previous commit added
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
CLAUDE.md asks bug fixes to extend the existing mapped test file rather than
add a new one, and tests/test_litellm/test_router.py already covers
Router.aretrieve_batch. Fold the two cases in next to that coverage and drop
the standalone file.
The helpers are prefixed so they read unambiguously in a shared file, and the
respx context stays open while the payload is awaited, since the usage
accounting reads the batch's output file from the success handler.
Signed-off-by: mynkyu <mynkyu@bucketplace.net>
The test-quality gate flagged the first version for patching an SDK internal
(litellm.batches.main.vertex_ai_batches_instance) and for writing
litellm.callbacks directly.
Drive an openai-compatible deployment through respx instead, so the retrieve
call and the usage accounting that reads the completed batch's output file both
run for real, and install the collector with monkeypatch so nothing leaks into
the next test.
Signed-off-by: mynkyu <mynkyu@bucketplace.net>
Batch token usage is accounted on the retrieve call, not on create: a provider
only reports token counts once the job finishes, so the usage arrives on
aretrieve_batch and that is the spend log row the tokens land on.
Router.acreate_batch stamps the requested model group into its metadata, but
Router.aretrieve_batch never did. A batch is retrieved by id, so the request
carries no model, and the router fans the lookup out over its deployments -
leaving model_group unset on the one record that carries the tokens.
/global/activity/model groups the spend logs by model_group, so every batch's
tokens were bucketed under an empty group.
Stamp the model group inside the per-deployment retrieve attempt, preferring an
explicitly requested group and otherwise using the model_name of the deployment
that answered, which is unambiguous even when the request named no model. An
existing model_group in the metadata is left untouched, so nothing that already
resolves a group changes.
Scope is limited to aretrieve_batch: acompletion, aresponses and acreate_batch
logging are untouched, and cost/spend attribution by model is unchanged.
Signed-off-by: mynkyu <mynkyu@bucketplace.net>
The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.
Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.
Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
The chat and Responses bridges serialize tool_use blocks with model_dump(), so every
bridged /v1/messages response carried LiteLLM's internal provider_specific_fields key
(null, or a Gemini thought signature). Clients replay the block verbatim, and the next
turn that lands on a native Anthropic deployment (auto-router tier change, model swap)
is rejected with "tool_use.provider_specific_fields: Extra inputs are not permitted"
Strip the key from replayed content blocks at the single native Anthropic dispatch so
already-poisoned transcripts self-heal on every native provider, and stop emitting the
null on new responses. The bridges keep reading the signature for the Gemini round trip
Closes#19739
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
