* ci: limit Rust workflows to Rust directory changes
* ci: run Rust checks when their workflow changes
* ci: report Rust wheels only for successful Rust changes
* ci: keep Rust wheel reports in the workflow summary
* ci: group Rust lint and validation jobs
* ci: keep Rust job names distinct from required lint and test checks
* ci: drop the unused Python setup from the Rust lint job
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
_live_routers was only joined when a router was constructed with a model_list,
but a router built empty is populated through add_deployment, and the empty
branch exists for exactly that. Such a router was invisible to the live-router
scan, so deleting the deployment from another router released the shared
cost-map key while it was still serving that id.
Joining the set where a deployment enters the list covers every path, and it
also lets a price reload rebuild what a dynamically built router serves.
The claim is process-wide but the release was per-deletion, so with two routers
serving one deployment id, the first deletion put the survivor back on merging
and the price it had just cleared would keep billing.
Release the key only once no live router still serves that id.
The ownership ledger only grew. A deleted deployment kept its claim, so if a
later catalog refresh started publishing a model under that same name, the next
registration would treat the catalog entry as the deployment's own and evict it.
Deleting a deployment now gives the key back, which also stops the ledger
growing for the life of the process.
The previous guard read a catalog entry off `litellm_provider`, so a deployment
that declares its own provider in model_info was treated as one and kept billing
at a price it no longer carried. It also only held for a single registration: a
second one under a colliding id saw the id the first merge left behind and
evicted the catalog entry anyway.
Track the cost-map keys this registrar creates instead. A key it created is
evicted before re-registration; one it did not is left to merge, which is what a
deployment id colliding with a catalog model name needs.
Also folds the required-fields comment into the docstring that already gives the
reason.
Deployments are keyed into litellm.model_cost alongside the built-in catalog, so
evicting a deployment's stale entry by id could take a real model's entry with it:
registering a deployment whose model_info.id is "gpt-4o" stripped that model's
pricing, context window and capability flags process-wide, for every other
deployment of it, until the next price-map reload.
Only evict an entry this registration owns. A colliding id keeps the previous
merge, which pollutes the catalog entry rather than emptying it.
Also pins the Admin UI round trip: the model edit form echoes the whole /model/info
row back on save, and that read reports every key the deployment never stored as an
explicit null, so the clear path has to leave those keys alone.
PATCH /model/{model_id}/update merged the patch with exclude_none and then
popped explicit nulls only for the mirrored pricing fields, so a null sent for
max_input_tokens, mode, supports_vision or any other key was dropped and a value
pinned by an earlier save could never be removed.
The route now follows JSON Merge Patch over both blobs: a key absent from the
body is unchanged, a key sent as null is removed from the stored row, and a key
sent with a value is set. Ownership and identity keys keep ignoring a null, as
do the fields the stored models require, since clearing one writes a row no
reload can rebuild. Mirrored pricing keys still clear from both blobs.
Clearing a price also needed the router to stop merging a deployment's cost-map
entry onto its previous registration, which left the old rate in place and kept
billing at a price the deployment no longer carried.
Adds a create, read, partial-update, clear, enforce, delete lifecycle e2e that
reads back on every replica, and a harness helper for that read-back.
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
A fake-streamed provider hands the adapter one chunk carrying both the
delta payload and the finish_reason, which is exactly what the combined
chunk splitter exists for, but its content check never listed the refusal.
The translation short-circuits on finish_reason, so that refusal text was
dropped and the client got `stop_reason: refusal` over an empty content
array, the symptom this PR set out to fix.
Both refusal accumulators also drop their `mutable-ok` lists for a plain
string attribute
The first-delta guard read `delta.refusal` directly, while the translation
three lines later goes through `openai_chat_refusal_text`, which also reads
the `provider_specific_fields` LiteLLM parks unrecognized fields in. A
provider that sends the refusal that way had its only refusal delta skipped
as blank, so the client got `stop_reason: refusal` over an empty content
array, which is the symptom this PR set out to fix
The /v1/messages adapter lowers a tier the entry does not accept, so dropping max from the astra
rows moves that path from Foundry's 400 to a request at xhigh. Nothing pinned that, and the guard
test's docstring named gpt-6-astra as the only gpt-5 name with an azure_ai row, which 11 rows
contradict.
The openai gpt-6-astra card carries supports_prompt_cache_breakpoint, so a Foundry deployment
reported it as true until the azure_ai row took over the lookup. The cache control hook still
honours breakpoints for that deployment through the bare name, so /model/info was the only thing
that changed, and it now agrees with the hook again.
`vertex_ai/lyria-3-clip-preview` and `vertex_ai/lyria-3-pro-preview` were
registered with `supports_vision`, `supports_image_input`, and an `image`
modality, which contradicts their `gemini/lyria-3-*` siblings and makes
/model/info advertise image input on text-to-music models.
The new Lyria passthrough branch runs before the image-generation branch
and keys on the same `predictions[0].bytesBase64Encoded` shape imagen
returns, so only the cost-map lookup separates them. Cover an imagen
predict response end to end so a future change that drops that lookup
fails here instead of misbilling images as audio.
The takeover of a $0 row an older proxy left behind used to charge the batch when
the update could not reach the database. That leaves the row still reading $0, so
every later retrieve finds the same row and charges the batch again, which is the
repeat charging this PR exists to stop. The retrieve that does take the row over
is the one that charges, and a batch nobody retrieves again after that failure is
never charged, the same as one whose proxy died inside the write window.
Google prices Lyria per generated clip, so every Vertex Lyria entry in the
price map now carries a single output_cost_per_image and both the speech
and the passthrough cost paths read that one field. The old
output_cost_per_second and audio_seconds_per_prediction pair assumed a
30 second clip, which does not match the 32.768 second WAV Vertex returns,
and no other model in the map priced audio that way
Drops max_audio_length_hours and max_audio_per_prompt from the price map,
its schema, the generator, and ModelInfo, since nothing reads them, and
drops the audio_mime_type hidden param for the same reason: the response
already carries the resolved content type on its own header
Folds the per-model bundled catalog lookups into one cached parse of the
local cost map, validated with a TypeAdapter over a ReadOnly TypedDict
A proxy without this fix wrote the batch's cost row on every poll while the batch
was still running, so that row reads $0 and the insert that claims the charge has
nowhere to land. The retrieve that charges the batch now writes its own payload
over that row under a where clause that still names spend 0.0, so exactly one
retrieve takes it over and every later one reads the charge and charges nothing
gpt-6-astra is the only gpt-5-family name with an azure_ai row. Prefixing the rest
cost them every effort flag, since get_llm_provider sends an azure_ai name down the
azure provider when a global AZURE_AI_API_BASE points at an openai.azure.com host and
azure/<model> is not a key either, which turned temperature, top_p and logprobs on
azure_ai/gpt-5.1-chat-latest from accepted into an UnsupportedParamsError.
Both Azure routes refuse it. A live call to the same deployment through
openai/deployments/gpt-6-astra/chat/completions on api-version 2025-04-01-preview
answers reasoning_effort max with a 400 unsupported_value naming none, low, medium,
high and xhigh as the values it takes, and xhigh returns 200, so azure/gpt-6-astra
and azure/us/gpt-6-astra now match the azure_ai row.
A proxy running the old code wrote <batch id>_batch_cost at $0 every time it polled a batch that was still running, so after an upgrade the claim found that row and read it as proof the batch had already been charged. Only a row that recorded a charge counts now, which leaves those $0 rows, and any row a client planted under the batch id, to be charged over
disable_spend_logs skipped the claim entirely, so under that setting every retrieve of a finished batch charged again. The claim now runs either way and writes the one row per batch that makes the charge exactly once, while the per-request logs stay off
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
Both bridges opened an empty text block on a refused streaming turn and
closed it without a single delta, so a client replaying that assistant
turn got HTTP 400 "text content blocks must be non-empty" from Anthropic.
The safeguard-refusal fallback that motivated withholding the text only
runs on the awaited non-streaming response, so nothing needed it withheld
Move the refusal readers into the shared messages/utils helpers so the
adapters stop reaching into each other's private statics, which is also
what put reportPrivateUsage over its budget
Foundry rejects reasoning_effort max on the gpt-6-astra deployment with a 400 that
names none, low, medium, high, and xhigh as the supported values, so the card no
longer lists max. The request path never gated max (only xhigh is opt-in), so this
only changes /model_group/info and router capability gating. The azure/ twin stays
as is because it was not verified on an Azure OpenAI host
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card
for the reasoning_effort none gates, so temperature and top_p were refused while the
azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series
params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/
prefix the way the azure route does
Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a
flex service tier cost test that fails at the merge base, and keeps the wildcard test
from stripping azure_ai/gpt-6-astra out of the provider set
The cost callback used to look for an existing `<batch id>_batch_cost` row before charging a
completed batch, which left a window where concurrent retrieves on any instance all charged the
key, and it would honor a row any request had written under that id. The spend update writer now
inserts the batch cost row itself with `create_many(skip_duplicates=True)` and only the retrieve
whose insert lands charges the key, team, and user. An existing row only takes the charge when it
is a successful `aretrieve_batch` row, so a client-chosen `x-litellm-call-id` on another endpoint
cannot suppress billing. Batch cost rows no longer get their own immediate flush path
`batch_cost_is_final` now treats the proxy's normalized `complete` status like `completed`, which
the enterprise batch cost poller relies on when it decides whether a completed batch is safe to
retire. Tests build that status with `model_copy` since the OpenAI `Batch` model rejects it
The `test-quality-ok` markers sit on the `patch(` lines the gate keys on, and the logging tests no
longer wrap the priced retrieve in `contextlib.suppress`
The first canary run failed pass 1 because the stage-mirror config had no
openai-text-embedding-3-small while test_llm_api_routes_group_grants_every_llm_endpoint
calls /embeddings with it; the public log named the test, which is the
behavior the previous commit added
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
The MongoDB Atlas vector store provider imports pymongo lazily from the
opt-in `mongodb` extra, but none of the shipped images installed that
extra. Any image-based deployment that configured a MongoDB vector store
failed at search time with "requires the 'pymongo' package", which the
user cannot fix without extending the image
Adds `--extra mongodb` to every uv sync in the root Dockerfile,
Dockerfile.database, Dockerfile.non_root, and the gateway component
image. The backend component does not serve /vector_stores so it is left
as is. The extra resolves from the existing uv.lock to pymongo 4.17.0
plus dnspython 2.8.0, no lock change needed
(cherry picked from commit 16fd14f537)
