num_retries_per_request has always capped the retries of one request with its fallback hops included. #40930 started reading the per-hop attempted_retries counter instead, and every fallback hop restarts that counter at zero, so a request could spend a fresh retry budget on each hop and the legacy fallback cap test started seeing the hop run.
Router.log_retry now also keeps request_retry_count on the request metadata, incremented on every retry and fallback hop and never truncated the way previous_models is, and max_retries_per_request_hit reads that count. The flat retry records, the litellm_metadata coverage and caps above four from #40930 stay as they are, and the legacy test goes back to its previous_models == 0 assertion.
get_litellm_params() keys such as model_alias_map, completion_call_id and
custom_prompt_dict were missing from all_litellm_params, so when a bridge
forwarded litellm_params as completion kwargs they were swept into extra_body
and OpenAI rejected the call with "Unknown parameter: 'model_alias_map'".
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
_client_async_logging_helper re-submitted logging_obj.success_handler to the
executor after _dispatch_success_logging had already done so, running the same
success pipeline twice per async request and racing on shared logging state.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Router.log_retry used to copy the failed attempt's kwargs and metadata into
metadata.previous_models. Nothing downstream read those copies, but they carried
client credentials into spend logs and grew the payload on every retry. Each
attempt now leaves a flat record (model group, deployment id, exception type and
string, attempt number), which drops RETRY_BREADCRUMB_EXCLUDED_KWARGS and the
per-retry credential masking.
num_retries_per_request was enforced from len(previous_models), which only
looked at the metadata bucket and never exceeded four records. The sync and
async client wrappers and the Rust lifecycle guard now read attempted_retries
from whichever metadata bucket the call carries.
Resolves LIT-7505
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Every proxy and router cost lookup goes through get_model_info, which copies
cost map keys explicitly, so the new audio cache-read branch always fell back
to the text cache-read rate there. Copy the key so models whose audio
cache-read rate differs from the text one bill cached audio correctly.
* fix(logging): finish response metadata before the sync logging thread reads it
The async and sync client wrappers handed the response to the threaded success handler before computing its cost, call id, and api_base, so that thread inserted into the same metadata dict the request coroutine was still iterating and a finished chat completion turned into a 500 (dictionary changed size during iteration). Metadata is now finalized first, and the merge and header copies snapshot their dicts before iterating.
* fix(logging): snapshot metadata with a dict copy and drop redundant comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): copy metadata via dict.copy and dedupe Final import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(mock): emit admission-time usage chunk on streaming mock_response
Streaming mock_response chunks carried no usage, so the chunk builder re-tokenized the whole prompt in Python after the stream ended even when budget reservation had already counted it at admission. The mock streaming generators now yield a final usage-only chunk carrying the admission prompt count (same completion count as the non-streaming path). Without an admission count the old tokenizer fallback stays.
* fix(mock): type the mock stream generators and keep the usage chunk on the content stream id
Review follow-up: the usage-only chunk was built with a fresh id, so CustomStreamWrapper switched response_id for the finish-reason and usage chunks. It now copies the content stream id. The generators also get full parameter and return annotations.
---------
Co-authored-by: yassin <yassin@berri.ai>
* fix(proxy): load db credentials inside the model reconcile so a worker never serves a model before its credential
* fix(proxy): load db credentials in the model read-through so a request miss never adds a model before its credential
* fix(proxy): read credentials from the writer db before the router update and look a credential up once
* test(proxy): assert the credential is loaded when db models reach the router instead of the call order
Resolve the conflicts in utils.py, types/router.py, and the tests, and collapse the 56 per-provider isinstance(drop_params, bool) gates to bool(drop_params) now that get_optional_params normalizes the flag once at the top
`vertex_ai/lyria-3-clip-preview` and `vertex_ai/lyria-3-pro-preview` were
registered with `supports_vision`, `supports_image_input`, and an `image`
modality, which contradicts their `gemini/lyria-3-*` siblings and makes
/model/info advertise image input on text-to-music models.
Google prices Lyria per generated clip, so every Vertex Lyria entry in the
price map now carries a single output_cost_per_image and both the speech
and the passthrough cost paths read that one field. The old
output_cost_per_second and audio_seconds_per_prediction pair assumed a
30 second clip, which does not match the 32.768 second WAV Vertex returns,
and no other model in the map priced audio that way
Drops max_audio_length_hours and max_audio_per_prompt from the price map,
its schema, the generator, and ModelInfo, since nothing reads them, and
drops the audio_mime_type hidden param for the same reason: the response
already carries the resolved content type on its own header
Folds the per-model bundled catalog lookups into one cached parse of the
local cost map, validated with a TypeAdapter over a ReadOnly TypedDict
A live cost map older than this release, or a proxy whose map fetch lags, could
strip `reasoning` from a model this release knows accepts it. The bundled map is
now the floor: any OpenAI entry it flags as reasoning keeps the param whatever
the live map says. Fine-tuned ids with an empty suffix (`ft:gpt-4o-2024-08-06:org::id`)
now resolve to their base entry instead of failing open, `chat-latest` carries
the flag, and the schema test keeps every codex, deep-research, and chat-latest
entry flagged. The none-effort check goes through a public wrapper so the
responses config stops importing a private helper.
