Vertex Gemini 3.x models route through cost_per_character (the cost_router
token-path gate only matches gemini-2), and its token fallbacks dropped
service_tier, so ON_DEMAND_FLEX responses were still billed at the standard
rate. Pass the tier through the call site and all four fallbacks.
The GA vertex model had no cost map entry, and the realtime cost handler
accepted the router's price-less auto-registered deployment entry for the
session.created model at zero-defaulted rates, so sessions billed 0.0 even
when base_model pointed at the priced preview key. Adds the GA entry at its
published rates and makes the handler fall through zero-defaulted candidates
unless their cost map entry explicitly declares pricing.
get_fireworks_session_id fell back to litellm_trace_id when no session id was
given. That id is generated per request (uuid4 when absent), so x-session-affinity
carried a different value every time and Fireworks prompt caching never hit;
cached_tokens stayed 0 across identical prompts.
The None path the original change described was effectively unreachable because
of it. Drop the fallback so affinity comes only from an id the caller actually
supplied: litellm_session_id, session_id, or metadata.session_id.
Callers who were relying on a trace id for affinity can pass litellm_session_id
instead, which is stable across the requests they want grouped.
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Gemini API Maps-grounded prompts were billed as web search and Vertex AI Maps-grounded prompts were not billed at all. Classify grounding metadata per candidate into web search vs Maps requests, carry a distinct google_maps_grounding_requests usage counter through non-streaming and streaming paths, and price it via the new google_maps_grounding_cost_per_query cost map key with per-query and per-prompt defaults keyed off web_search_billing_unit. Fixes#35906
The image_edit health probe sent a 512x512 solid-gray PNG with the generic
chat prompt "test from litellm", an ambiguous pair OpenAI's gpt-image-1
output moderation sometimes rejects as moderation_blocked, which reported a
working deployment as unhealthy. The probe now sends a blue circle on a
white background with a descriptive edit prompt, and a provider moderation
verdict (ContentPolicyViolationError or a moderation_blocked error body) is
treated as proof the endpoint works rather than as an unhealthy deployment.
Live probes against every gemini_native_audio model on both providers show
setup accepts a valid prebuilt voice and 1007s only unknown voice names, so
the strip predicate rested on a false premise and silently discarded the
client's voice on AI Studio native-audio sessions
reload_search_tools_from_db is a read-modify-write of the shared llm_router
global: it reads the whole table, merges the config tools in, and replaces
router.search_tools wholesale. Two of those interleaving lets the older
snapshot's assignment land last and put back a tool the newer one deleted, so a
revoked tool keeps serving on the provider key it carried until the next reload.
Take MODEL_RECONCILE_LOCK, which add_deployment already uses to serialize the
same shape of work on the same global. It has to go on this entry point rather
than in _init_search_tools_in_db, because _init_non_llm_objects_in_db calls that
while already holding the lock and asyncio.Lock is not reentrant.
A separate search-tools-only lock would not close the race: the periodic
reconcile reaches _init_search_tools_in_db under MODEL_RECONCILE_LOCK, so only
that same lock orders an endpoint refresh against a cron tick.
Ordering across workers is unchanged and still reconciles on the next tick.
Gemini 2.5 Flash Preview TTS, Gemini 2.5 Pro Preview TTS, and the three
gemini-2.5-flash-native-audio entries carried rates copied from the text
models, so audio output was billed 2x to 6x under Google's published
prices. Set the published per-token rates on all ten keys, add
output_cost_per_audio_token to the native-audio entries, and drop the
long-context tier rates Google does not publish for Pro TTS.
The committed snapshot behind /openapi.json for unloaded lazy features had drifted on 30 of 31 fragments and never had one for a2a_registration or gemini_agents, so those routes showed as placeholder GET stubs or old docstrings until traffic loaded them. Regenerate the snapshot and schema.d.ts, make the check-ui-api-types job and make check regenerate the snapshot and fail on drift, and make the generator refuse to write a snapshot when any feature fails to import so a broken import cannot silently drop fragments.
On /v1/responses the prompt template ran inside litellm.aresponses, after the
router had already resolved a deployment and injected its api_key/api_base, so a
prompt whose metadata.model pointed at another provider sent the old
deployment's credentials cross-provider (401). The proxy now runs the prompt
template for aresponses in the pre-call hook, before routing, so the router
picks the deployment that matches the swapped model. As a backstop, the SDK
refuses a cross-provider swap when explicit credentials are already present
instead of forwarding them.
ignore_prompt_manager_model and ignore_prompt_manager_optional_params saved on
a prompt were only read by the generic manager, so dotprompt prompts ignored
them on every endpoint. PromptManagementBase now merges the prompt spec's flags
with the per-request ones for every manager, and the generic manager no longer
drops caller flags when no spec is present.
