The GA vertex model had no cost map entry, and the realtime cost handler
accepted the router's price-less auto-registered deployment entry for the
session.created model at zero-defaulted rates, so sessions billed 0.0 even
when base_model pointed at the priced preview key. Adds the GA entry at its
published rates and makes the handler fall through zero-defaulted candidates
unless their cost map entry explicitly declares pricing.
get_fireworks_session_id fell back to litellm_trace_id when no session id was
given. That id is generated per request (uuid4 when absent), so x-session-affinity
carried a different value every time and Fireworks prompt caching never hit;
cached_tokens stayed 0 across identical prompts.
The None path the original change described was effectively unreachable because
of it. Drop the fallback so affinity comes only from an id the caller actually
supplied: litellm_session_id, session_id, or metadata.session_id.
Callers who were relying on a trace id for affinity can pass litellm_session_id
instead, which is stable across the requests they want grouped.
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Gemini API Maps-grounded prompts were billed as web search and Vertex AI Maps-grounded prompts were not billed at all. Classify grounding metadata per candidate into web search vs Maps requests, carry a distinct google_maps_grounding_requests usage counter through non-streaming and streaming paths, and price it via the new google_maps_grounding_cost_per_query cost map key with per-query and per-prompt defaults keyed off web_search_billing_unit. Fixes#35906
The image_edit health probe sent a 512x512 solid-gray PNG with the generic
chat prompt "test from litellm", an ambiguous pair OpenAI's gpt-image-1
output moderation sometimes rejects as moderation_blocked, which reported a
working deployment as unhealthy. The probe now sends a blue circle on a
white background with a descriptive edit prompt, and a provider moderation
verdict (ContentPolicyViolationError or a moderation_blocked error body) is
treated as proof the endpoint works rather than as an unhealthy deployment.
Live probes against every gemini_native_audio model on both providers show
setup accepts a valid prebuilt voice and 1007s only unknown voice names, so
the strip predicate rested on a false premise and silently discarded the
client's voice on AI Studio native-audio sessions
reload_search_tools_from_db is a read-modify-write of the shared llm_router
global: it reads the whole table, merges the config tools in, and replaces
router.search_tools wholesale. Two of those interleaving lets the older
snapshot's assignment land last and put back a tool the newer one deleted, so a
revoked tool keeps serving on the provider key it carried until the next reload.
Take MODEL_RECONCILE_LOCK, which add_deployment already uses to serialize the
same shape of work on the same global. It has to go on this entry point rather
than in _init_search_tools_in_db, because _init_non_llm_objects_in_db calls that
while already holding the lock and asyncio.Lock is not reentrant.
A separate search-tools-only lock would not close the race: the periodic
reconcile reaches _init_search_tools_in_db under MODEL_RECONCILE_LOCK, so only
that same lock orders an endpoint refresh against a cron tick.
Ordering across workers is unchanged and still reconciles on the next tick.
The dashboard resolved the complexity-router tier set three different ways: a
private TIER_KEYS in build_complexity_router_config.ts, TIER_ORDER in
complexity_router_tiers.ts, and TIER_KEYS in ComplexityRouterConfig.tsx. The
edit modal went further and re-implemented the whole create payload builder,
kept in sync only by a comment reading "Mirrors buildComplexityRouterConfig".
tier_rows.ts now owns the tier set. Every consumer reads activeTierRows(value)
and a row carries its own id, so the plan-mode floor and per-model params point
at a row rather than at a position, and the leaves that already wanted entries
(buildAutoRouterTestTargets, getRequiredModels, model_info_view) take them.
buildUpdatedComplexityRouterConfig becomes preserve-unmanaged-keys around the
shared builder instead of a second copy of it.
Also drops the literal ", ]" that renders as visible text in two DialogFooter
blocks on the auto-router routing-test and connection-test dialogs, left over
from a JSX array-to-fragment conversion.
No behaviour change: all 566 tests over the touched modules pass with fixture
shape changes only, no assertion edited.
Gemini 2.5 Flash Preview TTS, Gemini 2.5 Pro Preview TTS, and the three
gemini-2.5-flash-native-audio entries carried rates copied from the text
models, so audio output was billed 2x to 6x under Google's published
prices. Set the published per-token rates on all ten keys, add
output_cost_per_audio_token to the native-audio entries, and drop the
long-context tier rates Google does not publish for Pro TTS.
The committed snapshot behind /openapi.json for unloaded lazy features had drifted on 30 of 31 fragments and never had one for a2a_registration or gemini_agents, so those routes showed as placeholder GET stubs or old docstrings until traffic loaded them. Regenerate the snapshot and schema.d.ts, make the check-ui-api-types job and make check regenerate the snapshot and fail on drift, and make the generator refuse to write a snapshot when any feature fails to import so a broken import cannot silently drop fragments.
On /v1/responses the prompt template ran inside litellm.aresponses, after the
router had already resolved a deployment and injected its api_key/api_base, so a
prompt whose metadata.model pointed at another provider sent the old
deployment's credentials cross-provider (401). The proxy now runs the prompt
template for aresponses in the pre-call hook, before routing, so the router
picks the deployment that matches the swapped model. As a backstop, the SDK
refuses a cross-provider swap when explicit credentials are already present
instead of forwarding them.
ignore_prompt_manager_model and ignore_prompt_manager_optional_params saved on
a prompt were only read by the generic manager, so dotprompt prompts ignored
them on every endpoint. PromptManagementBase now merges the prompt spec's flags
with the per-request ones for every manager, and the generic manager no longer
drops caller flags when no spec is present.
The three member-budget tests patched litellm internals and asserted only on
the mock, which tripped the TQ002 and TQ008 test-quality ratchet. Fake the
prisma budget table on the shared client and assert on the row that reaches
the database plus the returned team payload.
