get_fireworks_session_id fell back to litellm_trace_id when no session id was
given. That id is generated per request (uuid4 when absent), so x-session-affinity
carried a different value every time and Fireworks prompt caching never hit;
cached_tokens stayed 0 across identical prompts.
The None path the original change described was effectively unreachable because
of it. Drop the fallback so affinity comes only from an id the caller actually
supplied: litellm_session_id, session_id, or metadata.session_id.
Callers who were relying on a trace id for affinity can pass litellm_session_id
instead, which is stable across the requests they want grouped.
Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Gemini API Maps-grounded prompts were billed as web search and Vertex AI Maps-grounded prompts were not billed at all. Classify grounding metadata per candidate into web search vs Maps requests, carry a distinct google_maps_grounding_requests usage counter through non-streaming and streaming paths, and price it via the new google_maps_grounding_cost_per_query cost map key with per-query and per-prompt defaults keyed off web_search_billing_unit. Fixes#35906
The image_edit health probe sent a 512x512 solid-gray PNG with the generic
chat prompt "test from litellm", an ambiguous pair OpenAI's gpt-image-1
output moderation sometimes rejects as moderation_blocked, which reported a
working deployment as unhealthy. The probe now sends a blue circle on a
white background with a descriptive edit prompt, and a provider moderation
verdict (ContentPolicyViolationError or a moderation_blocked error body) is
treated as proof the endpoint works rather than as an unhealthy deployment.
Live probes against every gemini_native_audio model on both providers show
setup accepts a valid prebuilt voice and 1007s only unknown voice names, so
the strip predicate rested on a false premise and silently discarded the
client's voice on AI Studio native-audio sessions
reload_search_tools_from_db is a read-modify-write of the shared llm_router
global: it reads the whole table, merges the config tools in, and replaces
router.search_tools wholesale. Two of those interleaving lets the older
snapshot's assignment land last and put back a tool the newer one deleted, so a
revoked tool keeps serving on the provider key it carried until the next reload.
Take MODEL_RECONCILE_LOCK, which add_deployment already uses to serialize the
same shape of work on the same global. It has to go on this entry point rather
than in _init_search_tools_in_db, because _init_non_llm_objects_in_db calls that
while already holding the lock and asyncio.Lock is not reentrant.
A separate search-tools-only lock would not close the race: the periodic
reconcile reaches _init_search_tools_in_db under MODEL_RECONCILE_LOCK, so only
that same lock orders an endpoint refresh against a cron tick.
Ordering across workers is unchanged and still reconciles on the next tick.
Gemini 2.5 Flash Preview TTS, Gemini 2.5 Pro Preview TTS, and the three
gemini-2.5-flash-native-audio entries carried rates copied from the text
models, so audio output was billed 2x to 6x under Google's published
prices. Set the published per-token rates on all ten keys, add
output_cost_per_audio_token to the native-audio entries, and drop the
long-context tier rates Google does not publish for Pro TTS.
The committed snapshot behind /openapi.json for unloaded lazy features had drifted on 30 of 31 fragments and never had one for a2a_registration or gemini_agents, so those routes showed as placeholder GET stubs or old docstrings until traffic loaded them. Regenerate the snapshot and schema.d.ts, make the check-ui-api-types job and make check regenerate the snapshot and fail on drift, and make the generator refuse to write a snapshot when any feature fails to import so a broken import cannot silently drop fragments.
On /v1/responses the prompt template ran inside litellm.aresponses, after the
router had already resolved a deployment and injected its api_key/api_base, so a
prompt whose metadata.model pointed at another provider sent the old
deployment's credentials cross-provider (401). The proxy now runs the prompt
template for aresponses in the pre-call hook, before routing, so the router
picks the deployment that matches the swapped model. As a backstop, the SDK
refuses a cross-provider swap when explicit credentials are already present
instead of forwarding them.
ignore_prompt_manager_model and ignore_prompt_manager_optional_params saved on
a prompt were only read by the generic manager, so dotprompt prompts ignored
them on every endpoint. PromptManagementBase now merges the prompt spec's flags
with the per-request ones for every manager, and the generic manager no longer
drops caller flags when no spec is present.
The three member-budget tests patched litellm internals and asserted only on
the mock, which tripped the TQ002 and TQ008 test-quality ratchet. Fake the
prisma budget table on the shared client and assert on the row that reaches
the database plus the returned team payload.
POST /prompts silently stored an empty template when litellm_params.prompt_id
was combined with prompt_data keyed by template name, because the loader
wrapped the already-keyed dict under prompt_id a second time. The loader now
wraps only a flat template (a dict carrying a content key), and create,
update, and patch reject the ambiguous keyed+prompt_id combination with a 400
that names both valid shapes. The API also returned version null on every
create and lost version, environment, and created_by on registry reload; both
now carry through. Versioned ids like my-prompt.v1, which the create API
itself returns, now resolve to their base template on the SDK prompt hooks,
and a flat DB prompt with no litellm_params.prompt_id registers under its base
API id instead of garbage.
The Default Budget Duration field in Team Member Settings only offered daily, weekly and monthly, so a team member budget could never be set to never reset. It now uses the shared BudgetDurationDropdown, and /team/update writes an explicitly null duration through to the member budget row along with its reset time.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A model_group_alias whose target is an auto-router shows up in /v1/models and
/model_group/info but 400s on call with "Unmapped LLM provider for this
endpoint. You passed model=complexity_router, custom_llm_provider=auto_router".
async_pre_routing_hook picks the strategy using the name the caller passed,
while the alias is only resolved further down in
_common_checks_available_deployment, so the four strategy registries, all keyed
by the marker deployment's model_name, never match. The hook then declines, and
the auto_router/ marker deployment goes out as if it were a real model
Resolve the alias once at the top of the hook, for lookups only, so the
registries, the tag-filtering escape hatch and the marker's forwardable params
all see the name they are keyed under. The caller-facing name is untouched:
spend metadata is stamped before routing and the response still carries the
alias the client sent
Second half, so the same symptom cannot reach a provider through the entry
points this does not fix (the sync selection path that never runs the hook, a
team-scoped router keyed on its internal name), a group that resolves only to
strategy markers is no longer callable: it raises a BadRequestError naming the
marker instead of handing the auto_router/ pseudo-model to the provider
