* fix(langfuse): give session-header calls their own trace id
A client that sends only a session header (x-litellm-session-id, a vendor
x-<name>-session-id such as Claude Code's X-Claude-Code-Session-Id, a bare
x-session-id, or the Codex session/thread/conversation family) has that value
stamped into both trace_id and session_id by the proxy. Langfuse upserts a
trace by id, so every turn of a session collapsed into one growing trace and
the Sessions view showed "Total traces: 1"
Detect that aliasing in the Langfuse callback from the request headers the
callback already receives, and use litellm_call_id as the trace id for those
calls. session_id still carries the header value, so the turns stay grouped
under one session. An explicit x-litellm-trace-id, langfuse_trace_id, or
langfuse_existing_trace_id keeps its trace id, including when the caller sets
it to the same value as the session id
Co-authored-by: jesus <jesus@berri.ai>
* test(langfuse): cover direct-SDK callers without proxy request headers
* fix(langfuse): preserve session trace provenance
---------
Co-authored-by: jesus <jesus@berri.ai>
The completed marker batch is now retrieved by its raw provider id with this
run's key, so the proxy prices it inline and the {provider_batch_id}_batch_cost
row must join that key's token and alias. The CheckBatchCost poller only bills
batches it created in the same database, which a stack booted fresh per run
never holds for a completed marker, so the old unified-id assertion had no row
to find. Every run also replays one callback log through
POST /v1/rust_control_plane/logs, the third spend writer, and asserts its row
joins the key like the eight request paths
A team-scoped auto-router is stored under an internal
model_name_{team_id}_{uuid} with the caller-facing name in
model_info.team_public_model_name, and the four pre-routing strategy
registries key on that internal name. A team key asks for the public name,
so the strategy lookup missed, the team early-resolve exit handed back the
marker deployment itself, and every call 400'd with "Unmapped LLM provider".
The strategy lookup now resolves the requested name through the same
team-first, then global, then admin-across-teams deployment resolution the
deployment path uses, and looks the registries up under the model_name of
whatever that resolves to. Both exits of _common_checks_available_deployment
drop strategy markers through one helper, so a marker-only resolution is
rejected as uncallable on every path. The request team id has one reader.
Resolves LIT-7363
Claude-Session: https://claude.ai/code/session_01NU97S7d2FUDDvTk59k53Wp
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
It is a functional e2e test, not a load test, so give it its own workflow instead of a job
inside the load anomaly run. It keeps the Saturday 12:00 UTC cadence and manual dispatch, and
boots the timeout-config proxy with Postgres and Valkey services exactly as before.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
Add tests/e2e/router/test_redis_timeout_e2e.py against a proxy booted from
tests/e2e/gateway/redis_timeout_ci_config.yml: a real Redis with socket_timeout 0.001, so every
command times out and the circuit breaker opens, plus a primary deployment that always fails and
falls back to a healthy one, so every request carries retry breadcrumbs into cost tracking. The
test drives twenty chat requests through the proxy and asserts each answers within ten seconds,
the last third is no slower than the first, /health/liveliness stays fast, and every request
still reaches the spend log.
Gate it behind the redis_timeout marker and E2E_REDIS_TIMEOUT, exclude it from the per-PR
e2e-changed selector, register the reliability.circuit_breaker.redis_timeout.stays_responsive
cell, and run it as its own job in the weekly load anomaly workflow with a Postgres and Valkey
service. Against a v1.100.0 proxy the run wedges the worker: requests time out and liveliness
stops answering (LIT-6780).
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014ZDULyJPp17ZFiJenRxs2T
* fix(proxy): accept non-string callback vars in default_team_settings
A YAML boolean such as turn_off_message_logging: true in a
default_team_settings block failed TeamCallbackMetadata's str-only
callback_vars validation and errored the request before any callback
ran. Stringify the value the same way AddTeamCallback does.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop docstring from default_team_settings bool regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move default_team_settings bool regression test to mapped pre_call_utils suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
LiteLLM_JWTKeyMapping_token_fkey was created ON DELETE RESTRICT, so deleting
a virtual key that a JWT mapping pointed at failed with a foreign key
violation on every deletion path (/key/delete, Admin UI, alias delete,
team and user cascades). Declaring onDelete: Cascade on the relation lets
the database clean the mapping up uniformly, so the next JWT call from that
identity re-registers against the newly created key.
Rebase of #33703 onto current staging.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
When a provider returns choices as null, an object, a string or a number, the
converter said the response had no 'choices' even though the key was present in
the raw keys it listed. A shared message now keeps the old wording for a missing
key and names the offending type otherwise.
The cached-stream regression test also pins the chunk count so a leaked extra
chunk fails it.
cancel_batch resolved credentials and polled job status without the
deployment's aws_session_tags, so on a tag-gated trust policy batch
create and retrieve succeeded while cancel failed with AccessDenied.
Thread the tags through both calls and cover it with a regression test
that fakes STS behind a tag-gated trust policy.
* fix(proxy): keep a body litellm_session_id in SpendLogs under missing_session_id omit
Under general_settings.missing_session_id: omit, apply_missing_session_id_policy now
mirrors a client-supplied top-level litellm_session_id into metadata.session_id when the
client did not set one there, so SpendLogs.session_id and Langfuse agree with the session
callbacks already report through StandardLoggingPayload.session_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep client metadata.session_id ahead of body litellm_session_id on litellm_metadata routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop docstrings from the missing_session_id omit regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
One aliased key owned by a user with an email drives chat, queued chat,
messages, responses, embeddings, the Gemini passthrough, a batch file upload,
and a batch create against a live proxy. Each row must carry api_key equal to
the key's LiteLLM_VerificationToken.token and the alias in metadata, and
/spend/logs?api_key= and /user/daily/activity must report the key with its
alias and email. Health-check rows must keep the literal service-account key,
and the batch cost row for a completed marker batch must join the key that
created it. A re-hashed api_key (the v1.99.0 regression fixed by #39568 and
#39572) now fails the Buildkite e2e stage naming the write path
Resolves MAT-180
initialize_azure_sdk_client now falls back to litellm.constants.DEFAULT_MAX_RETRIES
when litellm_params carries no max_retries, so off-router Azure clients (files,
batches, fine-tuning, assistants, audio) honor the env var like OpenAI clients do.
Router paths already default max_retries to 0 and are unchanged.
Regression tests cover the default, explicit 0/5/None values, and the env var
reaching the SDK client in a fresh interpreter.
When litellm_params does not include max_retries (the common case for
deployments configured without explicit retry settings),
initialize_azure_sdk_client() previously passed None through the
'if max_retries is not None' guard, resulting in AsyncAzureOpenAI being
created without a max_retries argument. The OpenAI SDK then uses its
own hardcoded default of 2, ignoring the DEFAULT_MAX_RETRIES env var.
Fix: fall back to litellm.constants.DEFAULT_MAX_RETRIES when
litellm_params has no max_retries. This ensures the SDK client
respects the configured retry count.
Steps to reproduce:
1. Set env var DEFAULT_MAX_RETRIES=0
2. Configure a deployment without explicit max_retries in litellm_params
3. Make a request that triggers a timeout
4. Observe: SDK retries (x-stainless-retry-count=1) despite env var=0
Related: https://github.com/BerriAI/litellm/issues/5124
A streamed /v1/responses request with tool_choice {"type": "function", "name": ...}
that reaches a chat-completions-only deployment failed with HTTP 500 before the
first byte: the synthetic response.created and response.in_progress events copied
the chat-shaped tool_choice into ResponsesAPIResponse, whose ToolChoice type expects
the flat Responses API shape. The non-streamed path echoed "auto" regardless of the
request.
Both paths now normalize the request's tool_choice through the existing chat
transform and map it back to the Responses API vocabulary, validated by a
TypeAdapter(ToolChoice), so a named function is echoed as {"type": "function",
"name": ...} and a missing tool_choice is echoed as "auto".
Fixes#33689
Skipping the name write-back in either handler left every test green; a
guardrail that renames a tool call now has a regression test on both the
chat chunk path and the Anthropic SSE path
Preset #40341 pointed the Anthropic family REASONING tier at claude-fable-5-1,
but this test still hardcoded claude-opus-5, so the payload it saw no longer
matched. Rebase the assertion on ANTHROPIC_PRESET.complexity_router_config.tier_model_configs
so a preset refresh flows through instead of redding the suite on staging.
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Krrish Dholakia <krrish-berri-2@users.noreply.github.com>
The bridge now asks for reasoning.encrypted_content whenever the provider's
Responses config lists include, independent of the client's thinking block,
and leaves it out for providers such as Perplexity that reject the param.
Bridge-tagged blocks are stripped on the chat adapter path too, so a mid
session model switch to Gemini or Bedrock no longer forwards them as real
signatures, a bare prefix counts as bridge-tagged, and non-mapping messages
pass through the strip untouched.
A stream cache hit on an entry stored with choices == [] indexed choices[0]
in the cached_response branch and failed with IndexError, so the streaming
converters' empty chunk had no working consumer. The branch now treats a
chunk without choices as empty and lets the wrapper close the stream with
its usual finish_reason stop chunk
Post-call guardrails on /v1/responses only treated function_call output
items as tool calls, so a custom_tool_call item (Codex's exec shell tool
on GPT-5.6 models) was never scanned or masked, non-streaming and
streaming alike. Both item types now flow through the shared
tool_call_dict_from_output_item helper, ended-stream delivery syncs the
custom_tool_call_input delta/done events and the item's input field, and
the completed-response scan key fingerprints both kinds of item.
Non-streaming Responses tool-call MASK rewrites were also never written
back to the output item even for function_call; they are now.
The two test methods, the policy_engine fixture, and the two inner
stubs in TestBackgroundResponseRetrievalGovernance now carry full
parameter and return annotations, closing the Greptile thread that
94f9230d13 left open.
The Databricks chat transformation only parsed reasoning out of FMAPI-style
reasoning content blocks, so external models behind Databricks AI Gateway that
return the OpenAI-style top-level reasoning_content string lost it, both in the
final message and in every streamed delta. Fall back to the shared OpenAI
reasoning helper when no reasoning block exists, and keep the delta's own
reasoning_content when streaming.
Operators can now set aws_session_tags on a Bedrock or SageMaker
deployment and every AssumeRole call carries them as STS session tags,
so trust policies gated on sts:TagSession admit the session and
CloudTrail and cost reports see the tags. Tags are validated up front,
sorted into the credential cache key so tag order does not fork
sessions, stripped from invoke and embedding request bodies, and
blocked from client request bodies like the other AWS identity params.
Based on #34073.
Co-authored-by: Satya Yedida <222552552+satya-rubrik@users.noreply.github.com>
Narrows the no-choices guard so a dict, string, or None still raises the APIError while an empty list passes through,
guards the non-stream Anthropic bridge against indexing an empty choices list, and repairs test_completion_missing_role,
whose raw-response mock was patched in as the create() callable itself so the handler only ever saw a MagicMock
Annotate the return types of dispatch_async and transform_then_dispatch in llm_http_handler and _send_batch in azure_sentinel, and mark four legitimate broad catches with the repo's noqa convention, so the promote PR's lint job passes the strict gate again. Supersedes #40328.
