cancel_batch resolved credentials and polled job status without the
deployment's aws_session_tags, so on a tag-gated trust policy batch
create and retrieve succeeded while cancel failed with AccessDenied.
Thread the tags through both calls and cover it with a regression test
that fakes STS behind a tag-gated trust policy.
* fix(proxy): keep a body litellm_session_id in SpendLogs under missing_session_id omit
Under general_settings.missing_session_id: omit, apply_missing_session_id_policy now
mirrors a client-supplied top-level litellm_session_id into metadata.session_id when the
client did not set one there, so SpendLogs.session_id and Langfuse agree with the session
callbacks already report through StandardLoggingPayload.session_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep client metadata.session_id ahead of body litellm_session_id on litellm_metadata routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop docstrings from the missing_session_id omit regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
One aliased key owned by a user with an email drives chat, queued chat,
messages, responses, embeddings, the Gemini passthrough, a batch file upload,
and a batch create against a live proxy. Each row must carry api_key equal to
the key's LiteLLM_VerificationToken.token and the alias in metadata, and
/spend/logs?api_key= and /user/daily/activity must report the key with its
alias and email. Health-check rows must keep the literal service-account key,
and the batch cost row for a completed marker batch must join the key that
created it. A re-hashed api_key (the v1.99.0 regression fixed by #39568 and
#39572) now fails the Buildkite e2e stage naming the write path
Resolves MAT-180
initialize_azure_sdk_client now falls back to litellm.constants.DEFAULT_MAX_RETRIES
when litellm_params carries no max_retries, so off-router Azure clients (files,
batches, fine-tuning, assistants, audio) honor the env var like OpenAI clients do.
Router paths already default max_retries to 0 and are unchanged.
Regression tests cover the default, explicit 0/5/None values, and the env var
reaching the SDK client in a fresh interpreter.
A streamed /v1/responses request with tool_choice {"type": "function", "name": ...}
that reaches a chat-completions-only deployment failed with HTTP 500 before the
first byte: the synthetic response.created and response.in_progress events copied
the chat-shaped tool_choice into ResponsesAPIResponse, whose ToolChoice type expects
the flat Responses API shape. The non-streamed path echoed "auto" regardless of the
request.
Both paths now normalize the request's tool_choice through the existing chat
transform and map it back to the Responses API vocabulary, validated by a
TypeAdapter(ToolChoice), so a named function is echoed as {"type": "function",
"name": ...} and a missing tool_choice is echoed as "auto".
Fixes#33689
Skipping the name write-back in either handler left every test green; a
guardrail that renames a tool call now has a regression test on both the
chat chunk path and the Anthropic SSE path
The bridge now asks for reasoning.encrypted_content whenever the provider's
Responses config lists include, independent of the client's thinking block,
and leaves it out for providers such as Perplexity that reject the param.
Bridge-tagged blocks are stripped on the chat adapter path too, so a mid
session model switch to Gemini or Bedrock no longer forwards them as real
signatures, a bare prefix counts as bridge-tagged, and non-mapping messages
pass through the strip untouched.
A stream cache hit on an entry stored with choices == [] indexed choices[0]
in the cached_response branch and failed with IndexError, so the streaming
converters' empty chunk had no working consumer. The branch now treats a
chunk without choices as empty and lets the wrapper close the stream with
its usual finish_reason stop chunk
Post-call guardrails on /v1/responses only treated function_call output
items as tool calls, so a custom_tool_call item (Codex's exec shell tool
on GPT-5.6 models) was never scanned or masked, non-streaming and
streaming alike. Both item types now flow through the shared
tool_call_dict_from_output_item helper, ended-stream delivery syncs the
custom_tool_call_input delta/done events and the item's input field, and
the completed-response scan key fingerprints both kinds of item.
Non-streaming Responses tool-call MASK rewrites were also never written
back to the output item even for function_call; they are now.
The two test methods, the policy_engine fixture, and the two inner
stubs in TestBackgroundResponseRetrievalGovernance now carry full
parameter and return annotations, closing the Greptile thread that
94f9230d13 left open.
The Databricks chat transformation only parsed reasoning out of FMAPI-style
reasoning content blocks, so external models behind Databricks AI Gateway that
return the OpenAI-style top-level reasoning_content string lost it, both in the
final message and in every streamed delta. Fall back to the shared OpenAI
reasoning helper when no reasoning block exists, and keep the delta's own
reasoning_content when streaming.
Operators can now set aws_session_tags on a Bedrock or SageMaker
deployment and every AssumeRole call carries them as STS session tags,
so trust policies gated on sts:TagSession admit the session and
CloudTrail and cost reports see the tags. Tags are validated up front,
sorted into the credential cache key so tag order does not fork
sessions, stripped from invoke and embedding request bodies, and
blocked from client request bodies like the other AWS identity params.
Based on #34073.
Co-authored-by: Satya Yedida <222552552+satya-rubrik@users.noreply.github.com>
Narrows the no-choices guard so a dict, string, or None still raises the APIError while an empty list passes through,
guards the non-stream Anthropic bridge against indexing an empty choices list, and repairs test_completion_missing_role,
whose raw-response mock was patched in as the create() callable itself so the handler only ever saw a MagicMock
A Responses API follow-up that replays reasoning.encrypted_content is pinned to the
deployment that minted it. Behind an auto-router the pre-routing hook rebinds the model
to the tier it picked before the candidate pool is built, so a turn that classifies into a
different tier never finds the origin and the affinity check raised its fail-fast 503,
whose text claims a cooldown that does not exist
When the deployment that minted the reasoning is not a member of the model group this turn
is routed to, strip the encrypted reasoning (keeping any readable summary, string or block
form) and dispatch to the routed group. Membership is tested by deployment id against the
candidate set the router itself resolved for the route (routing group, model_name, team,
and pattern alike), not by model-group name, so an alias, a provider-qualified spelling, a
team-public name, or a pattern route of the same group is not misread as a tier change.
An unknown origin (a removed deployment, or a forged/unauthenticated marker) is handled the
same as a cross-group one and its reasoning is stripped, so a real cross-group id and a
nonexistent id return the same response and cannot be used to enumerate deployment ids.
Unavailability within the origin's own group keeps the existing 429/503 fail-fast, so the
cooldown contract is unchanged
Resolves LIT-7195
Claude-Session: https://claude.ai/code/session_01KAumQbhzk6jdWWHFLA8Jar
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Nine cases assert the sts client is built with verify=True, but get_ssl_verify
reads SSL_CERT_FILE and SSL_VERIFY, so the argument depended on the ambient
environment. The published images set SSL_CERT_FILE, so the suite failed there
while passing in CI.
Fixes#40357
Each proxy worker flushed one Prisma upsert per active (date, category, route)
bucket every interval, so the Postgres primary saw workers x routes statements
per interval across the deployment. A flush now builds a single multi-row
INSERT ... ON CONFLICT DO UPDATE, and with use_redis_transaction_buffer on the
workers push snapshots to a Redis list that one lease-holding pod folds and
commits, so the whole deployment costs one statement per interval. The leader
keeps popping until the list is empty so a deployment wider than the dequeue
cap cannot build a backlog, and rows that fail both the commit and the Redis
re-queue fall back to the leader's own accumulator instead of being lost.
Resolves LIT-7371
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(router): support percentile-based TTFT routing
* fix(router): apply routing_strategy_args updates to the live selector
Runtime routing_strategy_args updates (config reload, update_settings)
only rebuilt the strategy selector when routing_strategy itself changed,
so a newly added ttft_percentile sat unused until the proxy restarted.
Also drops a comment that only restated the code it sat above.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* refactor(router): drop unreachable empty-samples guard in percentile latency
_percentile_latency is only called behind use_ttft, which already requires
a non-empty ttft sample list, so the early return was dead code and the one
line Codecov flagged as uncovered on this patch.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): cover the no-selector path of a routing_strategy_args update
simple-shuffle has no selector attribute to re-link, so the early return
guards a setattr with a None attribute name. Dropping the guard makes the
new test fail with "attribute name must be string, not 'NoneType'".
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(test): assert ValidationError on out-of-range ttft_percentile
pytest.raises(ValueError) tripped PT011 for being too broad. Pydantic
raises ValidationError for the gt/le constraint, so naming it satisfies
the rule and pins the assertion to the constraint under test.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(router): drop Final from a per-deployment loop variable
basedpyright rejects "A Final variable cannot be assigned within a loop",
which pushed reportGeneralTypeIssues one over its budget. selected_latency
is rebound each iteration, so it matches its unannotated neighbours in the
same loop.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): exempt _apply_updated_routing_strategy_args from the name scan
The scan only reads test files with "router" in the filename, so it cannot
see the update_settings tests in router_strategy/test_lowest_latency.py.
Calling the private helper directly would test structure rather than
behaviour, so it joins the existing entries ignored for the same reason.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* perf(proxy): pipeline spend counter increments into one redis call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply surviving spend increments before raising scope error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format spend counter helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): settle inner spend counter gathers and fall back per key on pipeline failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): suppress BLE001 on pipeline fallback catch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): invalidate all batched spend counters on pipeline failure
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The estimate looked the reported per-token rates up a second time, with the
provider this endpoint resolved rather than the one completion_cost infers.
The provider decides whether a token tier threshold is inclusive, so an
unrouted xai model sitting exactly on 200k billed at the tier rate and
reported the base rate, half of it.
completion_cost now hands back the rates its own lines were billed at, and
the endpoint reports those.
Claude-Session: https://claude.ai/code/session_01RLKy5DMi3XCBUJ37WzfNi1
