Backport of BerriAI/litellm-private#5 (12981f93d3) onto stable/1.100.x, applied as
the PR's net diff so main-only intermediate refactors stay out.
The dashboard and lite CLI SSO specs under tests/e2e/ui/oidc are left out
because this line has no OIDC e2e harness to run them.
Adapted for stable/1.100.x: added Final to the test module's typing import; upstream's test file already imported it.
(cherry picked from commit 8a4fae0e17)
Adapted for stable/1.100.x: reflowed one generator to this line's ruff format (upstream reformatted it in a later style commit).
Signed-off-by: amasen02 <amasen02@users.noreply.github.com>
(cherry picked from commit daced81f20)
Adapted for stable/1.100.x: import-context conflict only (upstream's neighbouring custom_tools import is not on this line); the added and removed lines are identical to upstream.
(cherry picked from commit 025a3ca42f)
Backport of #41607 to stable/1.100.x.
Cherry-picked from deb9d8aedd (main).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Backport of #42048 to stable/1.100.x.
Cherry-picked from 7966f50c34 (main). The safeguards backport maps the dangerous-tool-use-2026-09-03 beta for Bedrock, which Claude Opus 4.5 on Bedrock Invoke rejects as an invalid beta flag, so the all-beta-headers Bedrock cases run on Claude Fable 5.1 as they do on main.
The picked TypedDict fields use read-only Sequence[Mapping[str, object]] annotations and the picked Vertex test carries a test-quality-ok marker, so stable/1.100.x's LIT001, LIT012 and TQ008 budgets hold. Static typing only, no runtime change.
Hand-ported to stable/1.100.x from 47b2479c94 on main (fix(bedrock): gate Invoke tool search on the model map's supports_tool_search flag), the one prerequisite the #42288 handler tests need; the rest of that commit stays on main.
Backport of #42288 to stable/1.100.x.
Cherry-picked from merge commit fc82f6e8fa (litellm_safeguards_bedrock_vertex_messages).
The line has no bedrock_mantle beta-header mapping and no Mantle /v1/messages route, so the Mantle mapping, its test file, and the bedrock_mantle test parameter are left out.
Backports the Converse part of fbc6fb56ae from main (PR #31884). The gate only matched
openai.gpt-5, so a GPT-6 model fell through to Anthropic's thinking block and Bedrock
rejected the first real turn after a Claude Code /model switch with 400 Unknown
parameter: 'thinking'. The Nova 2 tool_choice registry keys and the invoke json_mode
forwarding in that commit stay on main
Backport of #41870 to stable/1.100.x. Cherry-picked from a6e3a72ed8 (main) with -m 1.
converse_transformation.py conflicted because this line has no `import re` and no
_is_openai_gpt_reasoning_model helper next to the insertion point. The resolution adds
exactly the four hunks #41870 merged: the import, the 16-token constant,
_requires_min_max_tokens, and the clamped maxTokens assignment. The test file applied clean.
Move the BaseLLMException import into _map_error_event_exception so the
module no longer imports it at load time, clearing the module-level cyclic
import CodeQL flagged. The class is used only on the cold error path.
Replace the mutable list-append test collector with aiter/anext so the
regression tests read the stream immutably.
(cherry picked from commit c246372859)
Mid-stream error events on the streaming Responses API were all raised as
APIError, so a content_policy_violation event never matched the router's
content-policy fallback dispatch and the client got the raw error instead
of the fallback model's answer. Map each error event's code and status
through the existing exception_type mapping, matching the non-streaming
path, and unwrap the typed ContentPolicyViolationError and
ContextWindowExceededError so the router routes them to the configured
content_policy_fallbacks and context_window_fallbacks.
(cherry picked from commit 073d4fe2b0)
* fix(otel/v2): detach credential-routed tenant spans into their own trace
Multi-tenant OTel v2 routes a team or key's LLM-call span to that tenant's
own vendor account (New Relic, Arize, Langfuse, Weave) via dynamic OTLP
credential headers, while the request-root, auth, and db spans stay on the
operator's default backend. The span was still parented into the request
trace, so the tenant account received a child whose parent it never got,
and New Relic rendered it as a fragmented trace with a missing parent.
Detach a credential-routed span the same way a project-routed (Phoenix)
span already detaches: root a fresh trace in the tenant account and link
back to the request trace for correlation. Service-name routing keeps
parenting, since it only relabels service.name on the same operator
backend where the parent is present.
Guard the detach on the callback actually owning an OTLP exporter the
credentials can reach: a callback owning only a console or in_memory
exporter has nowhere to stamp them, so the span would export to the
default backend unchanged and detaching would orphan it on the very
backend that holds its parent. In that case warn once and keep the
default tracer.
* fix(otel/v2): derive tenant-route routability from resolved exporter transport
A denylist classified an owned exporter as routable whenever its kind was
not console/in_memory, so a typo'd or unavailable kind (e.g. "otlp",
"grcp") passed the check while _exporter_from_spec falls it back to a
header-ignoring console exporter. Detaching such a span would root a fresh
trace that only ever reaches the operator console, never the tenant
backend, orphaning it on both sides.
Route on a shared exporter_transport() predicate that resolves the kind the
same way _exporter_from_spec builds it (registered factories + otlp_http
aliases -> http, otlp_grpc aliases -> grpc, else headerless), so an
unresolvable kind is headerless and stays parented. Fixes the same latent
gap in project routability.
The driver retries rate-limit-shaped failures twice with a 65s backoff, but
RATE_LIMIT_SHAPED_RE only matches 429 / rate limit / too many requests /
throttled / CLI timeout. A saturated provider does not always say any of
those: litellm-e2e-pr build 182 turned a green cell red on
status 503: litellm.ServiceUnavailableError: BedrockException -
{"message":"Bedrock is unable to process your request."}
which matched nothing and so was never retried.
Add TRANSIENT_UPSTREAM_SHAPED_RE for 503, 529, "service unavailable",
"overloaded" and Bedrock's "unable to process your request", and have the
retry loop ask is_retryable_shaped (either shape) instead of
is_rate_limit_shaped.
Kept as a second pattern rather than widened into the first on purpose. The
conftest feeds RATE_LIMIT_SHAPED_RE into the rate-limit summary, which the
binary-search helper reads to decide whether to lower a provider's request
rate. A 503 says the provider is out of capacity, not that we are asking too
often, so folding it in there would keep ratcheting our rate down against a
condition our rate never caused.
Retry-loop logging now names which shape fired, so a saturated upstream is
distinguishable from a throttled one in the run output.
Verified against the literal failure text from build 182: is_retryable_shaped
is True while is_rate_limit_shaped stays False.
Note this covers CLI-driven rows only. HTTP-probe rows (tool_search,
count_tokens) have no retry layer at all, so the specific cell that failed in
build 182 is still unprotected -- that is a separate change to http_probe.
#34940 widened the mask-in-place safety guard so a Responses-API
`instructions` field (and a combined messages+input body) skips the
PII masking branch. With `on_flagged: "monitor"` that fell straight
through to "allow", so PII that used to be masked now reaches the
model unredacted.
Monitor means "don't block", not "don't redact". Recover the one shape
whose payload is still fully writable: mask it and write the redacted
instructions back into `data["instructions"]` directly, since
apply_redacted_messages_back has no path for that field and would
otherwise fold the instructions text into `data["input"]`.
The combined messages+input and multimodal shapes stay unmasked - both
are unsafe to write back, not merely unwritable - and now log an error
naming the reason instead of passing silently.
No block/allow decision changes: block and inject_system_message keep
the exact outcomes #34940 shipped.
Direct access on /model/info was read from the user record alone, so an
unrestricted user calling with a key limited to a few models saw every non-team
deployment, including ones the key gets a 403 on. Resolve the key's grant the
same way and intersect the two.
Resolving a grant now also expands access groups, which the key path needs and
the user path was missing.
Claude-Session: https://claude.ai/code/session_01XL7LBFEew4wi8gphVCDq6n
Model access group budgets shipped API-only, so the only way to give a group a
budget was a curl. Adds an Access Group Budgets tab under Models & Endpoints
that lists every group with the spend drawn against its shared pool, and a
modal to set, edit or clear the budget.
/access_group/list now carries each group's budget and spend inline, so the
table renders from one read instead of one follow-up request per row.
The first pass accepted any empty completion whose finish_reason was "length",
which also swallowed a fallback that produced nothing at all. Require the
response to have billed completion tokens as well, so empty content is
accepted only when the budget was demonstrably spent on non-visible reasoning.
Asserts on completion_tokens rather than reasoning_tokens because the latter
is provider-optional; with empty content and a refusal of null, consumed
completion tokens are reasoning by elimination, since visible text would be
content. Both counts are reported in the failure message.
Folds the three body accessors onto one _parsed helper instead of re-parsing
per call, and adds completion_tokens_of / reasoning_tokens_of alongside.
The LLM classifier's cost was recorded on the routing decision but never
reached any savings surface: per-request autorouter_savings stayed gross
and the session rollup recorded only the served request's spend, so
/auto_router/benchmarks overstated savings and understated routed spend.
Net the classifier cost into the savings figure at its one computation
owner and fold it into the rollup turn's spend, keeping
baseline_spend = spend + saved_spend. The response header's numeric
guard now shares the same reader.
Fixes#38816
