A shadow eval whose judge_model is one of the router's tier models, the router's
default model, or a reverse job's baseline_model was accepted with no warning. An
LLM judge scores its own output higher than a rival's, so that tier's win rate
measures the judge instead of the models, and the job's whole budget buys a result
that has to be thrown away.
start_shadow_eval now rejects it with a 400 naming the colliding arm.
`judge_target` is the single answer to "where does a call to this name go for this
caller, and what answers it", and the resolvability gate, the collision gate and
the judge dispatch all read it. It has three outcomes and no others: the router
serves the name, the SDK serves it, or nothing does. Splitting that question is
what every bug here came from, so `router_resolves_model` and `answering_models`
are gone rather than joined by a third.
Two spellings of one model are one identity. A name is compared by what would
answer it, resolved through every channel `get_model_list` composes and then put
in the provider-qualified form litellm itself uses, so a judge given as `gpt-4o`
collides with a tier deployment serving `openai/gpt-4o`, and a judge given as
`openai/gpt-4o` collides with a deployment configured as bare `gpt-4o`. Both ends
are normalised because an admin writes them at different times.
Answering is also per-caller. The shadow and judge calls carry the shadowed key's
`user_api_key_team_id`, which is what the router selects deployments with, so the
endpoint derives the job's teams once from the keys it already looks up and every
check runs under them, and the judge dispatch picks its arm under the same team.
A team's public model name resolves to nothing for everyone else and a team's own
deployment resolves for nobody else, so a check that omits the team answers for a
caller who does not exist. A collision under any one team fails the job, because
every key's verdicts land in the same win rates.
Three sites were separately re-deriving "the provider models this name resolves
to", with unexplained divergence in whether they fell back to the literal name.
`Router.resolved_litellm_models` is now the one owner; the routing-plugin
candidate list and the stream-options check both delegate to it, and
`_deployment_litellm_model` is gone.
The router's arms come from `strategy_router_dependencies`, the same enumeration
the health check reads. Only the roles that serve are arms: a classifier or
embedding model picks the tier and never produces a response anyone judges. A
semantic auto-router keeps its routes in an opaque config blob, so only its
default model is enumerable and the guard is incomplete there by design, able to
miss a collision but never to invent one
The two regenerated artifacts carry `presidio_analyze_chunk_size_bytes` from
alters the spec; the sync gate runs on any PR touching litellm/proxy, so this one
has to carry the base's drift to go green
The endpoint built messages=[{"role": "user", "content": prompt}], so a dry run
could not carry prior turns, the caller's system prompt, or the tool definitions
a request advertises. A real agentic turn reduced to its last sentence classified
as trivial, which is why a config sweep reported savings for every configuration.
Accept messages, system and tools, and forward them to the same pre-routing hook
untranslated, with the raw-body snapshot built by the serving path's own owner,
refresh_proxy_server_request_body_snapshot. Loose types are deliberate: the hook
reads whatever dialect the surface produced, so validating against one surface's
schema would reject the others.
prompt stays as the single-ask shorthand, normalized into one user turn inside the
request model so the handler carries no mode branch.
* fix(langfuse): warn and drop invalid LANGFUSE_TRACING_ENVIRONMENT instead of failing requests
* fix(langfuse): treat a dynamic environment equal to the raw deployment value as redundant
* fix(guardrails): add fail-open mode to CrowdStrike AIDR guardrail
Add a fail_on_error param (default True, preserving existing behaviour) to
the CrowdStrike AIDR guardrail, mirroring model_armor and generic_guardrail_api.
When fail_on_error=False the guard fails open only on server errors (5xx) and
connectivity failures, so the request proceeds unmodified. Caller-controlled
4xx responses and result.blocked policy blocks always fail closed. The
applied-guardrails header is recorded even on the fail-open path.
* fix(guardrails): fail open AIDR 4xx
* refactor(guardrails): isolate AIDR fail-open
* style(guardrails): format AIDR fail-open
* ci: satisfy unit workflow timeout invariant
* refactor(guardrails): accept AIDR mappings
* test(guardrails): inject AIDR HTTP client
* fix(guardrails): harden AIDR fail-open against delivered verdicts and record fail-open status
Reads the blocked verdict from the raw body before guard_output validation so schema drift or a changed verdict type cannot fail open past a delivered block. A transformed response that cannot be parsed fails closed so delivered redactions are never dropped. Fail-open runs record guardrail_status guardrail_failed_to_respond with timings instead of success. Restores the fail-open behavior tests dropped mid-PR and reverts the payload Mapping widening
* test(guardrails): cover fail_on_error wiring and fail-closed default for CrowdStrike AIDR
* chore(guardrails): annotate the transformed-drift detail payload for the LIT002 budget
---------
Co-authored-by: abrekhov <abrekhov@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Route records below WARNING to stdout (WARNING and above stay on stderr),
emit ANSI color codes only when both streams are a TTY (honoring NO_COLOR),
and parse JSON_LOGS strictly so JSON_LOGS=false no longer enables JSON logs.
* fix(presidio): chunk oversized text before /analyze so large content blocks do not fail
The Presidio PII guardrail sent each content block to the analyzer as a
single /analyze call with no size check. Analyzer deployments commonly cap
the request body (the reporting deployment rejects bodies over 1,000,000
bytes with HTTP 413), so large blocks failed closed, and analyzer latency
grew linearly with payload size.
analyze_text now splits texts larger than presidio_analyze_chunk_size_bytes
(default 500,000 UTF-8 bytes, configurable per guardrail) into overlapping
chunks, analyzes them concurrently, remaps each detection's start/end onto
the original text, and deduplicates detections from the overlap regions.
Anonymization, blocked-entity checks, score filtering, numbered-token
unmasking, telemetry, and the dashboard entity positions all consume the
remapped global offsets unchanged.
Resolves LIT-4785
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(presidio): review-round hardening for chunked analyze
- measure the chunk budget on the JSON-serialized text (non-ASCII escapes
expand beyond raw UTF-8, so a raw-byte budget could still exceed the
analyzer body limit)
- share the chunk fan-out semaphore per event loop and instance instead of
per call, so many oversized blocks cannot multiply concurrent analyzer
calls
- apply configured score thresholds and deny list per chunk BEFORE overlap
resolution, so a below-threshold span cannot displace a detection the
thresholds keep
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The /v1/messages bridge decided a Claude target could take `reasoning_effort` from
the model name, which says nothing about the params the provider in front of it
accepts. Snowflake serves Claude over the Anthropic dialect and declares `thinking`
alone, so `get_optional_params` raised `UnsupportedParamsError` before the request
reached the wire: every adaptive request carrying an effort tier turned a 200 into
a 400 for all seven of its Claude entries.
The tier is now offered only where the target declares the param, reading the same
`get_supported_openai_params` the sibling `_supports_prompt_cache_key` reads twelve
lines up. A target declaring neither carrier keeps its bare `thinking` block, which
is what this bridge sent before it carried a tier at all.
Without a resolved provider the tier stays behind rather than being offered blind.
Resolving one from the model's prefix instead would run an OAuth device flow for
github_copilot and chatgpt, blocking for minutes, and one of the two callers in that
position is a logging callback. The copilot case is pinned by a test.
* feat(ui): session-level cache observability in request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: guard cache_hit filter against non-string defaults in direct calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): drop redundant cache_hit field comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A complexity-router setting placed beside complexity_router_config, or inside a
tier entry's litellm_params, is read by nobody: the router loads its settings only
from litellm_params.complexity_router_config. It does not stay inert. The
alias-marker forwarding and the per-tier param spread carry every unrecognized key
onto the outbound request, and all_litellm_params only knows the outer names, so
the key reaches the provider as an unknown body field and every call through that
model group fails with an error naming an internal config key.
Guard the whole set, derived from ComplexityRouterConfig.model_fields so a field
added later is covered, and scoped to complexity-router deployments because the
names only mean this there (embedding_model is a legitimate flat param on an
s3_vectors vector store). Scope is read from the same merged field view the naming
check is judged on, so a router named only by its default model is in scope and a
field added to the required-field table is covered without another edit. The write
endpoints reject with a 400 naming the keys and where they belong, config.yaml
refuses to start for the same reason max_agentic_loops does, and a tier entry is
judged by the config model itself.
An already-stored deployment keeps loading, so an upgrade cannot take a running
gateway down over a row that was written before the gate existed.
* feat(alerting): add native Microsoft Teams alerting destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): preserve active destinations on MS Teams save and confirm health test delivery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): read persisted alerting destinations at MS Teams save time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
/v1/messages forwarded `thinking` verbatim for a Claude-family model and then returned,
carrying `output_config.effort` only when the model string started with a Bedrock prefix.
Every other bridged provider got a bare adaptive thinking block, so the caller's effort did
nothing: max and minimal produced byte-identical upstream bodies.
Send those targets the tier as `reasoning_effort`, which is the param they take. Bedrock keeps
taking `output_config`, since the two are not interchangeable there: an application inference
profile ARN resolves to no chat config, so `reasoning_effort` is dropped and the tier vanishes,
and a provider that rebuilds `output_config` from it overwrites a caller-set `thinking.display`
on the way. The tier stays a plain string, the summary already travelling inside the forwarded
`thinking` block. Adaptive with no tier, and budgeted thinking, both stay exactly as they were.
Kimi K3 accepts exactly low, high and max, defaults to max, and always thinks.
The map could not say that: medium and high have no supports_*_reasoning_effort
flag because every other reasoning model takes them, so the ten kimi-k3 entries
carried supports_reasoning alone and resolved to unknown. The dashboard then fell
back to a capability-blind level list that deliberately omits max, which is why a
kimi-k3 tier cannot be set to max thinking today.
Add reasoning_effort_levels, an array key in the shape the map already uses for
supported_endpoints and supported_modalities. Where present it is read first and
wins whole; every other entry keeps answering through the per-level flags,
unchanged. It is deliberately a different name from the computed
ModelGroupInfo.supported_reasoning_efforts, which stays derived from a group's
deployments and is never seeded from one deployment's model_info.
The levels are per entry rather than per model, because the deployments differ:
Moonshot, Together, Fireworks and Azure Foundry all forward the level unchanged
and get the model's own low/high/max, while Perplexity documents a six-value
enum it maps down internally and gets that. The /v1/messages degradation chain
consults the same declaration, so the level the map advertises is the level that
path forwards.
The router reload triggered by /model/new read the model table through
the read replica, so a lagging replica made the reload miss the just
committed row and fail the request with a 500 even though the write was
durable. Fixes#38556
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A non-ProxyException from the team, project or access-group lookup used to
escape the fallback loop and replace the provider's error. Treat it as a
denial and log it. Also drop the unrelated reformatting of test_router.py
and test_fallback_event_handlers.py so both diffs are additions only.
An MCP server behind an API gateway needs two credentials on one request: the
gateway's own token on a private header, and a separate bearer on Authorization
for the server behind it. Every arm that minted or held a token hardcoded
Authorization, and the conflict rule then dropped the operator's static
Authorization to make room, so the second credential never arrived.
ApiKeyConfig already modelled this as header_name plus value_prefix behind a
header() method. Extend that carrier to the four minted-token configs, have each
resolver arm ask its config which header to use instead of naming one, and drop
only the header the resolved credential is about to occupy.
Operators set it per server via upstream_token_header, plumbed through
config.yaml, the credentials blob, the management API and the admin form, on the
M2M, token-exchange, authorization-code and ID-JAG arms. It is non-secret so it
stays plaintext and round-trips on admin reads. Unset keeps today's behaviour.
Moving a credential off Authorization means it stops inheriting what Authorization
gets for free, so the slot now carries those protections itself. httpx drops
Authorization when a redirect crosses origin and keeps every other header, so a
custom slot is dropped by the client on the same condition, mirroring httpx's own
scheme/host/port rule with an agreement test that fails if the two ever diverge.
The v1 path also mirrors the v2 conflict rule, so an injected header cannot shadow
the credential the gateway resolved for that slot.
Which header a credential occupies, and what counts as being that header, was
answered independently in nine places by four hand-rolled comparisons. same_header,
has_header and without_header in litellm/types/mcp.py are now the one owner, shared
by both MCP stacks, and the client derives its slot once instead of three times.
The header name reaches egress verbatim, so the RFC 7230 grammar lives in one
place and is checked where servers are built: a bad value fails the config load
and the management API returns 400, rather than raising while a spec is built
and emptying the aggregate tool list for every other server. A blank means unset,
matching what the endpoint already accepts.
