Each candidate may declare rules, short conditions tagged with the coverage
they imply. The judge names the rule matching its stated crux; the policy
derives the boundary from that operator-declared tag instead of trusting
the judge's own coverage opinion, and unlisted or unmatched picks step the
threshold as unmatched
Adapt the capability classifier to forecast a defined SUCCESS event with
reasoning ordered before the probability, add a capability_boundary bucket
that steps the qualification threshold per candidate, and cap classifier
payload values so large tool outputs and data URLs stop inflating
classification spend
* fix(proxy): keep passthrough logging metadata and model_info dicts when team callbacks are wired
Passing team callback vars into Logging(kwargs=...) makes get_litellm_params materialize a full litellm_params, where metadata and model_info default to None instead of being absent. Readers that resolve them as .get(key, {}).get(...) then raise, so any passthrough request from a team with logging callbacks 500s once a pre-call guardrail is on, and the router strategy loggers log a traceback per request.
* test(proxy): annotate the closure dicts the passthrough logging tests record into
The LLM Obs callback copied litellm's OpenAI-shaped objects into the span
verbatim, so every field Datadog names differently landed somewhere it does
not read: tool calls kept their nested `function` wrapper instead of DD's
name/arguments/tool_id, tool messages carried no result linking them to their
call, the request's tools were never sent, and prompt-cache counts sat inside
meta.metadata rather than the span metrics its cache dashboards chart.
One rule governs the message mapper: add the fields Datadog declares, and never
destroy content it did not understand. Content collapses to its text only when
it has text, so a content list carrying tool or image blocks rides along
unchanged, and absent messages map to an empty input rather than a fabricated
turn. Tool calls and results are read from both dialects, the OpenAI
`tool_calls` / `role: tool` shape and the Anthropic `tool_use` / `tool_result`
content blocks, so /v1/messages sessions gain tool linking they never had.
Cache counts come from the same owners the savings dashboard uses, so every
provider spelling resolves through one place rather than a second local guess.
The three cache metrics partition the input count: litellm's normalized prompt
total includes both cache categories, as the cost calculator's pricing helper
documents, so the non-cached residual subtracts reads AND writes. Counting a
primed prefix as ordinary input had inflated non-cached usage by exactly the
cache-write count on every priming request.
Correlating a result to its call reads ids and names structurally and parses no
arguments, so a tool call's arguments are decoded once per span rather than
once per pass, and arguments past a size bound ship as the raw string instead
of paying a decode that multiplies memory on hostile compact JSON.
The flat `output_tool_calls.*` metadata copies go away with this: they were a
second representation of a fact that now has its own field on the same span.
Bedrock rejects requests carrying cachePoint blocks for models whose entry in the cost map does not declare supports_prompt_caching (403 "You invoked an unsupported model or your request did not allow prompt caching"). Clients like Claude Code attach cache_control to every request, so any such model behind the gateway failed on every call. The new bedrock_model_accepts_cache_points predicate drops cachePoint emission for map-known non-caching models at all three emission funnels, keeps emitting for unmapped ids (application inference profile ARNs), and skips the gateway injection credit when the tool_config point is not placed.
aiohttp shields its DNS resolution task; when the connector closes it cancels
that child, so the request task sees CancelledError without ever being
cancelled itself. map_aiohttp_exceptions() only caught Exception, so the
BaseException skipped transport mapping, router retries and proxy error
handling, and /v1/responses answered 500 "No response returned".
Catch CancelledError in the mapper, re-raise when the current task is really
being cancelled (Task.cancelling() > 0), and otherwise map it to
httpx.ConnectError so the usual retry, fallback and error mapping apply.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Dockerfile, docker/Dockerfile.non_root and docker/Dockerfile.database uv sync stages never passed --extra bedrock-realtime, so aws-sdk-bedrock-runtime was absent from the image venv and Bedrock Nova Sonic /v1/realtime sessions failed with 'Missing aws_sdk_bedrock_runtime'. gateway/Dockerfile already had the extra (PR #34426).
Adds a static check over every uv sync in the proxy Dockerfiles and an image-level import probe that the image-scan workflow runs against the built root, non-root and gateway images.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GET /mcp-rest/tools/list without server_id returned only the tools of
the servers that answered and silently dropped any server whose listing
failed (for example an OAuth-protected server without credentials), so
clients could not tell a partial listing from a complete one.
The aggregate response now carries a server_outcomes map keyed by server
alias with the same classified outcome (ok/auth_required/forbidden/...)
that the MCP protocol path already puts in _meta. Healthy tools and the
HTTP 200 status are unchanged.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): resolve fallbacks against the tier a pre-routing hook selected
A complexity or auto router picks a tier behind the router group name, but
fallback lookup kept using kwargs["model"], which is still the router name. The
tier's configured chain never ran, so a provider failure on its first hop went
straight back to the client with "No fallback model group found for original
model_group=smart-router".
The hook assigns the selected model to a local only, and fallback resolution runs
on an outer kwargs dict that **kwargs already copied, so writing it there is not
visible. Record the selection in the metadata bucket instead, which is a nested
dict shared by reference across those copies and is how the router already
carries values back up, then key fallback lookup off it when present.
Applies to the generic, context-window, content-policy and weighted-failover
lookups. Reporting keeps using the router name, since that is what the caller
asked for.
Fixes#38832
* fix(router): annotate the recorded-selection helper with a read-only mapping
record_pre_routing_selection only reads the request kwargs, writing into the
nested metadata bucket it finds there, so Mapping states what it actually needs
and clears the LIT001 mutable-annotation budget without a suppression.
* test(router): assert the no-kwargs path leaks nothing
The tolerated-None case called the helper without checking anything, which the
test-quality gate counts as a test with no assertion. Assert that a fresh mapping
still reads back empty, so the case proves the call is a no-op rather than only
that it does not raise.
* fix(router): stop declaring loop-assigned locals Final in the selection helpers
Both helpers annotated a loop-assigned local as Final, which reassigns a Final on
every iteration and cost three basedpyright errors. Read the buckets through a
generator instead, so the write path iterates a for-target and the read path
resolves in one shot with next(), which also matches the functional style the
type-discipline rules ask for.
* style(router): apply ruff format to the selection helpers
* fix(router): derive the pre-routing tier fresh on every fallback hop
The metadata buckets also carry whatever the caller sent, so an inbound
pre_routing_selected_model let a client pick which fallback chain its
request fell into. A fallback hop also inherited the previous hop's tier,
so the second hop keyed its own failure off the tier that already failed
and never ran its own chain.
Clear the key at the top of async_function_with_fallbacks. Every hop
re-enters there, so only the hook that routed that hop can set it.
* fix(router): drop the cast at the fallback-hop clear call site
* feat(router): fall back on anthropic safeguard refusals on /v1/messages
---------
Co-authored-by: Priyansh Nandwana <nandwana.priyansh103@gmail.com>
Clearing the Team combobox in the Create Key modal left team_id set to an
empty string, so /key/generate treated the request as team key generation
and failed with a team-not-found error for non-admin members.
TeamDropdown now emits null on clear, and GenerateKeyRequest normalizes an
empty team_id to None so the request runs the personal key path.
The text-completion wire test set litellm.aclient_session, which the
test-quality gate (TQ005) flags as a process-wide global write. Pass an
AsyncOpenAI client through the router's client kwarg instead, so the test
owns its transport and needs no cache flush or global restore.
Claude-Session: https://claude.ai/code/session_01XKkTFa6g7Rmd6vtHL91GMn
* feat(alerting): slack alerts for per-user daily/monthly spend thresholds and spend anomaly detection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(alerting): use specific ValidationError matches in config rejection test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): tolerate mocked slack alerting args when scheduling user spend scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): reject non-finite values in user spend alert settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): quote the Claude Code apiKeyHelper for cmd.exe on Windows
lite up and lite login --config-claude wrote the helper command with
POSIX shlex quoting, so a backslashed Windows install path came out
wrapped in single quotes that cmd.exe and PowerShell take literally.
Quote every token with the cmd.exe rules already used for agent shims
when running on Windows, and keep the POSIX output unchanged elsewhere.
Resolves LIT-6627
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cli): split the Windows apiKeyHelper with cmd.exe and C runtime rules
The invocation test pulled tokens back out with a regex, which cannot see
the doubled quotes or the percent guard quote_for_cmd emits. Model the two
parsers that read the helper on Windows instead and check argv round
trips for backslashed, spaced, metacharacter, percent and quoted tokens
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Reading _target_order with .get left it in the request kwargs after selection, and only
nine provider boundaries stripped it. _atext_completion and _aadapter_completion spread
the raw kwargs, so an order-2 hop on /completions sent _target_order upstream, which
real providers reject as an unknown argument. Popping at selection strips it for every
path in one place; the PR's retry-keeping test already passed with pop because each retry
hands the callee its own kwargs copy.
Claude-Session: https://claude.ai/code/session_01XKkTFa6g7Rmd6vtHL91GMn
* fix(mcp): follow tools/list pagination from upstream servers
Adopts BerriAI/litellm#32244 by Jupiter363 onto litellm_internal_staging
with merge conflicts resolved
* fix(mcp): degrade buggy pagination to partial results and bound the preview walk
A repeated nextCursor now returns the tools collected so far instead of
discarding every page with a RuntimeError, an empty-string cursor is treated
as terminal, load_mcp_tools shares the same pagination walk instead of
returning only the first page, and the tools/list preview is bounded by the
listing timeout instead of only the per-request timeout times the page cap
* fix(mcp): annotate deliberate rebind for the preview timeout scope
* fix(mcp): bound the shared pagination walk with an overall listing deadline
The per-request session read timeout restarts on every page, so direct SDK
callers of list_tools and load_mcp_tools could run up to the page cap with
no overall bound. The walk now returns the tools collected so far when
max(MCP_CLIENT_TIMEOUT, MCP_TOOL_LISTING_TIMEOUT) expires
* fix(mcp): let a per-server timeout extend the pagination deadline
MCPClient carries a per-server timeout that can exceed the global default;
list_tools now passes max(self.timeout, MCP_TOOL_LISTING_TIMEOUT) into the
shared walk so a deliberately slow server is not silently truncated at the
global deadline
* fix(mcp): honor per-server timeouts in the preview deadline and test the walk sessionless
The preview deadline now extends with the created client's own timeout, and
the pagination walk's cap, repeated-cursor, and empty-cursor cases are tested
directly against the helper instead of through patched SDK internals
* fix(mcp): forward the preview request's per-server timeout to the temporary server model
The tools preview built its temporary MCPServer without the request's
timeout field, so the client factory always fell back to the global
default and a per-server timeout could never extend the preview's
listing deadline (or its per-request timeout).
* fix(proxy): default max_idle_connection_lifetime to 60s on DB URLs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): regenerate schema.d.ts for database_max_idle_connection_lifetime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep URL-pinned max_idle_connection_lifetime over config value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
