get_llm_provider runs the OAuth device flow for github_copilot and chatgpt,
so calling it on the event loop before the executor dispatch let an
authenticated caller block the loop for the length of the polling window.
Adopt the declared provider via declared_authenticating_provider, matching
the metadata callers in utils.py, and only resolve for everything else.
* fix(mcp): follow tools/list pagination from upstream servers
Adopts BerriAI/litellm#32244 by Jupiter363 onto litellm_internal_staging
with merge conflicts resolved
* fix(mcp): degrade buggy pagination to partial results and bound the preview walk
A repeated nextCursor now returns the tools collected so far instead of
discarding every page with a RuntimeError, an empty-string cursor is treated
as terminal, load_mcp_tools shares the same pagination walk instead of
returning only the first page, and the tools/list preview is bounded by the
listing timeout instead of only the per-request timeout times the page cap
* fix(mcp): annotate deliberate rebind for the preview timeout scope
* fix(mcp): bound the shared pagination walk with an overall listing deadline
The per-request session read timeout restarts on every page, so direct SDK
callers of list_tools and load_mcp_tools could run up to the page cap with
no overall bound. The walk now returns the tools collected so far when
max(MCP_CLIENT_TIMEOUT, MCP_TOOL_LISTING_TIMEOUT) expires
* fix(mcp): let a per-server timeout extend the pagination deadline
MCPClient carries a per-server timeout that can exceed the global default;
list_tools now passes max(self.timeout, MCP_TOOL_LISTING_TIMEOUT) into the
shared walk so a deliberately slow server is not silently truncated at the
global deadline
* fix(mcp): honor per-server timeouts in the preview deadline and test the walk sessionless
The preview deadline now extends with the created client's own timeout, and
the pagination walk's cap, repeated-cursor, and empty-cursor cases are tested
directly against the helper instead of through patched SDK internals
* fix(mcp): forward the preview request's per-server timeout to the temporary server model
The tools preview built its temporary MCPServer without the request's
timeout field, so the client factory always fell back to the global
default and a per-server timeout could never extend the preview's
listing deadline (or its per-request timeout).
* fix(proxy): default max_idle_connection_lifetime to 60s on DB URLs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): regenerate schema.d.ts for database_max_idle_connection_lifetime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep URL-pinned max_idle_connection_lifetime over config value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep litellm_credential_name from LiteLLM Params JSON when no credential is selected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drop null litellm_credential_name from AddModelPanel payload fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): validate JSON litellm_credential_name against accessible credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce proxy-admin-only credential attachment on model create/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): raise ProxyException for unauthorized credential attach and gate /model/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): fold credential-change detection into can_user_attach_credential to satisfy complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): decrypt stored credential name before unchanged-credential comparison
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover credential attach rejection on add_new_model and patch_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate proxy-global patches with test-quality suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(s3): bound object keys and download filenames to s3 limits
Long OpenAI-compatible Responses API ids pushed the s3 object key past s3's
1024 UTF-8 byte cap, so the PUT failed with a 400 and the log record was
dropped. Keys that still fit are unchanged, byte for byte. An oversized one
now keeps a readable head of the file name and appends the sha256 of the full
name. A configured path/alias prefix that is long enough to overflow on its
own keeps whole leading path segments, so a prefix-scoped IAM policy or
lifecycle rule still matches, and ends in a short digest of the full
configured value so two operators do not land in the same folder.
The Content-Disposition filename carried the same unbounded id and hit s3's
2048 byte metadata-header cap, so the upload still failed with
MetadataTooLarge once the key was bounded. It is bounded the same way, head
plus digest, so two records downloaded from the console stay distinct files.
The full response id stays in the uploaded JSON payload.
* fix(s3): keep the configured prefix whole and spend the whole key budget
Shorten the response id first and only trim the operator's configured prefix
when the prefix itself is what does not fit, so prefix scoped IAM policies and
lifecycle rules keep matching. Trim by bytes rather than whole segments so the
longest possible string prefix survives, and route the audit log key through
the same shared builder.
* chore(s3): trim the comments and docstrings the review flagged
Keep the two external facts that are not visible from the code, the 1024 byte
object key cap and the 2048 byte metadata header cap, and drop the rest.
Filtering copied fields by name dropped companion billing rules like
web_search_billing_unit and the regional uplift multipliers, so grounding
and uplifts billed differently through the deployment entry. Copy the
backend's raw litellm.model_cost entry wholesale instead, which also
removes the synthesized-zero special case since the raw entry only holds
real values.
This reverts merge commit 2b1bd20834 (#31125)
Two CircleCI jobs on the staging-to-main promotion went red the moment
that PR landed. proxy_multi_instance_tests boots two proxies against one
database, and both now race the same migration:
Error: P3018 A migration failed to apply
Database error code: 40P01, deadlock detected
Process 73 waits for ShareLock on virtual transaction 4/11;
blocked by process 75. Process 75 waits for ExclusiveLock on
advisory lock [16384,0,72707369,1]; blocked by process 73
Neither proxy comes up, so the job times out after 300s waiting on
localhost:4000. The same wait took 36.5s on the last green run
Timeline: #31125 merged at 18:46:14Z and the failing run started at
18:49:59Z. The merge commit is not an ancestor of the last green
revision (194a3cc) and is an ancestor of the first failing one
(01de2837)
The v2 resolver was meant to avoid exactly this class of contention, so
the deadlock looks like a bug in it rather than a reason to abandon it.
Putting the default back to v1 buys time to fix it without holding up
the release
rag_query reuses the store resolved during authorization instead of a
second registry lookup, merges registry data store-wins so callers
cannot override a managed store's provider or credentials, and logs ids
instead of the merged config, which can carry resolved credentials.
aquery forwards only allowlisted retrieval_config keys to vector store
search, keeping caller-supplied connection overrides like api_base and
api_key away from the search call
_generic_api_call_with_fallbacks requires a model, so sync
vector_store_search and vector_store_create raised a TypeError whenever
the call carried no model. Model-less calls now go directly to the SDK
function, with the router injected for search, matching the async
wrapper's behavior
Copying only the flat token rates dropped threshold, tiered, service-tier,
cache, character, and per-second rates from peak-hour billing once cost
lookup switched to the deployment entry, and get_model_info synthesizes
zero flat rates for backends without one, which would have marked
tiered-only backends explicitly priced free. Copy every price-bearing
field instead, deep-copied, rejecting the synthesized zeros the way
_inherit_builtin_tiered_output_rate already does.
* feat(guardrails): add Alice by ActiveFence guardrail
Adds `guardrail: alice` — policy-based guardrails for prompts and model
responses, evaluated against ActiveFence's Alice.
What makes this different from the other providers: Alice evaluates against
policies configured per *application*, and a proxy typically fronts several of
them, so the application cannot be a static config value. It is named on the
LiteLLM virtual key instead:
curl $PROXY/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"key_alias": "payments-bot",
"metadata": {"alice_app_id": "payments-bot"}}'
read via `CustomGuardrail._get_admin_metadata`, with `key_alias` as the
fallback. That helper is what makes it trustworthy: it reads whichever metadata
holder the proxy wrote the authenticated key's values into — which differs by
route — and the proxy strips caller-supplied `user_api_key_*` from both, so a
caller cannot point its own traffic at an application with laxer policies than
the one its key was issued for. A request whose key names no application is
refused rather than evaluated against a guess.
Implements `apply_guardrail` only, so pre_call, during_call, post_call and
streaming all come from UnifiedLLMGuardrails. Blocks with
GuardrailRaisedException; masks by substituting Alice's redacted text; a MASK
carrying no replacement blocks rather than passing the original through. A
verdict reporting `errors[]` is treated as a failure, not a pass — otherwise a
half-evaluated message would be allowed. `unreachable_fallback` (already on
LitellmParams) chooses fail-closed or fail-open on transport failure.
Config:
guardrails:
- guardrail_name: alice
litellm_params:
guardrail: alice
mode: [pre_call, post_call]
api_key: os.environ/ALICE_API_KEY
21 tests in tests/test_litellm/proxy/guardrails/guardrail_hooks/test_alice.py
cover registration, credential resolution, the app-id ladder including the
forged-metadata case, every verdict, and both unreachable policies.
No new LitellmParams field, so no schema.d.ts regeneration is needed.
* refactor(guardrails): post to Alice's LiteLLM endpoint and forward verbatim
Switches from `/v2/evaluate/message` — Alice's single-text endpoint — to
`/v2/evaluate/litellm`, which takes the hook's arguments as they arrive and
answers with a verdict.
That inverts where the work happens, and shrinks this plugin accordingly. It
now selects nothing and renames nothing: it posts `{input_type, inputs,
request_data}` and enforces `{verdict, categories, correlation_id, message,
replacements}`. Which parts of a conversation are worth evaluating, and how a
verdict is reached, are decided by Alice — so changing either is a change on
their side rather than a LiteLLM upgrade for every user.
The app-id resolution this plugin carried is gone with it. Alice reads the
application off the authenticated key's metadata itself, from the payload it is
handed, so the ladder here was duplicating a decision the far side already
makes. The security property is unchanged and still comes from the proxy
stripping caller-supplied `user_api_key_*` before a guardrail sees the request.
Masking is now positional — the far side chose which texts it was answering
for, so it says which by index. Only `texts` is written; a new
`structured_messages` object would make the chat translation layer skip the
`texts` write-back and silently drop the edits. A mask that lands nowhere
blocks rather than passing the original through.
`request_data` carries live Python objects (an OpenTelemetry span among them),
so `_json_safe` copies it into something serialisable by a mechanical rule
rather than a field list — a list drifts from what the far side needs, a rule
cannot. Serialising naively raises, and that error would read as "guardrail
unavailable" on every request.
26 tests, covering verbatim forwarding, each verdict, positional masking, the
`structured_messages` identity trap, both unreachable policies, and the
serialiser's handling of unserialisable values and cycles.
* fix(alice guardrail): satisfy lint and code-quality CI gates
- Bound _json_safe's recursion and register it in recursive_detector's
ignore list (it already caps depth and dedupes cycles by id, matching
the repo's established pattern for legitimate bounded recursion).
- Clear ruff-strict budget breaches: annotate __init__'s return type,
raise TypeError (not ValueError) for a bad response body, type
_json_safe's payload as object instead of Any, and file-scope-ignore
ANN401 for **kwargs (forwarding it as object broke the call into
CustomGuardrail.__init__, confirmed via basedpyright).
- Clear type-discipline budget breaches: suppress the construction/
annotation checks on one-shot HTTP payloads, the module-level
guardrail registries, and _json_safe's bounded accumulator; narrow
AliceVerdict's list fields to tuples and _evaluate's request_data to
Mapping[str, object] where nothing downstream mutates them.
* test(alice guardrail): assert the guardrail actually registers
The registration test called init_guardrails_v2 and asserted nothing, so it
passed whether or not the guardrail was ever registered — TQ001 in the
test-quality gate, and a fair catch: a test that cannot fail is not covering
the thing it names.
Now asserts exactly one AliceGuardrail lands in litellm.callbacks under the
configured name.
This surfaced only after the ruff-strict and type-discipline gates stopped
failing ahead of it; the lint job runs its gates in sequence, so an earlier
failure masks every later one.
* fix(alice guardrail): reach 100% patch coverage, drop the ActiveFence naming
Codecov flagged 10 uncovered lines, all of them error paths — which is where a
guardrail most needs covering, since each one decides whether traffic flows
unscreened.
Two of the ten turned out to be dead rather than untested, and are removed:
- `except GuardrailRaisedException: raise` in apply_guardrail. `_evaluate`
raises httpx errors, Timeout and TypeError, never that — so the clause could
never fire.
- the trailing `json.dumps` probe in `_json_safe`. Everything json.dumps
handles natively is caught by the isinstance branches above (a dict or list
subclass included), so anything reaching the bottom — bytes, datetime, an
OpenTelemetry span — cannot cross the wire regardless. It now says so and
returns None.
The rest are now tested: a timeout, 502/503/504 as unreachable, a 4xx as NOT
unreachable (a rejected credential is our misconfiguration, not an outage, and
must not fail open), a non-object response body, and a model whose model_dump
raises.
Also drops "by ActiveFence" throughout — the product is Alice — and points the
header at alice.io. `ui_friendly_name` is now "Alice", which is the key
guardrailLogoMap and the garden card look up, so all three moved together.
* fix(alice guardrail): strip caller credentials, widen unreachable detection, block partial MASK
Addresses PR review: request_data no longer forwards secret_fields.raw_headers or
the root api_key to Alice (the caller's Authorization token in the clear otherwise);
HTTP 500, malformed JSON, and a non-object body now route through the configured
unreachable_fallback instead of raising raw, so fail_open still fails open on those;
a MASK verdict with even one out-of-range replacement now blocks entirely instead of
silently letting the rest through unmasked. Also tightens request_data's type and
documents the known streaming-mask limitation on the class.
* fix(alice guardrail): strip credentials at any depth, stop filtering on texts
secret_fields/api_key/headers/provider_specific_header can appear nested
under proxy_server_request, metadata, litellm_metadata, and their
requester_metadata/body sub-paths in a real captured payload — a
top-level-only strip missed all of those. _json_safe now drops these keys
by name wherever they occur during serialization, so a new nesting path
can't reintroduce the leak.
apply_guardrail also stopped skipping the call whenever texts was empty,
even when tool_calls/images/structured_messages carried content — that
was the plugin making a selection decision Alice's design says belongs on
the far side. It now only skips when none of the selectable fields have
anything in them.
* fix(alice guardrail): route an undecodable response body through the fallback
`response.json()` raises UnicodeDecodeError when the body carries bytes that
are not valid UTF-8, and that escaped the except clause: UnicodeDecodeError is
a *sibling* of json.JSONDecodeError under ValueError, not a subclass of it, so
naming only JSONDecodeError left it uncaught. Both fallback modes surfaced a
raw decoding error instead of applying unreachable_fallback — which for a
fail_open deployment meant a hard failure where it had asked for an allow.
Named explicitly rather than widening to ValueError, so the clause still says
which three conditions it means. Tested under both policies.
A bare string or list under off_peak_pricing in YAML passed the truthy
guard and crashed _is_off_peak with AttributeError, breaking cost
calculation for that deployment. Malformed pieces of the block are
documented to not match rather than error, so guard the block itself the
same way and bill standard rates.
Since the Rust core handoff in #37241, BedrockConverseLLM.completion read
access_key, secret_key and token off the boto3 credentials before asking
the Rust gate whether it wanted the call. On a deployment that only sets
AWS_BEARER_TOKEN_BEDROCK boto3 resolves no credentials, so every Converse
call through /v1/chat/completions and /v1/responses failed with
"'NoneType' object has no attribute 'access_key'", with or without the
Rust opt-in
Bearer auth resolves no SigV4 principal at all, and both the Python and
the Rust path read the bearer token themselves, so only hand the
principal keys down when boto3 actually resolved one
get_request_headers now accepts credentials=None and raises botocore's
NoCredentialsError when neither a bearer token nor a principal exists
instead of handing SigV4Auth a None
Fixes#38579
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BFDpYC45u9p8eKd4aBATnS
The router_code_coverage gate only counts calls made from test files with
router in the filename, so the helper needs direct unit tests beside the
other inheritance helpers in test_router_model_cost_isolation.py: fills
missing base rates from the builtin entry, leaves explicit rates alone,
and no-ops without a block or for an unmapped backend model.
Cost lookup selects the deployment-scoped cost map entry only when custom
pricing is detected and the entry carries a base pricing field. A deployment
whose model_info held nothing but off_peak_pricing failed both conditions, so
its schedule was silently ignored and every request billed at the shared
backend rate.
use_custom_pricing_for_model now also treats deployment-scoped pricing fields
in the metadata model_info as custom pricing, and the router inherits the
backend model's built-in base token rates onto such an entry at registration,
which also lets cache pricing inheritance apply. Regression tests cover the
registration, the detection, and the costed request end to end.
