* fix(ui): keep litellm_credential_name from LiteLLM Params JSON when no credential is selected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drop null litellm_credential_name from AddModelPanel payload fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): validate JSON litellm_credential_name against accessible credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce proxy-admin-only credential attachment on model create/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): raise ProxyException for unauthorized credential attach and gate /model/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): fold credential-change detection into can_user_attach_credential to satisfy complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): decrypt stored credential name before unchanged-credential comparison
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover credential attach rejection on add_new_model and patch_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate proxy-global patches with test-quality suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This reverts merge commit 2b1bd20834 (#31125)
Two CircleCI jobs on the staging-to-main promotion went red the moment
that PR landed. proxy_multi_instance_tests boots two proxies against one
database, and both now race the same migration:
Error: P3018 A migration failed to apply
Database error code: 40P01, deadlock detected
Process 73 waits for ShareLock on virtual transaction 4/11;
blocked by process 75. Process 75 waits for ExclusiveLock on
advisory lock [16384,0,72707369,1]; blocked by process 73
Neither proxy comes up, so the job times out after 300s waiting on
localhost:4000. The same wait took 36.5s on the last green run
Timeline: #31125 merged at 18:46:14Z and the failing run started at
18:49:59Z. The merge commit is not an ancestor of the last green
revision (194a3cc) and is an ancestor of the first failing one
(01de2837)
The v2 resolver was meant to avoid exactly this class of contention, so
the deadlock looks like a bug in it rather than a reason to abandon it.
Putting the default back to v1 buys time to fix it without holding up
the release
* feat(guardrails): add Alice by ActiveFence guardrail
Adds `guardrail: alice` — policy-based guardrails for prompts and model
responses, evaluated against ActiveFence's Alice.
What makes this different from the other providers: Alice evaluates against
policies configured per *application*, and a proxy typically fronts several of
them, so the application cannot be a static config value. It is named on the
LiteLLM virtual key instead:
curl $PROXY/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"key_alias": "payments-bot",
"metadata": {"alice_app_id": "payments-bot"}}'
read via `CustomGuardrail._get_admin_metadata`, with `key_alias` as the
fallback. That helper is what makes it trustworthy: it reads whichever metadata
holder the proxy wrote the authenticated key's values into — which differs by
route — and the proxy strips caller-supplied `user_api_key_*` from both, so a
caller cannot point its own traffic at an application with laxer policies than
the one its key was issued for. A request whose key names no application is
refused rather than evaluated against a guess.
Implements `apply_guardrail` only, so pre_call, during_call, post_call and
streaming all come from UnifiedLLMGuardrails. Blocks with
GuardrailRaisedException; masks by substituting Alice's redacted text; a MASK
carrying no replacement blocks rather than passing the original through. A
verdict reporting `errors[]` is treated as a failure, not a pass — otherwise a
half-evaluated message would be allowed. `unreachable_fallback` (already on
LitellmParams) chooses fail-closed or fail-open on transport failure.
Config:
guardrails:
- guardrail_name: alice
litellm_params:
guardrail: alice
mode: [pre_call, post_call]
api_key: os.environ/ALICE_API_KEY
21 tests in tests/test_litellm/proxy/guardrails/guardrail_hooks/test_alice.py
cover registration, credential resolution, the app-id ladder including the
forged-metadata case, every verdict, and both unreachable policies.
No new LitellmParams field, so no schema.d.ts regeneration is needed.
* refactor(guardrails): post to Alice's LiteLLM endpoint and forward verbatim
Switches from `/v2/evaluate/message` — Alice's single-text endpoint — to
`/v2/evaluate/litellm`, which takes the hook's arguments as they arrive and
answers with a verdict.
That inverts where the work happens, and shrinks this plugin accordingly. It
now selects nothing and renames nothing: it posts `{input_type, inputs,
request_data}` and enforces `{verdict, categories, correlation_id, message,
replacements}`. Which parts of a conversation are worth evaluating, and how a
verdict is reached, are decided by Alice — so changing either is a change on
their side rather than a LiteLLM upgrade for every user.
The app-id resolution this plugin carried is gone with it. Alice reads the
application off the authenticated key's metadata itself, from the payload it is
handed, so the ladder here was duplicating a decision the far side already
makes. The security property is unchanged and still comes from the proxy
stripping caller-supplied `user_api_key_*` before a guardrail sees the request.
Masking is now positional — the far side chose which texts it was answering
for, so it says which by index. Only `texts` is written; a new
`structured_messages` object would make the chat translation layer skip the
`texts` write-back and silently drop the edits. A mask that lands nowhere
blocks rather than passing the original through.
`request_data` carries live Python objects (an OpenTelemetry span among them),
so `_json_safe` copies it into something serialisable by a mechanical rule
rather than a field list — a list drifts from what the far side needs, a rule
cannot. Serialising naively raises, and that error would read as "guardrail
unavailable" on every request.
26 tests, covering verbatim forwarding, each verdict, positional masking, the
`structured_messages` identity trap, both unreachable policies, and the
serialiser's handling of unserialisable values and cycles.
* fix(alice guardrail): satisfy lint and code-quality CI gates
- Bound _json_safe's recursion and register it in recursive_detector's
ignore list (it already caps depth and dedupes cycles by id, matching
the repo's established pattern for legitimate bounded recursion).
- Clear ruff-strict budget breaches: annotate __init__'s return type,
raise TypeError (not ValueError) for a bad response body, type
_json_safe's payload as object instead of Any, and file-scope-ignore
ANN401 for **kwargs (forwarding it as object broke the call into
CustomGuardrail.__init__, confirmed via basedpyright).
- Clear type-discipline budget breaches: suppress the construction/
annotation checks on one-shot HTTP payloads, the module-level
guardrail registries, and _json_safe's bounded accumulator; narrow
AliceVerdict's list fields to tuples and _evaluate's request_data to
Mapping[str, object] where nothing downstream mutates them.
* test(alice guardrail): assert the guardrail actually registers
The registration test called init_guardrails_v2 and asserted nothing, so it
passed whether or not the guardrail was ever registered — TQ001 in the
test-quality gate, and a fair catch: a test that cannot fail is not covering
the thing it names.
Now asserts exactly one AliceGuardrail lands in litellm.callbacks under the
configured name.
This surfaced only after the ruff-strict and type-discipline gates stopped
failing ahead of it; the lint job runs its gates in sequence, so an earlier
failure masks every later one.
* fix(alice guardrail): reach 100% patch coverage, drop the ActiveFence naming
Codecov flagged 10 uncovered lines, all of them error paths — which is where a
guardrail most needs covering, since each one decides whether traffic flows
unscreened.
Two of the ten turned out to be dead rather than untested, and are removed:
- `except GuardrailRaisedException: raise` in apply_guardrail. `_evaluate`
raises httpx errors, Timeout and TypeError, never that — so the clause could
never fire.
- the trailing `json.dumps` probe in `_json_safe`. Everything json.dumps
handles natively is caught by the isinstance branches above (a dict or list
subclass included), so anything reaching the bottom — bytes, datetime, an
OpenTelemetry span — cannot cross the wire regardless. It now says so and
returns None.
The rest are now tested: a timeout, 502/503/504 as unreachable, a 4xx as NOT
unreachable (a rejected credential is our misconfiguration, not an outage, and
must not fail open), a non-object response body, and a model whose model_dump
raises.
Also drops "by ActiveFence" throughout — the product is Alice — and points the
header at alice.io. `ui_friendly_name` is now "Alice", which is the key
guardrailLogoMap and the garden card look up, so all three moved together.
* fix(alice guardrail): strip caller credentials, widen unreachable detection, block partial MASK
Addresses PR review: request_data no longer forwards secret_fields.raw_headers or
the root api_key to Alice (the caller's Authorization token in the clear otherwise);
HTTP 500, malformed JSON, and a non-object body now route through the configured
unreachable_fallback instead of raising raw, so fail_open still fails open on those;
a MASK verdict with even one out-of-range replacement now blocks entirely instead of
silently letting the rest through unmasked. Also tightens request_data's type and
documents the known streaming-mask limitation on the class.
* fix(alice guardrail): strip credentials at any depth, stop filtering on texts
secret_fields/api_key/headers/provider_specific_header can appear nested
under proxy_server_request, metadata, litellm_metadata, and their
requester_metadata/body sub-paths in a real captured payload — a
top-level-only strip missed all of those. _json_safe now drops these keys
by name wherever they occur during serialization, so a new nesting path
can't reintroduce the leak.
apply_guardrail also stopped skipping the call whenever texts was empty,
even when tool_calls/images/structured_messages carried content — that
was the plugin making a selection decision Alice's design says belongs on
the far side. It now only skips when none of the selectable fields have
anything in them.
* fix(alice guardrail): route an undecodable response body through the fallback
`response.json()` raises UnicodeDecodeError when the body carries bytes that
are not valid UTF-8, and that escaped the except clause: UnicodeDecodeError is
a *sibling* of json.JSONDecodeError under ValueError, not a subclass of it, so
naming only JSONDecodeError left it uncaught. Both fallback modes surfaced a
raw decoding error instead of applying unreachable_fallback — which for a
fail_open deployment meant a hard failure where it had asked for an allow.
Named explicitly rather than widening to ValueError, so the clause still says
which three conditions it means. Tested under both policies.
The pure helpers express the same regression: the save-side drop must
leave the alias in place, and two registries must expand it to their
own ids. The full validate path is already covered by the
persists-verbatim test and the live e2e test.
Since PR #29128, key create/update/regenerate resolved every
object_permission.mcp_servers entry against the saving instance's
DB + config registry and persisted the resolved server ids. For
config-loaded servers the id is derived from a hash of the regional
URL, so in a shared-database multi-region deployment the rewrite baked
one region's ids into the row and every other region denied the key.
Grants written before v1.88.0 kept the raw alias and kept working,
which is why only newly provisioned keys broke.
Keep the validation and the stale-entry drop (the LIT-3278 fix), but
persist the caller's original identifiers for everything that resolves.
Read-time expand_permission_list already maps a name to each region's
local server id.
* fix(proxy): include litellm_model_table in GET /v2/team/list
GET /v2/team/list built its find_many queries without joining the
LiteLLM_ModelTable relation, so litellm_model_table (and the
model_aliases it carries) always read back as null there, same bug
class as GH #26312 which PR #33047 fixed on /team/info and /team/list
but never touched this endpoint.
* fix(test): assert observable output, not mock calls, in v2 team list test
The test-quality gate flagged the regression test for asserting on
find_many's call args instead of what the caller gets back. Rewritten
so the fake find_many only attaches litellm_model_table when its own
include kwarg asks for it, so the assertions are on the response.
* fix(proxy): drop invalid litellm_model_table include on deleted-team query
Greptile caught that LiteLLM_DeletedTeamTable has no litellm_model_table
relation in the Prisma schema, so passing that include on the deleted-team
find_many raised UnknownRelationalFieldError against a real database on
every GET /v2/team/list?status=deleted call. Confirmed live against
Postgres. Scope the fix to the active-team branch only, where the relation
exists; update the test to reflect that and assert the deleted branch no
longer requests it.
A non-admin could widen a read-only (info_routes) key to llm_api or full
access through the preset carve-out. Read-only keys now stay read-only
unless a proxy admin widens them; the other preset transitions, including
the LIT-4891 llm_api to full access switch, still work. Also converts the
transition tests to assert on a returned outcome so the no-403 cases
carry real assertions.
max_budget and reset_at already live on the matching budget_limits entry, so
repeating them (as budget_limit and reset_at) only invited confusion about which
copy is authoritative.
budget_limits now comes back exactly as stored on /key/info and /v2/key/info.
The per-window usage moves to a sibling budget_limits_usage field keyed by
budget_duration (current_spend, budget_limit, reset_at), mirroring
model_max_budget_usage, so the stored shape that /key/update accepts never
carries a computed field.
Pass window_duration to get_current_spend so /key/info re-checks a stale-low counter against the LiteLLM_BudgetWindowSpend row instead of aggregating LiteLLM_SpendLogs, and reuse _budget_limit_windows for the stored-column coercion. Drop the /v2/key/info batch cap (a new 422 for callers that work today) and the unrelated CI timeout bump and soft_budget test
* fix(proxy): wire team-level logging callbacks into passthrough endpoints
LIT-5152: passthrough routes now wire dynamic team-level callbacks
(success_callback, failure_callback, callback_vars) into Logging constructor,
mirroring the add_litellm_data_to_request behavior. Three hardening fixes:
1. Catch TypeError/AttributeError in _get_validated_callback_metadata when
team logging metadata has wrong shape (e.g., logging list instead of dict),
preventing HTTP 500 on passthrough routes with malformed config.
2. Wrap websocket passthrough logging initialization in try/except, since
the socket is already accepted at that point; errors after accept() yield
abrupt close (1006/1011) rather than clean HTTP error response.
3. Handle malformed deprecated callback_settings gracefully with try/except.
4. Wrap HTTP passthrough callback resolution in try/except to prevent 500 on
malformed team metadata (backward-compatibility fix).
Changes:
- pass_through_endpoints.py: wire dynamic callbacks in HTTP+WS paths, handle
malformed metadata gracefully with try/except fallbacks
- litellm_pre_call_utils.py: expand exception handling in validators
- test file: regression test for happy-path team callback wiring
* refactor(proxy): share passthrough team-callback resolution and cover its fail-open path
Collapse the duplicated callback wiring on the HTTP and websocket passthrough
paths into one helper that returns a frozen wiring value, log resolution
failures at error level so a broken logging config stays visible, and add
regression tests for malformed team metadata and an operational lookup failure.
Reverts the _get_validated_callback_metadata except widening: it changed
behavior for normal LLM routes, which is outside this ticket's scope.
* fix(proxy): keep passthrough alive when team callback vars hold env references
The deprecated team_metadata.callback_settings branch builds
TeamCallbackMetadata directly, skipping the AddTeamCallback validation
that strips os.environ/ references from the newer logging list. Stamping
those vars onto the Logging object made its constructor raise, so a team
on the legacy shape got HTTP 500 on every passthrough call. Validate the
resolved vars inside the fail-open boundary instead, so the request goes
through with dynamic callbacks skipped and the reason logged.
* fix(proxy): lint violations in team callback wiring helper
* style: format lint
* fix(auth): quiet malformed virtual key rejections to stdout
Reduce noisy invalid-api-key error logs by classifying malformed virtual
keys and routing their rejections to stdout as WARNING instead of stderr
as ERROR. Suppressible via LITELLM_LOG=ERROR or log_client_error_tracebacks=true.
Changes:
- auth_utils: is_invalid_virtual_key_error() classifier and marker functions
- auth_exception_handler: log invalid keys as WARNING to child logger before
identity seeding and callbacks, escalate non-401 transforms to ERROR
- user_api_key_auth: websocket early-raise WebSocketException(1008) to avoid
double-logging at HTTP layer
- _logging: child logger verbose_proxy_stdout_logger with no handler/level;
LevelRoutingStreamHandler routes its WARNING records to stdout; handler
setLevel in _turn_on_json() closes JSON config handler level leak
- test_auth_exception_handler: new test case verifying malformed-key logs
at WARNING with marker retention through transformations
Fixes LIT-5362
* fix(auth): classify malformed-key 401 by raise-site marker, not message text
Review round 1 (Greptile P2, veria Low):
- Move the marker attribute name to litellm/constants.py per the shared
sentinel convention
- Stamp the marker on the malformed-key 401 where it is raised and classify
only by it. Message text is caller-influenceable on other 401s (vector
store ids, organization ids are interpolated into their messages), so a
phrase match would let a request body demote an authorization failure to
the quiet log path
- Regression test: a 401 carrying the phrase but not the marker stays at
ERROR on stderr
Shadow eval jobs previously targeted only virtual keys, so deployments on
pure JWT auth (which present no key at all) could never sample their
traffic. Jobs now carry a typed (target_type, target_id) pair covering
keys, teams, and users; sampling matches the identity every request
resolves to at auth time, so team and user jobs cover JWT traffic with
no client changes.
Resolves LIT-6578
A non-admin switching an existing key's type between the safe preset
buckets (llm_api_routes, info_routes, and empty = full access) got a 403
from the allowed_routes admin gate, because /key/update, unlike
/key/generate and /key/regenerate, had no carve-out for preset-derived
values. Skip the gate only when both the incoming and the stored
allowed_routes consist entirely of safe presets, so clearing an
admin-set custom route restriction still requires proxy admin.
The batch start told the seed which LiteLLM_SpendLogs rows were its own, but
using it as a hard cutoff also dropped rows another pod had already persisted.
Those rows are only repaid by that pod's own increment, so if it died first the
window row stayed permanently under the recorded spend.
The seed now reads both sums in one scan and takes off this batch's own spend,
flooring at the pre-batch total for the case where its log rows have not landed
yet. Redis payloads keep an empty request_ids so a leader from before the field
was dropped can still merge what it pops during a rolling deploy.
Claude-Session: https://claude.ai/code/session_01QvQzYztinxj8ZuD5YxbVdL
precached_prompt_tokens is a subset of prompt_tokens (OpenAI cached_tokens
semantics), so map it to prompt_tokens_details.cached_tokens instead of
adding it on top of prompt/total. Emit stream usage from any final chunk
carrying it rather than only finish_reason stop, which dropped tokens for
function_call and length streams. Merge auth metadata into a new dict in
the gigachat router handler instead of mutating the shared parsed-body
cache in place.
- sync llm_passthrough_route: read and close an error-status streaming
response before mapping it, so upstream 4xx/5xx surface as the provider
error instead of httpx.ResponseNotRead
- AsyncPassthroughStreamingResponse: expose aiter_bytes() and carry
_hidden_params so the router attaches headers in place instead of
wrapping the stream in HiddenParamsAsyncIteratorWrapper, which 500'd
every streaming azure router-model passthrough request
- logging: swap the passthrough httpx result for the transformed
ModelResponse/EmbeddingResponse when firing success callbacks
- get_llm_provider: resolve gigachat from its api base and drop the dead
gigachat_models elif branch
- constants: register the gigachat api base in openai_compatible_endpoints
* feat(spend_tracking): persist router metadata in spend logs for internal router models
* test(spend_tracking): expect router_metadata key in exact-payload tests, type the routed-kwargs helper
The chat-to-Responses reverse transform kept only text blocks, so an image
input was dropped before the count went to OpenAI. A 256x256 image request
counted 13 tokens instead of 268.
/v1/responses/input_tokens returned 200 with a count for an empty
"input" ("" or []), while OpenAI returns a 400 missing_required_parameter.
The route also went through optimistic budget reservation, which is only
released by LLM success/failure callbacks that a token count never
reaches, so every call leaked a reservation until TTL expiry and could
429 real traffic. Both routes plus the /openai alias now join
/utils/token_counter in the reservation exemption set.
* fix(proxy): deliver budget alerts on webhook-only alerting and accept ALERTING_WEBHOOK_URL
ProxyLogging.budget_alerts forwarded to the alerting pipeline only when
'slack' was in general_settings.alerting, so alerting: ['webhook'] plus
WEBHOOK_URL silently never delivered a budget alert (the config
/health/services?service=webhook exists to test). Forward when 'webhook'
is present too; SlackAlerting.send_alert already fans out per channel.
Also accept a provider-neutral ALERTING_WEBHOOK_URL env fallback for the
Slack-format channel (any Slack-compatible receiver works), mark it as a
sensitive var, and de-brand the admin UI alerting copy.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): format settings.tsx with prettier
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for updated alerting description
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: retrigger checks after ALERTING_WEBHOOK_URL docs merged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
