* feat(router): support percentile-based TTFT routing
* fix(router): apply routing_strategy_args updates to the live selector
Runtime routing_strategy_args updates (config reload, update_settings)
only rebuilt the strategy selector when routing_strategy itself changed,
so a newly added ttft_percentile sat unused until the proxy restarted.
Also drops a comment that only restated the code it sat above.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* refactor(router): drop unreachable empty-samples guard in percentile latency
_percentile_latency is only called behind use_ttft, which already requires
a non-empty ttft sample list, so the early return was dead code and the one
line Codecov flagged as uncovered on this patch.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): cover the no-selector path of a routing_strategy_args update
simple-shuffle has no selector attribute to re-link, so the early return
guards a setattr with a None attribute name. Dropping the guard makes the
new test fail with "attribute name must be string, not 'NoneType'".
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(test): assert ValidationError on out-of-range ttft_percentile
pytest.raises(ValueError) tripped PT011 for being too broad. Pydantic
raises ValidationError for the gt/le constraint, so naming it satisfies
the rule and pins the assertion to the constraint under test.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* fix(router): drop Final from a per-deployment loop variable
basedpyright rejects "A Final variable cannot be assigned within a loop",
which pushed reportGeneralTypeIssues one over its budget. selected_latency
is rebound each iteration, so it matches its unannotated neighbours in the
same loop.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
* test(router): exempt _apply_updated_routing_strategy_args from the name scan
The scan only reads test files with "router" in the filename, so it cannot
see the update_settings tests in router_strategy/test_lowest_latency.py.
Calling the private helper directly would test structure rather than
behaviour, so it joins the existing entries ignored for the same reason.
Claude-Session: https://claude.ai/code/session_01PmqjhFYcUh6vA72d8W9gdB
`include:` or `exclude:` written as `${{ ... }}` read back as a string, and
the sweep treated that as the directive being absent, so it expanded every
combination GitHub would have dropped. A job whose `name:` holds no matrix
value then looked like it repeated one name across combinations that never
run. An absent directive still means no rows; anything that is not a list
of rows now joins the names left out of the comparison
A `name:` whose only leftover expressions read a `github.` property other
than `github.job` is filled in identically for every job of the run that
publishes it, so two jobs of one workflow carrying it land on the same
check run. Those names now compare against the other jobs of their own
file instead of sitting in the blind-spot bucket. They stay out of the
comparison across files, where two workflows can run on different events
An expression at `jobs.<id>.strategy` is legal on GitHub, but the model
required a mapping there, so a workflow using one made the whole file
unreadable and turned code-quality red. That job's names are now a blind
spot like any other name the sweep cannot work out offline.
A matrix whose `name:` holds no matrix value publishes that one name once
per combination, which leaves a required context just as ambiguous as two
jobs sharing a name, so it now reports instead of deduping.
A file that does not parse as one YAML document is reported the way the
module already promised, rather than escaping as a traceback.
Three ways the sweep could fail a workflow GitHub would publish fine.
`github.workflow` and `github.job` were counted as fixed for the whole run, so
two jobs naming themselves after the workflow they sit in were reported as a
collision. `runner` and `vars` were wrong the same way. Drop the exception
entirely: a name still holding an expression is one GitHub resolves per job, so
it is nothing to compare, which is what the rest of the module already does.
`format()` was resolved with Python's semantics, so an attribute lookup crashed
the script and a width specifier padded a name GitHub never pads. Fill `{0}`
holes and escaped braces, and treat anything richer as unresolved.
A matrix `include` or `exclude` row holding a value that is not a scalar lost
that key and became an empty row, which excludes every combination. Report the
row instead of quietly reshaping the matrix around it.
A job name holding an expression the sweep could not resolve was compared as
if it were the published name. Two jobs whose names differ per matrix value or
per caller input were reported as a collision, and a matrix that was itself an
expression collapsed onto the bare job id and did the same.
Model what a job publishes as known names beside the reasons the rest stay
unknown. Anything the sweep cannot work out contributes no name and is
reported as a note instead of guessed at. An expression over contexts that are
fixed for the whole run still compares, so two jobs sharing one of those are
still caught.
The collision sweep read a job's name as its `name:` or bare job id, which is
wrong for a matrix job that sets no name: GitHub publishes `build (3.12)`, one
per combination. That missed real duplicates and invented ones that don't exist.
It also crossed every matrix value while ignoring `exclude`, so it checked
combinations no job ever runs.
Four smaller gaps went with it. Boolean matrix values reached a name as `True`
rather than `true`. A `format()` whose arguments cannot fill its placeholders
raised straight out of the script instead of leaving the name unresolved. A job
calling a reusable workflow only ever chained one level, and a call outside the
repo fell back to the caller's own name, which GitHub never posts. A job whose
`name:` was not a string failed validation and silently dropped every job in
that file, so the sweep now renders any scalar and reports a file it cannot read
instead of skipping it.
The checker raised a custom exception and caught it two lines down in the
same module, which is the throw-then-catch the repo's coding guide rules
out. `main` now prints the same message and returns the exit code, so the
collision list stays a value the whole way out
The guard read each matrix key's values independently and crossed them, so
a job name reading two keys off one include row published pairs no job ever
runs, which could fail a valid workflow on a required check
It now builds the combinations GitHub builds: the listed keys crossed, each
include row folded into the combinations it overwrites nothing in, and a row
that fits nowhere standing on its own
The guard only substituted a bare `${{ matrix.key }}`, so any name built from a
larger expression stayed in the string as its own template. `_test-unit-base.yml`
names its job with a ternary over `format()`, which meant every shard published
an opaque name and 23 of the 33 required contexts, all of them `<shard> / Run
tests`, were invisible to the very check meant to protect them.
Job names are now evaluated per matrix combination over the pieces a name can
hold: string literals, `matrix.<key>`, `format()`, `==` and `!=`, and the
`<cond> && <a> || <b>` idiom. All 33 required contexts now resolve, and nothing
in the repo leaves an expression unresolved. An expression the evaluator does not
understand still falls back to its verbatim template, so two jobs sharing one
stays a collision.
The test also drops its `sys.path.insert`, which the test-quality budget counts
under TQ003; pytest already puts the file's own directory on the path.
A ruleset's required status check names a check run and GitHub matches it by
that name alone, so two jobs publishing the same name leave the gate unable to
say which job proved it. The new code-quality check reads every workflow,
expands matrix values and local reusable-workflow calls the way Actions does,
and fails when one name has more than one job behind it.
The selector picked up two suites that can never pass in this stack, so
editing either one turned the check permanently red: the presidio masking
suite calls pytest.fail without an analyzer and anonymizer that up.sh
never starts, and the pipecat audio suite skips itself at import time
unless the NLTK punkt_tab data is present, which nothing installs.
tests/e2e/coverage_registry/test_collector.py had the same problem for a
different reason. Its nested pytest.main autoloads pytest-retry from the
ci group the workflow installs and dies with "INTERNALERROR: no option
named 'filtered_exceptions'", so the collect-only pass now disables that
plugin. The plugin's entry point is pytest-retry, not retry, so the same
one-word fix lands on mutmut's pytest_add_cli_args, where "-p no:retry"
was disabling nothing.
Two smaller holes in the harness: a canary argument the shell never
expanded used to select nothing and let the gate pass green, and a secret
that cannot be represented in both bash and dotenv was rejected without
naming the key.
A harness-only change (proxy_client.py, conftest.py, pytest.ini, the gateway
config, .github/e2e-stack, or the workflow) selected nothing, so the stack was
never exercised by the change that touched it. select_tests.py keeps the
changed-file rule and adds the access_control suite whenever a harness file
changes. The run step now reports the pytest exit code before the evidence
check, prints pytest's summary line per pass so the rerun count is visible,
and assert_tests_ran.py names each failed or errored test as classname::name
A one-character value in the provider secret bundle was masked too, which
turned every 1 in the run log into ***, including the pass numbers and the
gateway addresses, so the only public diagnostics were unreadable
* fix(agents): redact secret litellm_params fields from all /v1/agents responses
Secret-bearing litellm_params fields (aws_secret_access_key, api_key, and
similar) are now write-only: list, get, create, update, and patch
responses always replace them with a fixed marker, regardless of caller
role. Editing an agent no longer requires resending a real credential --
an update that omits a sensitive field, or echoes the marker back,
preserves the stored value; a real value still rotates it.
* fix(agents): redact secrets nested inside dicts/lists in litellm_params too
Greptile found that a secret nested one level down under a
non-sensitively-named key, or inside a list of per-provider configs, was
neither redacted on read nor restored symmetrically on write (the marker
string could get persisted as the real value). Recurse into lists on the
read side, and mirror that recursion on the write side so restoration
isn't limited to top-level keys. Also fixes a regression the redact
rewrite introduced (a plain string leaf like a model name was being
misinterpreted as a JSON blob and redacted), and suppresses 3 new
test-quality-gate findings on an established repo-wide mocking pattern
this PR's new tests also use.
* fix(agents): guard list-position credential restore against misassignment
Two more real gaps Greptile/veria found in the recursive redact/restore
mechanism, verified directly against the exact reported shape
(litellm_params.model_list, each entry carrying its own nested
litellm_params.api_key/aws_secret_access_key) before fixing:
- Positional restoration inside a list could attach one entry's stored
credential to a different entry if the list were reordered or resized
between GET and PUT/PATCH. Restoration by index now only fires when the
incoming and existing entries match on every non-secret field; otherwise
the caller's own value is used (never a guessed cross-entry secret).
- A subtree collapsed to the flat REDACTED_BY_LITELM marker by the
read-side recursion depth cap couldn't be recovered on write (the marker
string itself would get persisted). Restore now recognizes that shape and
recovers the whole existing subtree.
Both covered by regression tests mirroring the exact model_list shape
reported, mutation-verified.
* fix(agents): simplify list-entry credential restore to positional matching
The content-match guard from the previous commit fixed one Greptile
finding (cross-entry misassignment on reorder) but introduced a worse one:
it also rejected restoration whenever an entry's own non-secret fields
changed, which is the common case (rename a model_list entry while
leaving its own secret masked) -- silently dropping the stored credential
on an ordinary edit.
There is no stable per-element identity in a plain dict[str, object]
schema, so no rule can satisfy both 'restore whenever the entry itself
only had its secret masked' and 'never restore across a reorder' at once.
Positional correspondence is what every other part of this restore (and
the endpoints' full-replace-on-PUT semantics) already assumes, so drop
the content-match gate and rely on it here too: this fixes the common
case correctly and accepts cross-entry misassignment on a simultaneous
reorder-plus-masked-echo as a known, narrow, documented limitation (not a
leak between different agents or tenants, since it only reshuffles one
agent's own stored values). Tests updated to pin the accepted trade-off
explicitly rather than asserting it away, and to cover the previously
broken ordinary-edit case.
* feat(guardrails): add Alice by ActiveFence guardrail
Adds `guardrail: alice` — policy-based guardrails for prompts and model
responses, evaluated against ActiveFence's Alice.
What makes this different from the other providers: Alice evaluates against
policies configured per *application*, and a proxy typically fronts several of
them, so the application cannot be a static config value. It is named on the
LiteLLM virtual key instead:
curl $PROXY/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"key_alias": "payments-bot",
"metadata": {"alice_app_id": "payments-bot"}}'
read via `CustomGuardrail._get_admin_metadata`, with `key_alias` as the
fallback. That helper is what makes it trustworthy: it reads whichever metadata
holder the proxy wrote the authenticated key's values into — which differs by
route — and the proxy strips caller-supplied `user_api_key_*` from both, so a
caller cannot point its own traffic at an application with laxer policies than
the one its key was issued for. A request whose key names no application is
refused rather than evaluated against a guess.
Implements `apply_guardrail` only, so pre_call, during_call, post_call and
streaming all come from UnifiedLLMGuardrails. Blocks with
GuardrailRaisedException; masks by substituting Alice's redacted text; a MASK
carrying no replacement blocks rather than passing the original through. A
verdict reporting `errors[]` is treated as a failure, not a pass — otherwise a
half-evaluated message would be allowed. `unreachable_fallback` (already on
LitellmParams) chooses fail-closed or fail-open on transport failure.
Config:
guardrails:
- guardrail_name: alice
litellm_params:
guardrail: alice
mode: [pre_call, post_call]
api_key: os.environ/ALICE_API_KEY
21 tests in tests/test_litellm/proxy/guardrails/guardrail_hooks/test_alice.py
cover registration, credential resolution, the app-id ladder including the
forged-metadata case, every verdict, and both unreachable policies.
No new LitellmParams field, so no schema.d.ts regeneration is needed.
* refactor(guardrails): post to Alice's LiteLLM endpoint and forward verbatim
Switches from `/v2/evaluate/message` — Alice's single-text endpoint — to
`/v2/evaluate/litellm`, which takes the hook's arguments as they arrive and
answers with a verdict.
That inverts where the work happens, and shrinks this plugin accordingly. It
now selects nothing and renames nothing: it posts `{input_type, inputs,
request_data}` and enforces `{verdict, categories, correlation_id, message,
replacements}`. Which parts of a conversation are worth evaluating, and how a
verdict is reached, are decided by Alice — so changing either is a change on
their side rather than a LiteLLM upgrade for every user.
The app-id resolution this plugin carried is gone with it. Alice reads the
application off the authenticated key's metadata itself, from the payload it is
handed, so the ladder here was duplicating a decision the far side already
makes. The security property is unchanged and still comes from the proxy
stripping caller-supplied `user_api_key_*` before a guardrail sees the request.
Masking is now positional — the far side chose which texts it was answering
for, so it says which by index. Only `texts` is written; a new
`structured_messages` object would make the chat translation layer skip the
`texts` write-back and silently drop the edits. A mask that lands nowhere
blocks rather than passing the original through.
`request_data` carries live Python objects (an OpenTelemetry span among them),
so `_json_safe` copies it into something serialisable by a mechanical rule
rather than a field list — a list drifts from what the far side needs, a rule
cannot. Serialising naively raises, and that error would read as "guardrail
unavailable" on every request.
26 tests, covering verbatim forwarding, each verdict, positional masking, the
`structured_messages` identity trap, both unreachable policies, and the
serialiser's handling of unserialisable values and cycles.
* fix(alice guardrail): satisfy lint and code-quality CI gates
- Bound _json_safe's recursion and register it in recursive_detector's
ignore list (it already caps depth and dedupes cycles by id, matching
the repo's established pattern for legitimate bounded recursion).
- Clear ruff-strict budget breaches: annotate __init__'s return type,
raise TypeError (not ValueError) for a bad response body, type
_json_safe's payload as object instead of Any, and file-scope-ignore
ANN401 for **kwargs (forwarding it as object broke the call into
CustomGuardrail.__init__, confirmed via basedpyright).
- Clear type-discipline budget breaches: suppress the construction/
annotation checks on one-shot HTTP payloads, the module-level
guardrail registries, and _json_safe's bounded accumulator; narrow
AliceVerdict's list fields to tuples and _evaluate's request_data to
Mapping[str, object] where nothing downstream mutates them.
* test(alice guardrail): assert the guardrail actually registers
The registration test called init_guardrails_v2 and asserted nothing, so it
passed whether or not the guardrail was ever registered — TQ001 in the
test-quality gate, and a fair catch: a test that cannot fail is not covering
the thing it names.
Now asserts exactly one AliceGuardrail lands in litellm.callbacks under the
configured name.
This surfaced only after the ruff-strict and type-discipline gates stopped
failing ahead of it; the lint job runs its gates in sequence, so an earlier
failure masks every later one.
* fix(alice guardrail): reach 100% patch coverage, drop the ActiveFence naming
Codecov flagged 10 uncovered lines, all of them error paths — which is where a
guardrail most needs covering, since each one decides whether traffic flows
unscreened.
Two of the ten turned out to be dead rather than untested, and are removed:
- `except GuardrailRaisedException: raise` in apply_guardrail. `_evaluate`
raises httpx errors, Timeout and TypeError, never that — so the clause could
never fire.
- the trailing `json.dumps` probe in `_json_safe`. Everything json.dumps
handles natively is caught by the isinstance branches above (a dict or list
subclass included), so anything reaching the bottom — bytes, datetime, an
OpenTelemetry span — cannot cross the wire regardless. It now says so and
returns None.
The rest are now tested: a timeout, 502/503/504 as unreachable, a 4xx as NOT
unreachable (a rejected credential is our misconfiguration, not an outage, and
must not fail open), a non-object response body, and a model whose model_dump
raises.
Also drops "by ActiveFence" throughout — the product is Alice — and points the
header at alice.io. `ui_friendly_name` is now "Alice", which is the key
guardrailLogoMap and the garden card look up, so all three moved together.
* fix(alice guardrail): strip caller credentials, widen unreachable detection, block partial MASK
Addresses PR review: request_data no longer forwards secret_fields.raw_headers or
the root api_key to Alice (the caller's Authorization token in the clear otherwise);
HTTP 500, malformed JSON, and a non-object body now route through the configured
unreachable_fallback instead of raising raw, so fail_open still fails open on those;
a MASK verdict with even one out-of-range replacement now blocks entirely instead of
silently letting the rest through unmasked. Also tightens request_data's type and
documents the known streaming-mask limitation on the class.
* fix(alice guardrail): strip credentials at any depth, stop filtering on texts
secret_fields/api_key/headers/provider_specific_header can appear nested
under proxy_server_request, metadata, litellm_metadata, and their
requester_metadata/body sub-paths in a real captured payload — a
top-level-only strip missed all of those. _json_safe now drops these keys
by name wherever they occur during serialization, so a new nesting path
can't reintroduce the leak.
apply_guardrail also stopped skipping the call whenever texts was empty,
even when tool_calls/images/structured_messages carried content — that
was the plugin making a selection decision Alice's design says belongs on
the far side. It now only skips when none of the selectable fields have
anything in them.
* fix(alice guardrail): route an undecodable response body through the fallback
`response.json()` raises UnicodeDecodeError when the body carries bytes that
are not valid UTF-8, and that escaped the except clause: UnicodeDecodeError is
a *sibling* of json.JSONDecodeError under ValueError, not a subclass of it, so
naming only JSONDecodeError left it uncaught. Both fallback modes surfaced a
raw decoding error instead of applying unreachable_fallback — which for a
fail_open deployment meant a hard failure where it had asked for an allow.
Named explicitly rather than widening to ValueError, so the clause still says
which three conditions it means. Tested under both policies.
GPT-5 and later accept a top-level anyOf natively and call tools better with it intact, so the flattening now runs only for the gpt-4, gpt-3.5, chatgpt-4o, o1, o3, and o4 families. Non-dict tool entries pass through untouched, a typeless root that carries properties counts as an object, and the bounded $ref walker is listed in the recursion detector allowlist.
A quoted routine call qualified by a schema and sitting inside a CREATE INDEX
expression, ON "Foo" (public."f"(col)), walked its qualifier read-through back
across the opening paren to the ON that introduces the indexed table, so the call
was misread as a relation and dropped from the call set, leaving a rewrite in that
routine unscanned. A word now only introduces the name when nothing but whitespace
and qualifier dots lies between them, so a paren in that gap keeps ON (and any
relation-introducing keyword) from reaching across it and the call stays a call.
The recursive_detector code-quality gate fails on litellm_internal_staging
because _flatten_form_field and _flatten_form_data_field in
llm_request_utils.py are recursive but absent from IGNORE_FUNCTIONS. Both are
bounded structural recursion over an already-parsed JSON-shaped request body
(a finite tree, no cycles possible), matching the existing ignored walkers, so
add them to the ignore list with a justification comment.
A quoted routine call with a SQL comment between its name and parenthesis,
`"backfill" /* reason */ ()`, is a real call that rewrites rows at boot, but the
call-site check read the raw SQL and stopped at the comment, so the routine was
read as uncalled and its rewrite slipped through. Read the call test from the
masked text instead, where every comment is already blanked to spaces, so a
comment between the name and its parenthesis is skipped exactly as whitespace is,
line, block and nested block comments alike, while a like-named non-call
identifier still opens no call and stays masked
A migration that defines an uncalled row-rewriting routine and elsewhere
references a quoted column, table, index, or constraint sharing the routine's
name was wrongly flagged: the guard restored every double-quoted identifier
before the name search, so a like-named identifier read as a call. Restore only
quoted names that open a call, followed by "(", so a routine invoked through a
quoted identifier is still caught while a like-named non-call identifier stays
masked and no rewrite-free migration is rejected
A migration that defines a row-rewriting routine and calls it as
"backfill"() at the top level slipped past the checker, since masking
blanks double-quoted identifiers before the routine-call search runs, so
the call could not be found by name and the body read as uncalled. mask()
now returns those identifier spans and outside_definition puts them back,
so a call written through a quoted identifier reads as the call it is and
the routine's body gets scanned the same as a bare call
The DML scan reconstructed a DO/EXECUTE'd literal with undouble, which collapses
each doubled quote to one character and shrinks the text. Every offset after a
collapsed pair then shifted, so a row-rewrite scanned out of the literal reported
an earlier file line and could miss a data-migration-ok marker placed on its real
line. Reconstruct with defuse_escapes instead, turning each doubled quote into a
quote and a space so the pair keeps its two characters and every offset holds,
while a nested -- or /* still stays inside its string.
The call-detection restore splices into a fixed-position list, so its
.ljust(end - start) holds that length invariant and a test guards it. The
DML-scan recursion instead hands the undoubled literal to a fresh scan_region
as its own region, whose length feeds nothing, so the pad only appends
trailing spaces that shift no keyword and change no reported line. Drop it and
the docstring clause that claimed it kept the offsets landing
Call detection restored a single-quoted DO or EXECUTE payload through
without_comments while its `''` escapes were still doubled. The first quote of
a pair opened an empty string and closed it on the second, leaving a `--` or
`/*` from a nested string bare, so it blanked the real call after it and the
routine read as uncalled: its rewrite body then went unscanned at boot. Undouble
each single-quoted payload before restoring it, and pad it back to the span it
fills so the later offsets still land. Dollar-quoted bodies do not escape quotes
and are left as they were.
