* feat(ui): keyset-paginate request logs by session trace
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep session grouping within type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format session grouping helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only group sessions when group_by_session is an explicit true
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reset session cursor on custom range and live tail toggles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover cursor reset on custom range toggle
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): ignore next page clicks while the grouped page is still fetching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): only block next page while grouped placeholder data is shown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Regenerating a key repointed ?key= at the rotated hash with a pushed history
entry, so pressing the browser Back button landed on the hash that had just
been revoked. /key/info answers 404 for it and the page shows "Key not found
in database".
The rotated hash now replaces the current entry instead of pushing a new one,
so Back from a just-regenerated key returns to the key list. Opening a key
from the table still pushes, so Back from a normally opened key is unchanged.
* feat(auto-router): support classifier reasoning effort
* fix(auto-router): harden classifier reasoning effort
* fix(ui): satisfy classifier config lint limits
* refactor(auto-router): simplify classifier effort support
* fix(auto-router): clear frontend-lint and type-discipline gates, trim LOC
---------
Co-authored-by: Tin Chi Lo <tin@berri.ai>
* fix(ui): clearing the organization picker no longer sends organization_id="" on key create
* fix(proxy): paginate Request Logs by conversation and aggregate session type counts and models server-side
* fix(proxy): keep access groups in sync when a model is renamed or deleted
* fix(proxy): cap the Request Logs conversation total like the row total
* fix(proxy): judge access group backing by the database for db models
A worker whose router has not polled the database yet still lists a sibling under its old
name, so a delete or rename handled there kept the stale name in every access group. Only
config-sourced deployments count as router backing now; db models are counted in the table.
* fix(ui): keep the conversation badge when an MCP call represents a conversation
A conversation that straddles the bounded page window can be represented by one of its MCP
rows, which showed a plain MCP badge and hid the session counts. The badge now reads the
server aggregates whenever the conversation has more than one call.
* fix(proxy): list every model of a conversation in Request Logs and keep the conversation badge for MCP representatives
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type session spend aggregates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): satisfy request logs lint budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cap per-session model aggregation in request logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: ratchet type-discipline budget after staging merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): send an explicit null when the key edit form clears the organization
Clearing the Organization picker in the key edit form wrote undefined into
the form value, and JSON.stringify drops undefined-valued keys, so
/key/update never saw the field and the key kept its old organization.
Writing null instead survives serialization, and the backend's
model_dump(exclude_unset=True) preserves it, so the column is set to NULL.
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep Virtual Keys list state in the URL so it survives leaving the page
The search term, sort, pagination and drawer filters lived in component
state, so navigating away from Virtual Keys and back reset the table to an
unfiltered first page. Move them into query state alongside the existing
?key= deep link, which also makes a filtered view shareable.
* fix(ui): namespace the Virtual Keys filter params and bound page inputs
The unprefixed team_id filter hijacked the /api-keys create-key deep link,
which already takes team_id as a prefill, so ?create=true&team_id=X silently
filtered the list underneath the modal. Prefix the four drawer filters.
Now that page and page_size come from the address bar, clamp them to what
/key/list accepts instead of forwarding 0, negatives or an int64-overflowing
page straight through, and trim filter values arriving from a URL the same
way the drawer already trims them.
* fix(ui): fall back to a sortable column when the URL names an unknown one
A hand-edited or stale sort_by reached /key/list, which 400s it, leaving the
Virtual Keys page on its loading skeleton with no error. Validate it against
the fields the table's own headers can produce, and clear sort_by rather than
blanking it when a sort is reset so the URL stays clean.
Also replaces a default-state URL assertion that ran before any query-state
write could land, so it could not fail for the regression it named.
* fix(ui): use TanStack's functionalUpdate instead of a hand-rolled updater resolver
The local helper narrowed typeof updater === "function" against an
unconstrained T, which TypeScript cannot do because T itself may be a function
type, so next build failed to type check. table-core already exports the same
helper.
* Fix hide-secrets guardrail: playground redaction, UI dropdown entry, spend-log telemetry
The hide-secrets guardrail never implemented apply_guardrail, so the UI test
playground echoed secrets verbatim; it was missing from the Add Guardrail
dropdown; and it recorded no guardrail_information, so Spend Logs could not
distinguish a redacted request from a clean one.
- implement apply_guardrail (unified interface) with use_native_lifecycle_hooks
so proxied traffic stays on async_pre_call_hook (per-key opt-out and
data["prompt"] handling live only there)
- record standard_logging_guardrail_information (allow/mask + masked_entity_count)
via _process_response/_process_error; opted-out keys and legacy nameless
callback instances record nothing
- advertise hide-secrets in /guardrails/ui/add_guardrail_settings (pre_call only)
and /guardrails/ui/provider_specific_params with a config model
Resolves LIT-3548
* Fix hide-secrets passthrough telemetry and JSON config input
* fix(guardrails): validate hide-secrets object config before submit
- apply_guardrail treats empty-string-only texts as no input, so no
false allow is recorded
- the UI object field keeps raw text while editing and blocks submission
until it parses to a JSON object, instead of posting a string to an
object-only API
- supported_modes_by_provider keeps its dict[str, list[str]] value type
* fix(guardrails): record no hide-secrets telemetry when nothing was inspected
walk_user_text and the prompt redaction now report how many non-empty
strings they visited; when neither inspected anything (image-only
content, empty strings), the run records no guardrail entry instead of
an 'allow' row that counts a check which never saw any text.
* fix(ui): stop the create team form resetting organization and models
The organization preselect ran in an effect keyed on the organizations
query, so any refetch of that list while the Create Team modal was open
overwrote the user's organization pick, which in turn cleared their
models pick. The models field was also cleared whenever the available
models fetch resolved.
Preselect the organization when the modal opens instead, and clear the
models only when the user picks a different organization. An org admin
whose admin orgs narrow to one while the form is open can still pick,
rather than facing a locked empty field.
* fix(ui): block team create when the picked organization is no longer available
An organization picked in the Create Team form now survives a refetch of
the organization list, so it can outlive the admin's access to it. Refuse
the create with a message on the field rather than letting the request
fail authorization at the proxy.
* fix(ui): keep the team create organization field usable when the pick goes stale
Locking the field on a single admin organization also locked it while it
held a rejected organization, so an admin who lost access could not pick
the one organization left. Lock it only while it holds that organization.
* test(ui): hoist the created team fixture out of the mock call
The inline object pushed the repo past its no-large-inline-object-arg
lint budget, which has no headroom.
The autoRouterPresets mock imported litellm/proxy/public_endpoints/autorouter_presets.json
as a module. That path sits outside ui/litellm-dashboard, the only directory the UI
Dockerfile copies, so `next build` type-checking inside the image failed with
"Cannot find module" and the ui-image job went red on every PR that touched an
image-scan path. Read the file with fs at runtime instead; vitest still derives
expectations from the real bundled catalog.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HqEPCNLDrssxsuezhAaL4j
A session pinned to a text-only model failed every image turn with a provider 400, because the modality gate exempts a kept pin by cause. Add modality_pin_override so that exemption is conditional: the image turn is re-placed on a capable model for that request only, reported as cause modality_pin_override, and the stored pin is left untouched so the next text turn replays it.
The pin write on the replay path already happens upstream of the gate and stores the session's own model, so pin survival is structural rather than bookkeeping. The new cause joins the non-pinnable set. Default off at every layer.
* fix(proxy): keep SpendLogs and callback session ids in sync when the request has none
Add general_settings.missing_session_id (generate | reject). In generate mode one id is
stamped into litellm_session_id, litellm_trace_id and metadata.session_id before callbacks
run, so LiteLLM_SpendLogs.session_id and the Langfuse session id match. In reject mode such
requests get a 400. Unset keeps the legacy behavior. MCP routes are not affected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(proxy): regenerate schema.d.ts and shorten mutable-ok comment for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark generated session ids so affinity consumers do not pin on them
Fireworks x-session-affinity, the router session_affinity pre-call check and the
complexity router session pin all read metadata.session_id as a caller-chosen
stable key. A missing_session_id: generate id is fresh per request, so it now
carries metadata.litellm_session_id_generated and those consumers skip it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The dashboard's template picker imported autorouter_presets.json at build time, so every
catalog change needed a dashboard rebuild and artifacts refresh. The catalog now lives in
litellm/proxy/public_endpoints/ and GET /public/autorouter_presets serves it, fetching
litellm.autorouter_presets_url (GitHub raw on main, 1h in-process cache, bundled fallback)
so a merged catalog change propagates to running proxies like the model cost map does.
The dashboard fetches it at runtime via useAutoRouterPresets and keeps no local copy.
Resolves LIT-6764
* fix(ui): paginate request logs by session groups server-side
The logs table server-paginated raw spend logs and then collapsed
multi-call sessions client-side, so a page could render 3 rows while
the footer claimed 25 and sessions straddled pages. Adds an opt-in
group_by_session param to /spend/logs/ui that pages and counts one
representative row per session (DISTINCT ON, newest non-MCP call),
keeps the bounded count contract, enriches whole-session llm/agent
composition counts, and deletes the client-side collapse pipeline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* feat(ui): add a 10 rows-per-page option and default request logs to it
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): key session aggregates per api key in the logs enrichment
Grouped pagination splits a reused session id into one row per api key,
but the enrichment still aggregated by session_id alone, so both rows
showed combined spend and counts. The aggregate query now groups by
(session_id, api_key), the count folds into it (the separate group_by
query is deleted), and each row reads its own key's totals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
* fix(ui): treat an empty api_key as a real session group value
The spend-log schema defaults api_key to an empty string; truthiness
guards in the enrichment treated it as missing, so keyless multi-call
sessions lost their count and spend. Only None means missing now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QxT89fiygmzz2ALcjpu7Ve
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(logging): redact credential query params from the uvicorn access log
Raw virtual keys reached container stdout two ways:
- `GET /key/info?key=sk-...`, `/global/spend/report?api_key=sk-...`,
`/key/spend/report`, `/spend/logs`, `/user/daily/activity` and the Gemini
passthrough routes all put the credential in the request target, and
`uvicorn.access` had no redaction filter (only `uvicorn.error` did).
- the key budget error interpolates `LiteLLM_VerificationToken.key_name`,
a column with no enforced shape, into a message that is both logged and
returned to the caller.
`SecretRedactionFilter` cannot be reused on an access logger: it collapses the
record into `record.msg` and clears `record.args`, and uvicorn's AccessFormatter
unpacks those args at emit time, so every access line would raise TypeError.
`AccessLogRedactionFilter` scrubs the positional args in place instead.
An access line is the one input to the secret regex an unauthenticated caller
controls end to end, so two bounds go with it. The request target is cut back to
a whole query parameter under 512 characters before it is scanned, since a half
parameter is too short to match its own pattern and would be logged raw, and the
dropped tail is not logged at all. The connection-string pattern is bounded too,
because its user half could previously re-scan the rest of the string from every
`://`: a 16 KB URL of `a://` pairs took 314s and now takes 0.12s, with the caps
set high enough that an RDS IAM auth token used as a DSN password still redacts.
Credential query params are terminated by `&` like the existing `key=` and
`sig=` patterns, so redacting one param no longer swallows the rest of the
request line, and a second credential in the same query string is now redacted
on its own instead of surviving once the first one stops the span. `key_name` is
echoed into the budget error only when it still has the masked `sk-...abcd`
shape `abbreviate_api_key` writes, so a value put there by a direct DB write or
a migration falls back to the key alias.
Also point the `/key/info` and spend-report examples at the sha256 hash both
endpoints already accept, so callers stop putting raw keys in URLs that
third-party access logs record.
Resolves LIT-5909
* test(logging): assert on emitted access lines instead of filter registration
The two registration tests checked that an AccessLogRedactionFilter instance
sits in uvicorn.access.filters, which is the shape of the code rather than its
behavior. Handing the logger a real access record and reading what a handler
wrote covers the same wiring and still fails when the registration is removed.
* fix(logging): redact percent-encoded credentials from access logs
?k%65y=sk%2D... is a working credential once the request parser decodes it,
but the redaction patterns match literal text and never see it. Decode the
request target as a detector and drop the query when decoding reveals a
secret. The decoded text is never logged back, so a %0A cannot forge a
following log line
Also accept any four non-space characters in the masked key_name check, since
abbreviate_api_key copies the last four characters of a custom key verbatim
and those can be punctuation or non-ASCII
* fix(auth): keep control codes out of the masked key label
/key/generate accepts a custom key ending in an escape sequence, and
abbreviate_api_key copies those four characters into key_name verbatim, so
the over-budget message carried them to a terminal and a log viewer. Bar
whitespace and C0/C1 control codes from the four, and keep everything else
The mcp_tool_search virtual tool only did substring token matching, so a native MCP client asking for "FX" could not find a tool described as "foreign exchange rates" even though the same catalog is ranked by embeddings on /responses and /chat/completions.
Adds litellm_settings.mcp_tool_search (embedding_model, top_k, similarity_threshold, core_tools). With an embedding model the caller's authorized catalog from _list_mcp_tools is ranked by cosine similarity of name plus description; configured core tools the caller can reach come first and do not consume top_k. Without an embedding model the keyword fallback keeps the old behavior. Settings are hot-reloadable from the DB, exposed on /get and /update mcp_tool_search_settings, and editable from the Admin UI under MCP Servers > Tool Search. The embedding index is shared with agent_search via a new SemanticTextIndex.
Resolves LIT-6751
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds a configurable password-strength policy (default: min 12 chars,
upper/lower/number/special, all individually toggleable, floored at 8
so a misconfigured minimum cannot disable the length check, and
unicode-aware so an accented letter cannot satisfy the special-
character requirement) enforced on every path that sets a local
user's password: /user/update, /user/bulk_update, and the invitation
onboarding claim flow.
Adds general_settings.disable_password_login_when_sso_enabled, which
rejects username/password login on /login, /v2/login and /v3/login
(including the UI_USERNAME/UI_PASSWORD admin fallback) once ANY
configured SSO provider is FULLY ready: every companion secret/
endpoint an OAuth provider needs, checked independently per provider
so a stray leftover client id for an unused provider can't mask a
different, fully configured one; and for SAML, the optional
python3-saml runtime being importable, checked without letting a
fully-missing package's ModuleNotFoundError take down password login
itself. SSO becomes the enforced boundary for interactive UI access
without an incomplete, mixed, or half-installed SSO setup locking
every admin out or breaking login outright. Master-key API access is
untouched, and unsetting the setting plus a restart restores password
login as the documented recovery path.
* fix(security): restrict and validate file uploads at /v1/files and /upload/logo
Extends fast-fail upload validation to every purpose at POST /v1/files,
not just purpose=batch: a configurable max_file_size_mb size cap and a
blocked_file_extensions denylist, plus rejection of filenames carrying a
directory-traversal component before anything is read, stored, or
forwarded to a provider.
Also fixes two concrete gaps found while auditing every upload surface:
the Azure Blob Storage backend derived a blob path's extension with
filename.split(".")[-1], which does not parse path structure and let a
crafted filename embed a directory traversal sequence into the stored
blob path; and POST /upload/logo (the admin UI logo upload) had no
role check at all, so any authenticated API key, not just a proxy
admin, could write a file to the server's disk.
* fix(lint): drop cast()/mutation from settings coercion, sync blocked_file_extensions on reload
Replaces the TypeAdapter+cast() reads of max_file_size_mb and
blocked_file_extensions with small isinstance-based validators, since the
codebase's cast() budget (LIT006) had no headroom left. Also adds the
blocked_file_extensions reload block that was missing from
_update_general_settings: it was registered as an editable setting but
never re-synced into runtime state, so a value set through the DB-backed
settings editor would silently never take effect (Greptile finding).
* fix(security): declare max_file_size_mb and blocked_file_extensions on ConfigGeneralSettings
The DB-backed general-settings update endpoints validate every field
through ConfigGeneralSettings.model_fields before persisting it, so
without these declarations an operator could never actually set either
setting through that path even though both were registered for the
Admin UI's settings editor and reloaded on config refresh (Greptile
finding). blocked_file_extensions is typed as a tuple, not a list, to
stay out of the immutable-collections lint budget; the stored JSON
value is unaffected since the raw request payload, not the validated
model, is what gets persisted.
* chore: regenerate schema.d.ts for the new ConfigGeneralSettings fields
* fix(security): normalize configured blocked_file_extensions casing
check_blocked_extension lowercased the uploaded filename's extension
before comparing but compared it against blocked_extensions verbatim,
so an admin-configured blocked_file_extensions: ['.EXE'] would never
match an uploaded payload.exe (Greptile finding). Normalizes the
configured values the same way at comparison time, and adds the
missing case (mismatched-case config, lowercase upload) as a
regression test, mutation-checked against the unfixed comparison.
* fix(security): restore caller-owned stream position after size inspection
_file_size_bytes unconditionally seeked back to 0 after measuring a
BinaryIO's length, discarding wherever the caller had actually
positioned it (Greptile finding). Saves and restores the original
position instead. Rewrites the existing test that had encoded the
old "always resets to 0" behavior as its expectation, and adds a
sibling case for the under-cap path; both are mutation-checked
against the unfixed always-reset-to-0 behavior.
parseDynamicAgentForForm recovered a credential field's value from a
stored model string by splitting both the model_template and the model
on "/" and matching by array index. That breaks for any placeholder
value that itself contains "/", such as a Bedrock AgentCore runtime ARN
resource path (runtime/<runtime-id>), silently dropping everything
after the first slash when populating the edit form. Saving without
touching the field then persisted the truncated ARN.
Replace the index-matching split with a non-mutating template parse
(split on the placeholder pattern, escape and rejoin the literal
segments into a regex) so a placeholder captures everything it needs
regardless of embedded slashes. Also add a lightweight ARN-shape
validator for the AgentCore runtime ARN field, guarded against a
malformed pattern string, so a truncated value is rejected client-side
before it reaches the backend.
Resolves LIT-6737
Clearing the Team combobox in the Create Key modal left team_id set to an
empty string, so /key/generate treated the request as team key generation
and failed with a team-not-found error for non-admin members.
TeamDropdown now emits null on clear, and GenerateKeyRequest normalizes an
empty team_id to None so the request runs the personal key path.
* feat(alerting): slack alerts for per-user daily/monthly spend thresholds and spend anomaly detection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(alerting): use specific ValidationError matches in config rejection test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): tolerate mocked slack alerting args when scheduling user spend scan
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(alerting): reject non-finite values in user spend alert settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): default max_idle_connection_lifetime to 60s on DB URLs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): regenerate schema.d.ts for database_max_idle_connection_lifetime
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep URL-pinned max_idle_connection_lifetime over config value
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep litellm_credential_name from LiteLLM Params JSON when no credential is selected
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drop null litellm_credential_name from AddModelPanel payload fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): validate JSON litellm_credential_name against accessible credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce proxy-admin-only credential attachment on model create/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): raise ProxyException for unauthorized credential attach and gate /model/update
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): fold credential-change detection into can_user_attach_credential to satisfy complexity budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): decrypt stored credential name before unchanged-credential comparison
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover credential attach rejection on add_new_model and patch_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): annotate proxy-global patches with test-quality suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Clicking a Base UI select entry found by text or by a title attribute is a
race. The text node exists one render before the popup finishes entering,
and until then the positioner still carries pointer-events: none, so
user-event refuses the click and the test throws. Querying by role only
matches once the popup is exposed to the accessibility tree, which is after
that window closes.
Route the 37 remaining select interactions through chooseSelectOption, which
does the role query. Instrumenting the converted files shows the text query
resolving while the popup was still pointer-blocked on 6 of 41 samples; the
role query was never blocked.
Seven files kept their text queries because their popup entries carry no
accessible role, so there is nothing to query by.
The page set its headings, table borders, sidebar labels and tag pills
inline with a fixed light palette (#202124, #5f6368, #dadce0, #f8f9fa,
#fff), so in dark mode it drew dark text on hardcoded white surfaces.
Move those to the foreground/muted/border/card/info tokens, matching
the back link and Create Guardrail button that already used them.
* feat(guardrails): add Alice by ActiveFence guardrail
Adds `guardrail: alice` — policy-based guardrails for prompts and model
responses, evaluated against ActiveFence's Alice.
What makes this different from the other providers: Alice evaluates against
policies configured per *application*, and a proxy typically fronts several of
them, so the application cannot be a static config value. It is named on the
LiteLLM virtual key instead:
curl $PROXY/key/generate -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-d '{"key_alias": "payments-bot",
"metadata": {"alice_app_id": "payments-bot"}}'
read via `CustomGuardrail._get_admin_metadata`, with `key_alias` as the
fallback. That helper is what makes it trustworthy: it reads whichever metadata
holder the proxy wrote the authenticated key's values into — which differs by
route — and the proxy strips caller-supplied `user_api_key_*` from both, so a
caller cannot point its own traffic at an application with laxer policies than
the one its key was issued for. A request whose key names no application is
refused rather than evaluated against a guess.
Implements `apply_guardrail` only, so pre_call, during_call, post_call and
streaming all come from UnifiedLLMGuardrails. Blocks with
GuardrailRaisedException; masks by substituting Alice's redacted text; a MASK
carrying no replacement blocks rather than passing the original through. A
verdict reporting `errors[]` is treated as a failure, not a pass — otherwise a
half-evaluated message would be allowed. `unreachable_fallback` (already on
LitellmParams) chooses fail-closed or fail-open on transport failure.
Config:
guardrails:
- guardrail_name: alice
litellm_params:
guardrail: alice
mode: [pre_call, post_call]
api_key: os.environ/ALICE_API_KEY
21 tests in tests/test_litellm/proxy/guardrails/guardrail_hooks/test_alice.py
cover registration, credential resolution, the app-id ladder including the
forged-metadata case, every verdict, and both unreachable policies.
No new LitellmParams field, so no schema.d.ts regeneration is needed.
* refactor(guardrails): post to Alice's LiteLLM endpoint and forward verbatim
Switches from `/v2/evaluate/message` — Alice's single-text endpoint — to
`/v2/evaluate/litellm`, which takes the hook's arguments as they arrive and
answers with a verdict.
That inverts where the work happens, and shrinks this plugin accordingly. It
now selects nothing and renames nothing: it posts `{input_type, inputs,
request_data}` and enforces `{verdict, categories, correlation_id, message,
replacements}`. Which parts of a conversation are worth evaluating, and how a
verdict is reached, are decided by Alice — so changing either is a change on
their side rather than a LiteLLM upgrade for every user.
The app-id resolution this plugin carried is gone with it. Alice reads the
application off the authenticated key's metadata itself, from the payload it is
handed, so the ladder here was duplicating a decision the far side already
makes. The security property is unchanged and still comes from the proxy
stripping caller-supplied `user_api_key_*` before a guardrail sees the request.
Masking is now positional — the far side chose which texts it was answering
for, so it says which by index. Only `texts` is written; a new
`structured_messages` object would make the chat translation layer skip the
`texts` write-back and silently drop the edits. A mask that lands nowhere
blocks rather than passing the original through.
`request_data` carries live Python objects (an OpenTelemetry span among them),
so `_json_safe` copies it into something serialisable by a mechanical rule
rather than a field list — a list drifts from what the far side needs, a rule
cannot. Serialising naively raises, and that error would read as "guardrail
unavailable" on every request.
26 tests, covering verbatim forwarding, each verdict, positional masking, the
`structured_messages` identity trap, both unreachable policies, and the
serialiser's handling of unserialisable values and cycles.
* fix(alice guardrail): satisfy lint and code-quality CI gates
- Bound _json_safe's recursion and register it in recursive_detector's
ignore list (it already caps depth and dedupes cycles by id, matching
the repo's established pattern for legitimate bounded recursion).
- Clear ruff-strict budget breaches: annotate __init__'s return type,
raise TypeError (not ValueError) for a bad response body, type
_json_safe's payload as object instead of Any, and file-scope-ignore
ANN401 for **kwargs (forwarding it as object broke the call into
CustomGuardrail.__init__, confirmed via basedpyright).
- Clear type-discipline budget breaches: suppress the construction/
annotation checks on one-shot HTTP payloads, the module-level
guardrail registries, and _json_safe's bounded accumulator; narrow
AliceVerdict's list fields to tuples and _evaluate's request_data to
Mapping[str, object] where nothing downstream mutates them.
* test(alice guardrail): assert the guardrail actually registers
The registration test called init_guardrails_v2 and asserted nothing, so it
passed whether or not the guardrail was ever registered — TQ001 in the
test-quality gate, and a fair catch: a test that cannot fail is not covering
the thing it names.
Now asserts exactly one AliceGuardrail lands in litellm.callbacks under the
configured name.
This surfaced only after the ruff-strict and type-discipline gates stopped
failing ahead of it; the lint job runs its gates in sequence, so an earlier
failure masks every later one.
* fix(alice guardrail): reach 100% patch coverage, drop the ActiveFence naming
Codecov flagged 10 uncovered lines, all of them error paths — which is where a
guardrail most needs covering, since each one decides whether traffic flows
unscreened.
Two of the ten turned out to be dead rather than untested, and are removed:
- `except GuardrailRaisedException: raise` in apply_guardrail. `_evaluate`
raises httpx errors, Timeout and TypeError, never that — so the clause could
never fire.
- the trailing `json.dumps` probe in `_json_safe`. Everything json.dumps
handles natively is caught by the isinstance branches above (a dict or list
subclass included), so anything reaching the bottom — bytes, datetime, an
OpenTelemetry span — cannot cross the wire regardless. It now says so and
returns None.
The rest are now tested: a timeout, 502/503/504 as unreachable, a 4xx as NOT
unreachable (a rejected credential is our misconfiguration, not an outage, and
must not fail open), a non-object response body, and a model whose model_dump
raises.
Also drops "by ActiveFence" throughout — the product is Alice — and points the
header at alice.io. `ui_friendly_name` is now "Alice", which is the key
guardrailLogoMap and the garden card look up, so all three moved together.
* fix(alice guardrail): strip caller credentials, widen unreachable detection, block partial MASK
Addresses PR review: request_data no longer forwards secret_fields.raw_headers or
the root api_key to Alice (the caller's Authorization token in the clear otherwise);
HTTP 500, malformed JSON, and a non-object body now route through the configured
unreachable_fallback instead of raising raw, so fail_open still fails open on those;
a MASK verdict with even one out-of-range replacement now blocks entirely instead of
silently letting the rest through unmasked. Also tightens request_data's type and
documents the known streaming-mask limitation on the class.
* fix(alice guardrail): strip credentials at any depth, stop filtering on texts
secret_fields/api_key/headers/provider_specific_header can appear nested
under proxy_server_request, metadata, litellm_metadata, and their
requester_metadata/body sub-paths in a real captured payload — a
top-level-only strip missed all of those. _json_safe now drops these keys
by name wherever they occur during serialization, so a new nesting path
can't reintroduce the leak.
apply_guardrail also stopped skipping the call whenever texts was empty,
even when tool_calls/images/structured_messages carried content — that
was the plugin making a selection decision Alice's design says belongs on
the far side. It now only skips when none of the selectable fields have
anything in them.
* fix(alice guardrail): route an undecodable response body through the fallback
`response.json()` raises UnicodeDecodeError when the body carries bytes that
are not valid UTF-8, and that escaped the except clause: UnicodeDecodeError is
a *sibling* of json.JSONDecodeError under ValueError, not a subclass of it, so
naming only JSONDecodeError left it uncaught. Both fallback modes surfaced a
raw decoding error instead of applying unreachable_fallback — which for a
fail_open deployment meant a hard failure where it had asked for an allow.
Named explicitly rather than widening to ValueError, so the clause still says
which three conditions it means. Tested under both policies.
Ports the Model Hub search to the AI Hub Agent Hub tab and the admin
/agents toolbar as a client-side filter over agent name and description.
Extracts the hub search matching into utils/searchUtils and fixes the
public Model Hub rendering the whole catalog when a search matches
nothing (LIT-5230)
