The corroboration check belongs to adopting a token_url from any non-manual
source, not to discovery alone. Carry-forward is the other such source: it
copied a prior registry entry's token_url/registration_url onto a rebuild
whose authorization_url had been re-pointed to a different server, reviving an
uncorroborated token endpoint the discovery gate would reject. Both sites now
share one predicate, _endpoints_corroborate_authorization_url: previous
endpoints carry forward only when the previous authorization_url corroborates
the authorize endpoint the build will use (absent -> the previous one is
adopted too, a consistent group; else it must match). Endpoint comparison now
elides the default port so :443 and formatting-only differences still match.
Discovery is rooted at the MCP resource, so a compromised upstream can
advertise an attacker-run authorization server. When authorization_url is
manually configured and another field is blank, the per-field merge would
combine the trusted authorize endpoint with the advertised token_url, and
the gateway would redeem authorization codes (with the stored client secret
and PKCE verifier) at that endpoint, then persist it. Discovered token_url
and registration_url are now accepted only when the same metadata document
advertises an authorization_endpoint matching the configured value
(scheme+host+path). Scope backfill is unaffected. Applies to both the DB
and config build paths.
* feat(bedrock guardrails): add resource-less InvokeGuardrailChecks (detect-only) mode
Adopted from #30830 by OS-joaocastilho; the original PR was merged into
litellm_oss_staging_230626, which never landed, so this re-lands it on
litellm_internal_staging
Beyond the original diff, this fold includes the review fixups that were
made on the staging branch (warn on unrecognized check keys, keep empty
known checks as enable-with-defaults, fail fast when the checks block has
no usable keys, tz-aware datetimes, stricter typing) and adapts the block
path to the ModifyResponseException contract from LIT-4186, which replaced
GuardrailInterventionNormalStringError after the original PR was written
* fix(bedrock guardrails): only evaluate configured checks in violation collection
An unsolicited score in the InvokeGuardrailChecks response (e.g. a future
API revision returning checks the user never requested) previously fell
through to the default 0.5 threshold and could block a request the user
only asked to scan with other checks. Violation collection now skips any
check absent from the configured checks block
* fix(bedrock guardrails): fail closed on truncated PII results and tighten checks-path typing
Truncated sensitiveInformation results now count as a violation when the
PII check is configured: Bedrock omitted detections that were never
scored, so sub-threshold visible entries no longer let the request pass.
Also blocks on score == threshold per the documented contract (regression
test added), rejects checks combined with guardrailVersion, turns a
malformed 200 body into a logged guardrail_failed_to_respond 500 instead
of a raw ValidationError, types the checks parameter and violations
(BedrockChecksConfigModel, BedrockChecksViolation) instead of dict/object,
types _sign_and_post against AWSPreparedRequest, hoists stdlib imports,
and builds checks messages without intermediate mutation
* fix(bedrock guardrails): tag all InvokeGuardrailChecks INPUT content as user
Bedrock excludes system content from prompt-attack evaluation (per the
AWS guardrails docs), so mapping a caller-supplied system/developer
message onto the system role let a caller hide a prompt injection from
the promptAttack check by self-labeling its role. At the proxy every
INPUT message is caller-controlled, so all of it is now tagged as
untrusted user input, which also matches AWS guidance to tag untrusted
content as user input. OUTPUT stays assistant. Removes the now-unused
role map; the input-message test asserts the new tagging as a regression
* fix(bedrock guardrails): pass prepared request headers to httpx without dict coercion
httpx accepts botocore's HTTPHeaders mapping directly, and wrapping it in
dict() broke the existing test_bedrock_guardrail_make_api_request_passes_api_key
which supplies a bare Mock as the prepared request (dict(Mock) calls
Mock.keys())
---------
Co-authored-by: OS-joaocastilho <144790013+OS-joaocastilho@users.noreply.github.com>
* test(claude_code): move the Claude Code compatibility matrix under tests/e2e
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(claude_code): drop the CircleCI compat PR gate; the matrix runs in the scheduled e2e suite instead
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: restore the upload-coverage job dropped by mistake with the compat gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(e2e/claude_code): print rate-limit summary on failed compat runs and fix stale run_daily.sh header comments
* test(claude_code): assert fine-grained tool streaming via input_json_delta instead of an event-count floor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Extracts the adapter stream's closed delta-type set into
StreamingContentBlockDeltaType, shared by the translate layer's return
type and an exhaustive match in _delta_payload_field, so adding a new
delta type without handling it in the emission gate fails
type-checking instead of being silently dropped.
An empty upstream delta (e.g. Bedrock Converse's empty reasoning delta
mid-thinking-block) falls through the translate fallback as
text_delta {"text": ""} at the open thinking block's index, crashing
Anthropic SDK clients like Claude Code with "Content block is not a
text block". Payload-less deltas carry no information, so never emit
them.
* feat(guardrails): support streaming text transformation in generic_guardrail_api
* chore(guardrails): address PR review feedback
* fix(guardrails): fail closed on tool-call and prefix-rewrite leaks in streaming transform
* fix(guardrails): address Bugbot review on streaming transform correctness
* fix(guardrails): coerce holdback in handler for in-process guardrails
* fix(guardrails): harden streaming transform (holdback coercion, tool-call passthrough, n>1 finish_reason)
* test(guardrails): targeted _mode_matches coverage for all guardrail_mode shapes
* fix(guardrails): inspect streamed tool calls and harden incremental_diff edge cases
* test: move ComplianceChecker mode tests to the compliance PR
* fix(guardrails): strip content from tool-call passthrough so streamed text can't bypass the transform
* fix(guardrails): four correctness fixes for incremental_diff streaming path
Four bug fixes on top of the OSS PR's incremental_diff streaming text
transformation, all inside the incremental_diff code paths only. No
existing block_only, non-streaming, or pre_call behavior is touched.
Fix#1 — Mixed content+tool_call finish_reason ordering
_tool_call_passthrough_chunk now takes an optional finish_reason_per_choice
map. For a choice carrying both delta.content and delta.tool_calls,
finish_reason is stripped from the passthrough and recorded on the map so
the final synthetic text chunk delivers it. Without this, SSE-compliant
clients stopping at finish_reason drop the guardrailed text — defeating
the redaction the whole feature exists for. (Greptile P1 twice, Veria.)
Fix#2 — Choice index sort in _process_streaming_transform
indices/texts_to_check were derived from dict insertion order. For n>1
streams where choice 1 emits before choice 0, guardrail-returned texts
aligned to the input order mapped back to the wrong choice indices on
write-back — wrong text goes to wrong choice. Sort raw_by_index.keys()
up front so realignment is deterministic. (Bugbot Medium.)
Fix#3 — Cross-chunk pre-tool-call text flush
With default streaming_sampling_rate=5, text chunks followed by a pure
tool-call chunk carrying finish_reason='tool_calls' would emit the
passthrough with finish_reason before any transformed text delta had
fired. Same failure mode as fix#1 but cross-chunk. Now we flush any
accumulated text via _round(is_final=False) BEFORE yielding the
tool-call passthrough. (Greptile P1.)
Fix#4 — Terminator chunk for deferred finish_reason on empty mutated_text
_build_transform_chunk returned None early when mutated_text_per_choice
was empty. If a mixed content+tool_call chunk had deferred its
finish_reason (via fix#1) and the guardrail then suppressed the text
(empty return), the deferred finish_reason was never delivered. Now on
is_final=True with empty mutated_text_per_choice, we emit a terminator
carrying finish_reason per choice from finish_reason_per_choice.
(Bugbot High.)
Also normalized Optional[X] → X | None across the OSS PR's added surface
via ruff UP045 autofix to keep the strict-rule gate within budget. Pure
mechanical typing style change, no semantic effect.
Regression tests for all four fixes:
- test_mixed_chunk_finish_reason_arrives_after_transformed_text (#1)
- test_text_flush_precedes_tool_call_passthrough (#3)
- test_final_finish_reason_flushed_when_guardrail_suppresses_text (#4)
- test_transform_sends_texts_sorted_by_choice_index (#2)
All fixes reachable only when streaming_transform_mode == 'incremental_diff'
is configured (via _run_incremental_transform_stream) or when a
StreamTransformSink is present (via _process_streaming_transform). Verified
scope-clean: no changes to block_only, non-streaming, pre_call, moderation,
or sibling guardrails.
---------
Co-authored-by: Marton Schneider <marton@schneider.co.nl>
* feat(ui): migrate guardrails table onto shared DataTable
Move the guardrails list onto the shared DataTable + cell library as the
proof-of-concept for the simple-tables design migration, following the Teams
reference pattern.
Split the table into a thin container (guardrail_table.tsx) and column defs
(guardrailTableColumns.tsx): client-side sort defaulting to created_at desc, a
search + refresh toolbar, IdCell / DateCell / StatusBadge cells, real provider
logos, a rich empty state, and skeleton loading rows. Row actions move into a
per-row overflow menu; deletion stays disabled for config-file guardrails, now
surfaced as a disabled menu item instead of a greyed trash icon. Detail view
and the delete modal remain owned by GuardrailsPanel.
Restyle the "Add New Guardrail" control to the shared Button + dropdown menu.
Update the regression tests for the menu-based actions and drop the now-stale
eslint suppression entry that the rewrite eliminated.
* fix(ui): match guardrails table to the design
Address design-review feedback on the guardrails migration:
- Drop the search + refresh toolbar. The original table had neither and the
SimpleTable design has no toolbar; the container now just renders the sorted
table and its empty state.
- Give the Guardrail ID cell the design's hover affordance by rendering it with
the shared IdentityCell (monospace, chevron on hover) instead of the blue
IdCell pill.
- Stop pinning the actions column. Pinning added a sticky divider that the
design and the Teams table don't have; it is now a plain right-aligned menu
column, matching Teams.
* fix(ui): match loading skeleton row height to loaded rows
The compact skeleton row did not carry the h-8 height that real compact
rows get, so loading rows rendered shorter than loaded ones and the table
height jumped when data arrived. Mirror the same size-based height on the
skeleton row in the shared DataTable so every compact table loads at a
stable height
* test(ui): drop stale onGuardrailUpdated from guardrails table baseProps
The prop was removed from GuardrailTableProps when the toolbar went away;
the test baseProps still listed it. Harmless at the call site since it is
spread rather than an object literal, but dead and worth removing
* fix(ui): remove dead edit_guardrail_form after guardrails migration
The guardrails table migration dropped the last import of EditGuardrailForm,
which knip flags as an unused file. The form was already unreachable before
the migration: the table wired a delete button only, and nothing ever called
handleEditClick to open the modal, so the import was the sole thing keeping
the file referenced. Delete it and prune its now-stale eslint suppression
entry. Guardrail editing is unchanged and lives in the detail view
(GuardrailInfoView)
With enable_jwt_auth enabled but no enterprise license (premium_user
False), the JWT premium check fired on every request before the token
was inspected, so the master key, sk- virtual keys, and the encrypted
CLI/UI SSO session token that `lite login` issues all 401'd with "JWT
Auth is an enterprise only feature" and were never decoded. That broke
`lite login`, `lite claude`, and the proxy master key on any deployment
that turned JWT auth on without a license.
Move the premium check inside the is_jwt branch so it gates only real
JWTs. Non-JWT credentials fall through to their own auth paths
regardless of license; actual JWTs still require premium, so the
enterprise gate is unchanged for the feature it protects.
The rate-limited batch spend test snapshotted unattributed rows via the
unpaginated /spend/logs whole-table read, which grows with the environment
(58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every
scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an
explicit date window instead, and SpendLogsParams now rejects a filterless
read so the whole-table call cannot come back
* fix: enforce user budget on team keys
User budget was skipped when the key belonged to a team, letting
users exceed their personal budget by going through a team key.
Remove the team_object guard in _user_max_budget_check so user
budgets are always enforced. Add skip_user_budget_on_team_key
general_settings flag to opt back into the old behavior.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: update test to expect user budget enforcement on team keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): enforce user budget on team keys in reservation path and expose skip flag in UI
Extends the read-time fix so the optimistic budget reservation also reserves the user spend counter for team-scoped keys, register skip_user_budget_on_team_key in ConfigGeneralSettings so /config/field/update accepts it, and surface it as a Boolean toggle on the Admin UI General Settings table via allowed_args in /config/list.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: assert budget_exceeded ProxyException in personal budget test
Tighten the broad pytest.raises(Exception) so the test only passes when
the auth flow rejects with a budget_exceeded ProxyException, and switch
the new ConfigGeneralSettings field to Optional[bool] to match the
surrounding annotation style
* fix: revert to bool | None to stay under UP045 strict budget
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(anthropic/passthrough): drop temperature and cap thinking budget when downgrading adaptive thinking for pre-4.6 models
* test(anthropic/passthrough): use sufficient max_tokens for reasoning_effort thinking mapping
* fix(anthropic/passthrough): drop incompatible temperature when downgrading adaptive thinking for pre-4.6 models
Narrow the fix to the temperature reconciliation; the reasoning_effort
budget cap is reverted because the live translation grid relies on
budget_tokens >= max_tokens to reject unsupported effort tiers
(xhigh/max) on budget-mode models, so capping turned those 400s into
200s.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Move the Create New Key and Create Team buttons out of the page header's
right-side action slot. On Teams the button now sits in the tab bar's left
slot, separated from the three tabs by a vertical rule, so the CTA and tabs
read as one left-anchored cluster. On Keys, which has no tabs, the button
anchors left on its own row beneath the title.
Raise the constraint floors for two transitive dependencies so resolution moves them to their latest maintenance releases: httplib2 0.31.2 -> 0.32.0 and setuptools 82.0.1 -> 83.0.0. Both are pulled in only by optional integrations (Google API client, grpc tooling, lunary observability, the nvidia-riva extra), all lower-bound only, so the floors stay inside every requirer's allowed range and a default install is unaffected
discoverable_endpoints.py had grown to 2695 lines mixing FastAPI route handlers with the dcr_bridge token-flow logic, against the no-monster-files convention. This moves the bridge token flow (the litellm-key/user resolution, the SCIM revalidation gate, and the mint/refresh envelope logic with their types and error mappers) into a dedicated bridge_token_flow.py, leaving the route handlers and the shared exchange_token_with_server orchestrator in discoverable_endpoints.py importing from it
Pure relocation, zero behavior change. The moved code is byte-verbatim except one type annotation quoted as a forward reference (_BridgeAuthorizationCode is used only for typing and imported under TYPE_CHECKING to avoid a cycle), and the new module imports nothing from discoverable_endpoints at runtime. 275 tests pass unchanged; the test patch targets for moved internals were repointed to the new module and verified to still apply
* test(e2e): OTEL trace completeness on /v1/messages
Extends the LIT-3787 trace-completeness suite to the Anthropic-native route:
one successful non-streaming /v1/messages call must land at the destination as
ONE connected trace (root SERVER span + auth/db/cost children + gen-AI CLIENT
span, no dangling parents). Adds the raw /v1/messages sender to the logging
suite client.
* test(e2e): reuse the shared AnthropicMessagesBody per review
Drops the duplicate /v1/messages request model in favor of the one models.py
already provides (budget_client uses the same one), passes max_tokens at the
call site to match the sibling chat test, notes in the docstring why the
gen-AI span is named chat on this surface, and adopts the hardened read-back
signature
* test(e2e): author the messages trace test docstring
* test(e2e): declare the messages surface on the covers marker
* test(e2e): otel trace completeness on /v1/responses (#33134)
* test(e2e): OTEL trace completeness on /v1/responses
Extends the LIT-3787 trace-completeness suite to the OpenAI Responses API
route: one successful non-streaming /v1/responses call must land at the
destination as ONE connected trace. Adds the raw /v1/responses sender, a
CHEAP_OPENAI_MODEL config constant, and registers responses in the otel
registry cell's exercised_on.
* test(e2e): author the responses trace test docstring
* test(e2e): declare the responses and chat surfaces on the covers markers
get_group_ids_from_service_principal only read the first page of the
Graph API appRoleAssignedTo response, so tenants with more than 100
groups assigned to the enterprise application silently lost group
memberships during SSO login. Loop over @odata.nextLink with the same
MAX_GRAPH_API_PAGES cap that get_user_groups_from_graph_api already
uses, and warn when the cap is hit.
Ported from #32792 by @saisurya237 so CI can run.
Fixes#32790
Co-authored-by: saisurya237 <saisurya.abhishek237@gmail.com>
* test(e2e): OTEL trace completeness on /chat/completions against a local Jaeger destination
Adds the logging-suite infrastructure for LIT-3787 trace-completeness coverage:
a jaeger service in the compose stack as the OTEL v2 destination (arize_phoenix
preset pointed at it via PHOENIX_COLLECTOR_HTTP_ENDPOINT, so gen-AI spans export
through a preset-owned provider - the code path where trace splits happen), a
typed Jaeger query read-back client, and the first test: one successful
non-streaming /chat/completions call exports ONE complete trace (root SERVER
span + auth/db/cost children + gen-AI CLIENT span, no dangling parents).
* test(e2e): harden the otel trace read-back per review
Jaeger reads now query server-side by the litellm.call_id span tag instead of
paging recent traces and filtering client-side; the compose stack's background
jobs alone can push a request trace past the page. A failed query hard-fails
instead of reading as an empty result, the settle predicate now also waits for
the prefix-matched db span the assertion demands, parent-chain walking follows
CHILD_OF references only, the zero-trace and split-trace failures get distinct
messages, jaeger gets a healthcheck so the depends_on condition is accurate,
and the chat docstring names the route the code actually asserts
* test(e2e): author the chat trace test docstring
* Update logging section in CLAUDE.md
Removed mention of OTEL trace-tree completeness from logging integration section.
* feat(router): opt-in session affinity for complexity router
Complexity router reclassified every turn, which could flip the routed
model group mid-session and break provider-side prompt caching. Add a
session_affinity config flag: when a session_id is resolvable, pin the
model chosen on the first turn and reuse it for the rest of the session,
skipping reclassification. Pinned turns still stamp the adaptive
bandit's chosen-model metadata so reward feedback keeps working when
adaptive=True.
* fix(router): refresh session-affinity TTL on hit, scope pin by API key
Two issues from review: the TTL was only set on the first classification,
so an active session outliving session_affinity_ttl_seconds silently lost
its pin instead of refreshing as documented. And the cache key was scoped
only by session_id, which is client-supplied and unauthenticated, so two
different callers reusing the same session_id could poison each other's
routing pin. Refresh the TTL on every cache hit, and namespace the cache
key by the proxy-derived API key hash when available.
* fix(ui): derive key model scope so SCIM/management/read-only keys stop showing 'All Proxy Models'
key_type is not persisted on a key (the proxy maps it to allowed_routes and
drops it), so the keys tables only inspected the models list and rendered
'All Proxy Models' for any key with an empty models array, including SCIM,
Management and Read-only keys that cannot call a single model.
Add deriveKeyModelScope(allowed_routes) and render 'No model access' with a
scope tooltip for those recognized scopes; unrestricted, AI-API and custom
keys keep the existing model-list rendering.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): move key_scope helper to components root
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(keys): persist key_type on virtual keys so the UI reads scope directly
Add a nullable key_type column to LiteLLM_VerificationToken (root, proxy,
and proxy-extras schemas plus an additive migration) and stop dropping the
value in handle_key_type, so management/read_only/llm_api/default keys store
their type alongside the derived allowed_routes. Surface it on the key read
and create response models. The dashboard now prefers the persisted key_type
for the no-inference buckets and keeps the allowed_routes derivation as the
fallback for keys created before the column existed (key_type null), so no
backfill is required.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(keys): use PEP604 X | None for new key_type annotations to satisfy ruff UP045 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(keys): add key_type column to LiteLLM_DeletedVerificationToken
The deleted-token archive model inherits key_type from the verification
token, so regenerate/delete flows write key_type into
LiteLLM_DeletedVerificationToken. Add the column (all schemas + migration)
so the archive insert does not fail with FieldNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(migrations): regenerate key_type migration via runbook (canonical ADD COLUMN)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
