* adding deepkeep as custom guardrail
* adding deepkeep as a custom guardrail
* adding deepkeep as a custom guardrail (hooks)
* adding litellm/proxy/_experimental/out/ to .gitignore
* adding deepkeep as custom guardrail in litellm
* removing sentinel_fortress
* comparing schema.prisma files
* fix(deepkeep): address greptile review comments
- extra_headers: fix type annotation (list -> Dict[str, str]) and actually
merge them into _build_request_headers() so user-configured headers
reach the DeepKeep API
- user_api_key_hash: only fall back to user_api_key_token when no
explicit hash is already set, avoiding silent overwrite
- apply_guardrail: preserve tool_calls and structured_messages in the
return value so downstream callers don't lose that content
Adds tests for all four fixes.
* fix(deepkeep): address greptile review comments
- extra_headers: fix type annotation (list -> Dict[str, str]) and actually
merge them into _build_request_headers() so user-configured headers
reach the DeepKeep API
- user_api_key_hash: only fall back to user_api_key_token when no
explicit hash is already set, avoiding silent overwrite
- apply_guardrail: preserve tool_calls and structured_messages in the
return value so downstream callers don't lose that content
Adds tests for all four fixes.
* fix: add missing __init__.py and allowlist entries for upstream merge
- tests/test_litellm/proxy/client/__init__.py: fixes pytest collection
collision with tests/test_litellm/models/test_models.py (same basename)
- tests/test_litellm/models/__init__.py: same fix
- backend/routes/allowlist.py: add /config_overrides/ and /v1/unified_access_group
prefixes for new routes added by upstream
* fix(ui/tests): resolve frontend-lint failures in new test files
- useLogDetails.test.ts: add Wrapper.displayName, replace 'null as any'
with null, type resolveCall promise resolver properly
- usePaginatedDailyActivity.test.ts: remove unused waitFor import,
add Wrapper.displayName, change Record<string,any> to Record<string,unknown>
- UsageViewSelect.adminFiltering.test.tsx: replace all props:any with
explicit SelectProps/BadgeProps/SelectOption types, replace (X as any).displayName
with direct X.displayName assignment
no-explicit-any count: 2034 (budget: 2040). Prettier check: clean.
* fix(ui): sync proxy/_experimental/out/ exactly to upstream
245 stale JS chunk files from earlier merges were left in the out/
directory but had been deleted in upstream. The Docker image in CI is
built by copying this directory verbatim, so the stale artifacts caused
the SERVER_ROOT_PATH redirect E2E to fail.
Synced by: git checkout upstream/litellm_internal_staging -- out/ (adds
new files) + git rm on every file present in HEAD but absent from
upstream.
* Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(makefile): fall back to upstream/litellm_internal_staging for strict-budget gate
origin/litellm_internal_staging exists on BerriAI's CI but not on forks
that use a different remote name (e.g. Azure DevOps as origin). Fall
back to upstream/litellm_internal_staging when the origin ref is absent.
* linter reformat
* fix(deepkeep): apply guardrail tool/tool_call redactions from API response
When DeepKeep returns GUARDRAIL_INTERVENED with redacted tools or
tool_calls, the previous code ignored those redactions and forwarded
the original (potentially sensitive) values to the model — a guardrail
bypass for content embedded in tool schemas or function arguments.
Fix: prefer response_json["tools"] / response_json["tool_calls"] when
present, falling back to the originals only when the guardrail did not
return replacements — consistent with the existing pattern for texts and
images.
Refactor _build_return_inputs() into a private static helper to keep
apply_guardrail() under the PLR0915 statement limit (50).
Adds test_apply_guardrail_applies_tool_redactions_from_response to
assert that redacted tool payloads from the API response are used.
* Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(lint): move base-ref fallback into ruff_strict_gate.py; revert Makefile
The previous Makefile fix had a shell bug: 'git rev-parse --verify'
writes the resolved SHA to stdout, so the $$(...) substitution captured
both the SHA and the echo output, handing '--base <sha>\norigin/...' as
two tokens to the Python script, causing exit code 1 in CI.
Fix: revert Makefile to its original single-line invocation and add
_resolve_base() to ruff_strict_gate.py. The function checks whether the
requested ref resolves; if not, it tries the 'upstream/' equivalent
before falling back to the original ref (letting git emit a clear error).
Behaviour in BerriAI CI: origin/litellm_internal_staging resolves → used
as before, no change.
Behaviour on forks with a different 'origin': falls back to
upstream/litellm_internal_staging transparently.
* fix(lint): fix UP006/UP045/F401 in changed files; add depth guard to check_any_discipline
- Replace Dict/List/Optional/Tuple typing imports with built-in equivalents
(UP006, UP045) across files touched in this PR diff, then clean up
the now-unused typing imports (F401).
- Add _MAX_CONTAINS_ANY_DEPTH guard to check_any_discipline.contains_any()
to prevent RecursionError on deeply-nested mypy types.
* fix(lint): resolve all three CI lint job failures
1. lint (ruff_strict_gate) — UP006/UP045/F401 violations introduced on
changed lines. Fixed Dict/List/Optional/Tuple → built-in equivalents
across every file in the PR diff; cleaned up now-unused typing imports.
2. any-discipline — RecursionError in check_any_discipline.contains_any()
on deeply-nested mypy types. Upstream fixed this by converting to an
iterative stack-based algorithm (merged). Also added deepkeep.py to
any-discipline-budget.json via 'make lint-any-budget-update' so the
new file's Any count is baselined instead of failing against the
zero-baseline default.
3. basedpyright reportMissingParameterType — **kwargs in DeepKeepGuardrail
__init__ lacked a type annotation. Added **kwargs: Any.
* Update litellm/deepkeep_tilt_config.yaml
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(lint): black reformat after merge
* fix(deepkeep): honour empty-list replacements in _build_return_inputs
When DeepKeep returns GUARDRAIL_INTERVENED with an intentional empty
replacement (e.g. texts:[], tool_calls:[]) the previous truthiness check
treated [] as absent and forwarded the original content downstream —
a guardrail bypass for any case where the firewall wants to fully clear
a field.
Fix: replace all response_json.get(field) truthiness checks with
'is not None' comparisons so that an empty list is respected as a
deliberate replacement. Applies to texts, images, tools, tool_calls,
and the original-input fallback guards.
Adds test_apply_guardrail_honours_empty_list_replacements.
* fix(test): replace live httpbin.org call with mocked transport in test_pass_through_with_httpbin_redirect
Root cause of OOM: the test made a real HTTP request to https://httpbin.org
inside a pytest-xdist worker. Under memory pressure the worker's httpx client
and redirect-following logic allocated enough virtual memory to trip the OOM
killer (confirmed by ulimit -v 16GB reproducing the crash with 'node down: Not
properly terminated' on this exact test).
Fix: replace the real network call with a custom httpx.AsyncBaseTransport that
returns a pre-built 302 -> 200 response sequence in-memory. The test now runs
hermetically with no network dependency and no excess memory allocation.
ulimit -v 16GB: 24,284 passed (0 crashes) after this fix.
* fix: merge upstream/litellm_internal_staging (197 commits), resolve conflicts
7 conflicts resolved:
- 6 Python files: upstream added new code with old-style typing (Optional,
Dict, List) on lines where we had ruff-fixed modern syntax (str | None,
dict, list). Took upstream's version then re-ran ruff UP006/UP045/F401
--fix to keep both the new content and ruff compliance.
- test_openapi_compliance.py: upstream replaced 'role' with 'steps' in
output_fields and updated the spec comment. Took upstream's version.
Also: added _resolve_base() fallback to type_check_gate.py and removed
the hard 'git fetch origin litellm_internal_staging' from the Makefile's
lint-basedpyright target (same pattern as ruff_strict_gate.py fix).
* fix: merge upstream (41 commits), resolve .gitignore conflict, fix BLE001
- .gitignore: upstream removed package.json/out/ ignore entries; took theirs
- deepkeep.py: added '# noqa: BLE001' on catch-all Exception handler
(BLE001 rule newly enforced in ruff-strict-budget)
- type_check_gate.py: added _resolve_base() fallback for basedpyright gate
- Makefile: removed hard 'git fetch origin' from lint-basedpyright target
* fix: merge upstream (57 commits), resolve conflicts
- Makefile: upstream added lint-fetch-base target; made it tolerant of
missing origin/litellm_internal_staging (git fetch || true)
- test_websearch_chat_completion.py: took upstream's new assertions and
skipif marker
- anthropic_cache_control_hook.py: upstream added new code using List/Dict/Tuple
which were undefined after our earlier UP006 cleanup; replaced with
built-in list/dict/tuple
* fix(coverage): revert ruff UP006/UP045 changes on upstream files
The previous ruff fixes (Dict→dict, Optional→X|None) on 7 upstream files
added ~500 changed lines of pure type-annotation no-ops to our PR diff.
codecov/patch penalised these uncovered lines, dropping patch coverage
to 51.35% (target 61.83%).
Fix: revert these files to exactly match upstream/litellm_internal_staging.
The ruff_strict_gate still passes because the violations exist equally in
both the base and HEAD (total == base_count → no breach).
* fix: merge upstream (130 commits), resolve Makefile + base_email conflicts
- Makefile: upstream changed lint deps to $(LINT_DEP_INSTALL)/$(LINT_DEP_BASE);
kept our --base removal (handled by _resolve_base in Python scripts)
- base_email.py: took upstream's dedup cache addition
- deepkeep.py: ruff format after merge
* chore: remove lint/format-only changes and non-feature files
Revert all lint-infra and black/ruff-reformat-only changes back to
upstream/litellm_internal_staging so the PR diff shows only the DeepKeep
guardrail feature:
- Makefile, scripts/ruff_strict_gate.py, scripts/type_check_gate.py
(lint-gate infra)
- credential_migration.py + enterprise/* + assorted test files
(black-reformat / xdist test-isolation drift)
- backend/routes/allowlist.py (merge glue)
Remove non-feature local artifacts: build-and-push.sh,
deepkeep_tilt_config.yaml, stray __init__.py collision shims, and
unrelated UI test files.
* fix(lint): add reason to BLE001 noqa to satisfy type-discipline gate (LIT003)
The type-discipline budget ratcheted LIT003's ceiling to 292 as upstream
fixed reasonless suppressions, so our '# noqa: BLE001' (code but no
reason) tipped the total to 293 and failed CI. Add a reason per the
required '# noqa: CODE # <reason>' shape.
* fix(deepkeep): apply structured_messages redactions returned by the guardrail API
_build_return_inputs dropped any structured_messages the DeepKeep API returned and
always forwarded the original input, so redactions on that field never took effect.
Check the response first, same as texts/images/tools/tool_calls
* chore(ui): drop redundant preserve prop from the guardrail form
preserve defaults to true in rc-field-form (isMergedPreserve falls back to true when
unset), so the explicit prop changed nothing and only widened this PR's blast radius
to every guardrail provider in the shared form
* fix(deepkeep): stop extra_headers list from crashing the guardrail call and name the real firewall id config key
litellm_params.extra_headers is a list of header names to forward, so passing it
straight into dict.update raised ValueError and, under fail_closed, took the request
down with it. Only merge mapping values and warn otherwise
The docstring example and the missing-secret error both said firewall_id, but
initialize_guardrail only reads deepkeep_firewall_id, so anyone following them
had their value silently ignored
* refactor(proxy): drop normalize_callback change; split to its own PR (#33905)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yaniv Israel <yaniv@deepkeep.ai>
Co-authored-by: DK-yaniv <164404355+DK-yaniv@users.noreply.github.com>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The MCP OAuth DCR relay's client-facing /register response handed back LiteLLM's own
<base>/callback as the client's redirect_uris on every arm except the true bridge relay.
A spec-compliant OAuth 2.1 DCR client (e.g. Open WebUI over Streamable HTTP) adopts that
value for its subsequent /authorize calls, so /callback redirects to itself; the second
hit carries the client's opaque state, fails to decrypt, and surfaces as the LIT-4197
"oauth_state ... Incorrect padding" error, making pass-through MCP OAuth unusable against
real DCR-capable upstreams.
The short-circuit arm keeps registering the gateway callback upstream (unchanged) and now
echoes the client's own redirect_uris back to the client across all register arms. Since
the client then authorizes with its own separate-origin redirect, the /authorize rejection
hint now points operators to MCP_TRUSTED_REDIRECT_ORIGINS, the mechanism a legitimate
cross-origin OAuth client needs (auto-trusting a DCR-registered redirect would reintroduce
the VERIA-57 open-redirect vector, since dynamic registration is unauthenticated).
temp_budget_increase was only applied on the DB-fetch path of _user_api_key_auth_builder, so a key served from the auth cache reverted to its original max_budget and was wrongly blocked with BudgetExceededError once spend crossed the original budget while staying under the effective budget.
Move _update_key_budget_with_temp_budget_increase out of the DB-only branch so it runs for every resolved token regardless of source. The cache stores the original budget and each cache hit returns a fresh model_copy(), so this never double-applies.
Fixes#25760
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The JWT first-login upsert in get_user_object creates the user row by
merging default_internal_user_params straight into table.create, so a
configured budget_duration landed with budget_reset_at NULL. The reset
sweep now heals such rows (PR #33623), but until the next sweep the row
shows a null reset time and its first window starts at the sweep instead
of one full duration after creation. Compute budget_reset_at at creation
like every other write path (/user/new, UI SSO, /key/generate, /team/new)
already does
Continues #29762. Response models (Message, Choices, Usage) delete unset
optional fields in __init__ so model_dump matches the OpenAI spec. Each delete
routed through pydantic's BaseModel.__delattr__, whose per-call
ModelMetaclass.__getattr__ lookup and _check_frozen dominate construction. When
the target is a declared field already present in __dict__ on a non-frozen
model, delete it with object.__delattr__ directly; that is exactly what pydantic
2.13 does for that case, minus the metaclass getattr and the frozen check. It
falls back to the previous super().__delattr__ path for extras, private
attributes, cached properties and missing names, so behavior is unchanged.
Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
Internal users seeded from default_internal_user_params (SSO/JWT first-login
upsert, or /user/new without an explicit budget_reset_at) get budget_duration
set but budget_reset_at = NULL. The ResetBudgetJob user/team queries filter on
{"budget_reset_at": {"lt": now}}, which never matches NULL, so these rows are
never reset: their spend accumulates for the lifetime of the row and silently
exceeds max_budget with no periodic reset.
The budget-table query already handles this by OR-ing in a
{budget_reset_at IS NULL AND budget_duration IS NOT NULL} branch. Apply the
same pattern to the user and team reset queries in PrismaClient.get_data.
Adds a regression test asserting both the user and team reset queries select
NULL-budget_reset_at rows that have a budget_duration.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
Continues #29761. Delta.__init__ set roughly ten attributes through pydantic's
__setattr__ and then deleted the five OpenAI omits on every chunk. Those keys
are extra fields (extra='allow'), so this builds __pydantic_extra__ and
__pydantic_fields_set__ directly after the parent init instead of round-tripping
each field through __setattr__/__delattr__. The resulting __dict__,
__pydantic_extra__, __pydantic_fields_set__ and model_dump output (including
exclude_unset, which the streaming path relies on) are byte-identical to the
previous behavior; a serialization-contract test locks that. A TYPE_CHECKING
block re-declares the extra attributes with their concrete types so type
checkers still see delta.content and friends.
Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
Staging now contains #33153, whose final rounds made _extract_upstream_auth_failure a thin delegate
to upstream_auth_challenge and introduced the response-level iterator this branch predates. The
resolution completes the consolidation both branches were converging on: iter_exception_tree
(faults/traversal.py) is the one tree walk, _iter_upstream_responses is rebuilt on top of it instead
of carrying a second copy of the traversal, the manager keeps the delegate, and the semantic filter
port from this branch stands. Test conflicts were append-append and both sides are kept
* fix(cache): make in-memory and disk increments atomic
* refactor(cache): narrow in-memory increment lock scope
* fix(cache): address follow-up review on increment tests/types
* fix(cache): refresh atomic increment coverage
* test(cache): widen increment race window with non-zero _SlowInt seed
The zero seed was falsy, so InMemoryCache.increment_cache's `get_cache(...) or 0`
and DiskCache.get_cache's truthiness guard both discarded the _SlowInt before
__add__ could run, leaving the sleep-based window-widening inert. Seed a non-zero
value and return _SlowInt from __add__ so the sleep fires on every read-modify-write
in both backends, making the concurrency regression deterministic.
* test(cache): cover InMemoryCache.async_increment delegation
Add a focused async test asserting async_increment accumulates through the
locked sync path, exercising the previously uncovered delegation line.
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Caller-supplied bedrock_tags land as AWS resource tags under the proxy's
AWS identity, letting an authenticated caller forge ownership or
cost-allocation labels. Add bedrock_tags to _BANNED_REQUEST_BODY_PARAMS
so per-request tags need general_settings.allow_client_side_credentials
or configurable_clientside_auth_params on the deployment, matching the
aws_bedrock_project_id precedent. Deployment-level bedrock_tags in
litellm_params are unaffected.
Also stop an explicit empty bedrock_tags list in litellm_params from
falling through to optional_params
The model_info / get_model_info_with_id endpoint tests drove refactored
endpoints with bare, unspec'd MagicMock routers and models. Because the
mocks were unspec'd, any attribute or method the (refactored) endpoints
newly read auto-materialized a child MagicMock, and whether that child
was reached depended on process-global state (premium_user, and the real
get_available_models_for_user chain reading litellm globals) that sibling
tests in the same xdist worker mutate. When reached, the MagicMock either
unpacked to empty (a, b = mock.method() -> 'not enough values to unpack
(expected 2, got 0)') or leaked into RouterModelInfo(**model_info) and
failed Pydantic str validation. Pass in isolation, fail under xdist.
The original TestModelInfoEndpoint failure (#33807 CI) was the same class
surfaced by merge skew: #33721 added a get_configured_token_limits unpack
to create_model_info_response, and CI's merge commit ran that against the
un-updated bare-mock test before the #33742 band-aid landed.
Fix (test-only, no product change):
- TestModelInfoEndpoint: mock the real seam (get_available_models_for_user),
configure the router methods the endpoint actually calls, return a real
Deployment, and drop the dead proxy_server.get_key_models/get_team_models/
get_complete_model_list patches the refactor had stranded.
- TestGetModelInfoWithIdBlocked: spec the model mock so unset enterprise
columns read as None instead of child MagicMocks.
- test_ProxyConfig_get_model_info_with_id_missing_model_id_raises: pin
premium_user so the asserted AttributeError no longer flips with the
ambient license global.
_load_persisted_master_key documents leniency on any unreadable prior
state but only caught parse errors; a permissions failure or non-UTF-8
bytes in config.yaml crashed configure instead of skipping the
carry-forward. Catch OSError and UnicodeDecodeError too and pin the
undecodable-file case with a regression test.
lite autoroute up minted a fresh master key and picked a fresh OS-ephemeral
port on every run, so any client configured against one session (an
already-open Claude Code session, a hand-configured script) broke on the
next run. The port is now a stable default (5483, overridable with --port)
that refuses loudly when busy or when 4000 is requested, since proxy_cli
silently rebinds a busy 4000 to a random port. The master key is minted
once, persisted in the generated config.yaml, reused by every later up,
and carried forward when configure regenerates the config.
_resolve_v2_auth dropped the resolved client_credentials auth when extra_headers
already carried Authorization (MCPJWTSigner, static_headers), so the upstream got
the injected header instead of the minted token and the one-shot 401 refetch was
lost. M2M now joins token_exchange and authorization_code in the authoritative
set; the conflicting header is dropped
The auth object is the httpx client's auth for the whole MCP session; after a
401 recovery it kept sending the rejected token first, burning a 401 round trip
and the single retry on every subsequent call
An expires_in of zero or below computes a ttl of 0; the entry could never be
served but still occupied a slot in the bounded backend, where it could evict
a live token. The mint still serves the current request and the next get
re-fetches under the per-server lock
Greptile P2s: the per-server lock dict now evicts its oldest entry past
max_locks so ephemeral server ids (REST tools preview) cannot grow it
unbounded, and the min-cache floor is capped at the token's actual lifetime so
an expires_in below the skew is never served past expiry
The graft test pinned the pre-migration contract (M2M defers to v1). Replaced
with two tests pinning the new one: a complete-config M2M server resolves via
the v2 arm into ClientCredentialsBearerAuth, and an incomplete-config server
fails closed with a 500 misconfigured naming the missing grant fields
Replaces the not_implemented stub with a live arm: ClientCredentialsTokenSource
mints and caches the M2M token (rotation-aware identity key, expires_in-driven
TTL, audience and token_endpoint_auth_method support) and
ClientCredentialsBearerAuth retries an upstream 401 exactly once with a freshly
minted token. to_server_spec owns oauth2_flow=client_credentials servers and
fails closed on incomplete grant config instead of connecting unauthenticated
