* adding deepkeep as custom guardrail
* adding deepkeep as a custom guardrail
* adding deepkeep as a custom guardrail (hooks)
* adding litellm/proxy/_experimental/out/ to .gitignore
* adding deepkeep as custom guardrail in litellm
* removing sentinel_fortress
* comparing schema.prisma files
* fix(deepkeep): address greptile review comments
- extra_headers: fix type annotation (list -> Dict[str, str]) and actually
merge them into _build_request_headers() so user-configured headers
reach the DeepKeep API
- user_api_key_hash: only fall back to user_api_key_token when no
explicit hash is already set, avoiding silent overwrite
- apply_guardrail: preserve tool_calls and structured_messages in the
return value so downstream callers don't lose that content
Adds tests for all four fixes.
* fix(deepkeep): address greptile review comments
- extra_headers: fix type annotation (list -> Dict[str, str]) and actually
merge them into _build_request_headers() so user-configured headers
reach the DeepKeep API
- user_api_key_hash: only fall back to user_api_key_token when no
explicit hash is already set, avoiding silent overwrite
- apply_guardrail: preserve tool_calls and structured_messages in the
return value so downstream callers don't lose that content
Adds tests for all four fixes.
* fix: add missing __init__.py and allowlist entries for upstream merge
- tests/test_litellm/proxy/client/__init__.py: fixes pytest collection
collision with tests/test_litellm/models/test_models.py (same basename)
- tests/test_litellm/models/__init__.py: same fix
- backend/routes/allowlist.py: add /config_overrides/ and /v1/unified_access_group
prefixes for new routes added by upstream
* fix(ui/tests): resolve frontend-lint failures in new test files
- useLogDetails.test.ts: add Wrapper.displayName, replace 'null as any'
with null, type resolveCall promise resolver properly
- usePaginatedDailyActivity.test.ts: remove unused waitFor import,
add Wrapper.displayName, change Record<string,any> to Record<string,unknown>
- UsageViewSelect.adminFiltering.test.tsx: replace all props:any with
explicit SelectProps/BadgeProps/SelectOption types, replace (X as any).displayName
with direct X.displayName assignment
no-explicit-any count: 2034 (budget: 2040). Prettier check: clean.
* fix(ui): sync proxy/_experimental/out/ exactly to upstream
245 stale JS chunk files from earlier merges were left in the out/
directory but had been deleted in upstream. The Docker image in CI is
built by copying this directory verbatim, so the stale artifacts caused
the SERVER_ROOT_PATH redirect E2E to fail.
Synced by: git checkout upstream/litellm_internal_staging -- out/ (adds
new files) + git rm on every file present in HEAD but absent from
upstream.
* Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(makefile): fall back to upstream/litellm_internal_staging for strict-budget gate
origin/litellm_internal_staging exists on BerriAI's CI but not on forks
that use a different remote name (e.g. Azure DevOps as origin). Fall
back to upstream/litellm_internal_staging when the origin ref is absent.
* linter reformat
* fix(deepkeep): apply guardrail tool/tool_call redactions from API response
When DeepKeep returns GUARDRAIL_INTERVENED with redacted tools or
tool_calls, the previous code ignored those redactions and forwarded
the original (potentially sensitive) values to the model — a guardrail
bypass for content embedded in tool schemas or function arguments.
Fix: prefer response_json["tools"] / response_json["tool_calls"] when
present, falling back to the originals only when the guardrail did not
return replacements — consistent with the existing pattern for texts and
images.
Refactor _build_return_inputs() into a private static helper to keep
apply_guardrail() under the PLR0915 statement limit (50).
Adds test_apply_guardrail_applies_tool_redactions_from_response to
assert that redacted tool payloads from the API response are used.
* Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(lint): move base-ref fallback into ruff_strict_gate.py; revert Makefile
The previous Makefile fix had a shell bug: 'git rev-parse --verify'
writes the resolved SHA to stdout, so the $$(...) substitution captured
both the SHA and the echo output, handing '--base <sha>\norigin/...' as
two tokens to the Python script, causing exit code 1 in CI.
Fix: revert Makefile to its original single-line invocation and add
_resolve_base() to ruff_strict_gate.py. The function checks whether the
requested ref resolves; if not, it tries the 'upstream/' equivalent
before falling back to the original ref (letting git emit a clear error).
Behaviour in BerriAI CI: origin/litellm_internal_staging resolves → used
as before, no change.
Behaviour on forks with a different 'origin': falls back to
upstream/litellm_internal_staging transparently.
* fix(lint): fix UP006/UP045/F401 in changed files; add depth guard to check_any_discipline
- Replace Dict/List/Optional/Tuple typing imports with built-in equivalents
(UP006, UP045) across files touched in this PR diff, then clean up
the now-unused typing imports (F401).
- Add _MAX_CONTAINS_ANY_DEPTH guard to check_any_discipline.contains_any()
to prevent RecursionError on deeply-nested mypy types.
* fix(lint): resolve all three CI lint job failures
1. lint (ruff_strict_gate) — UP006/UP045/F401 violations introduced on
changed lines. Fixed Dict/List/Optional/Tuple → built-in equivalents
across every file in the PR diff; cleaned up now-unused typing imports.
2. any-discipline — RecursionError in check_any_discipline.contains_any()
on deeply-nested mypy types. Upstream fixed this by converting to an
iterative stack-based algorithm (merged). Also added deepkeep.py to
any-discipline-budget.json via 'make lint-any-budget-update' so the
new file's Any count is baselined instead of failing against the
zero-baseline default.
3. basedpyright reportMissingParameterType — **kwargs in DeepKeepGuardrail
__init__ lacked a type annotation. Added **kwargs: Any.
* Update litellm/deepkeep_tilt_config.yaml
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix(lint): black reformat after merge
* fix(deepkeep): honour empty-list replacements in _build_return_inputs
When DeepKeep returns GUARDRAIL_INTERVENED with an intentional empty
replacement (e.g. texts:[], tool_calls:[]) the previous truthiness check
treated [] as absent and forwarded the original content downstream —
a guardrail bypass for any case where the firewall wants to fully clear
a field.
Fix: replace all response_json.get(field) truthiness checks with
'is not None' comparisons so that an empty list is respected as a
deliberate replacement. Applies to texts, images, tools, tool_calls,
and the original-input fallback guards.
Adds test_apply_guardrail_honours_empty_list_replacements.
* fix(test): replace live httpbin.org call with mocked transport in test_pass_through_with_httpbin_redirect
Root cause of OOM: the test made a real HTTP request to https://httpbin.org
inside a pytest-xdist worker. Under memory pressure the worker's httpx client
and redirect-following logic allocated enough virtual memory to trip the OOM
killer (confirmed by ulimit -v 16GB reproducing the crash with 'node down: Not
properly terminated' on this exact test).
Fix: replace the real network call with a custom httpx.AsyncBaseTransport that
returns a pre-built 302 -> 200 response sequence in-memory. The test now runs
hermetically with no network dependency and no excess memory allocation.
ulimit -v 16GB: 24,284 passed (0 crashes) after this fix.
* fix: merge upstream/litellm_internal_staging (197 commits), resolve conflicts
7 conflicts resolved:
- 6 Python files: upstream added new code with old-style typing (Optional,
Dict, List) on lines where we had ruff-fixed modern syntax (str | None,
dict, list). Took upstream's version then re-ran ruff UP006/UP045/F401
--fix to keep both the new content and ruff compliance.
- test_openapi_compliance.py: upstream replaced 'role' with 'steps' in
output_fields and updated the spec comment. Took upstream's version.
Also: added _resolve_base() fallback to type_check_gate.py and removed
the hard 'git fetch origin litellm_internal_staging' from the Makefile's
lint-basedpyright target (same pattern as ruff_strict_gate.py fix).
* fix: merge upstream (41 commits), resolve .gitignore conflict, fix BLE001
- .gitignore: upstream removed package.json/out/ ignore entries; took theirs
- deepkeep.py: added '# noqa: BLE001' on catch-all Exception handler
(BLE001 rule newly enforced in ruff-strict-budget)
- type_check_gate.py: added _resolve_base() fallback for basedpyright gate
- Makefile: removed hard 'git fetch origin' from lint-basedpyright target
* fix: merge upstream (57 commits), resolve conflicts
- Makefile: upstream added lint-fetch-base target; made it tolerant of
missing origin/litellm_internal_staging (git fetch || true)
- test_websearch_chat_completion.py: took upstream's new assertions and
skipif marker
- anthropic_cache_control_hook.py: upstream added new code using List/Dict/Tuple
which were undefined after our earlier UP006 cleanup; replaced with
built-in list/dict/tuple
* fix(coverage): revert ruff UP006/UP045 changes on upstream files
The previous ruff fixes (Dict→dict, Optional→X|None) on 7 upstream files
added ~500 changed lines of pure type-annotation no-ops to our PR diff.
codecov/patch penalised these uncovered lines, dropping patch coverage
to 51.35% (target 61.83%).
Fix: revert these files to exactly match upstream/litellm_internal_staging.
The ruff_strict_gate still passes because the violations exist equally in
both the base and HEAD (total == base_count → no breach).
* fix: merge upstream (130 commits), resolve Makefile + base_email conflicts
- Makefile: upstream changed lint deps to $(LINT_DEP_INSTALL)/$(LINT_DEP_BASE);
kept our --base removal (handled by _resolve_base in Python scripts)
- base_email.py: took upstream's dedup cache addition
- deepkeep.py: ruff format after merge
* chore: remove lint/format-only changes and non-feature files
Revert all lint-infra and black/ruff-reformat-only changes back to
upstream/litellm_internal_staging so the PR diff shows only the DeepKeep
guardrail feature:
- Makefile, scripts/ruff_strict_gate.py, scripts/type_check_gate.py
(lint-gate infra)
- credential_migration.py + enterprise/* + assorted test files
(black-reformat / xdist test-isolation drift)
- backend/routes/allowlist.py (merge glue)
Remove non-feature local artifacts: build-and-push.sh,
deepkeep_tilt_config.yaml, stray __init__.py collision shims, and
unrelated UI test files.
* fix(lint): add reason to BLE001 noqa to satisfy type-discipline gate (LIT003)
The type-discipline budget ratcheted LIT003's ceiling to 292 as upstream
fixed reasonless suppressions, so our '# noqa: BLE001' (code but no
reason) tipped the total to 293 and failed CI. Add a reason per the
required '# noqa: CODE # <reason>' shape.
* fix(deepkeep): apply structured_messages redactions returned by the guardrail API
_build_return_inputs dropped any structured_messages the DeepKeep API returned and
always forwarded the original input, so redactions on that field never took effect.
Check the response first, same as texts/images/tools/tool_calls
* chore(ui): drop redundant preserve prop from the guardrail form
preserve defaults to true in rc-field-form (isMergedPreserve falls back to true when
unset), so the explicit prop changed nothing and only widened this PR's blast radius
to every guardrail provider in the shared form
* fix(deepkeep): stop extra_headers list from crashing the guardrail call and name the real firewall id config key
litellm_params.extra_headers is a list of header names to forward, so passing it
straight into dict.update raised ValueError and, under fail_closed, took the request
down with it. Only merge mapping values and warn otherwise
The docstring example and the missing-secret error both said firewall_id, but
initialize_guardrail only reads deepkeep_firewall_id, so anyone following them
had their value silently ignored
* refactor(proxy): drop normalize_callback change; split to its own PR (#33905)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yaniv Israel <yaniv@deepkeep.ai>
Co-authored-by: DK-yaniv <164404355+DK-yaniv@users.noreply.github.com>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The MCP OAuth DCR relay's client-facing /register response handed back LiteLLM's own
<base>/callback as the client's redirect_uris on every arm except the true bridge relay.
A spec-compliant OAuth 2.1 DCR client (e.g. Open WebUI over Streamable HTTP) adopts that
value for its subsequent /authorize calls, so /callback redirects to itself; the second
hit carries the client's opaque state, fails to decrypt, and surfaces as the LIT-4197
"oauth_state ... Incorrect padding" error, making pass-through MCP OAuth unusable against
real DCR-capable upstreams.
The short-circuit arm keeps registering the gateway callback upstream (unchanged) and now
echoes the client's own redirect_uris back to the client across all register arms. Since
the client then authorizes with its own separate-origin redirect, the /authorize rejection
hint now points operators to MCP_TRUSTED_REDIRECT_ORIGINS, the mechanism a legitimate
cross-origin OAuth client needs (auto-trusting a DCR-registered redirect would reintroduce
the VERIA-57 open-redirect vector, since dynamic registration is unauthenticated).
temp_budget_increase was only applied on the DB-fetch path of _user_api_key_auth_builder, so a key served from the auth cache reverted to its original max_budget and was wrongly blocked with BudgetExceededError once spend crossed the original budget while staying under the effective budget.
Move _update_key_budget_with_temp_budget_increase out of the DB-only branch so it runs for every resolved token regardless of source. The cache stores the original budget and each cache hit returns a fresh model_copy(), so this never double-applies.
Fixes#25760
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The JWT first-login upsert in get_user_object creates the user row by
merging default_internal_user_params straight into table.create, so a
configured budget_duration landed with budget_reset_at NULL. The reset
sweep now heals such rows (PR #33623), but until the next sweep the row
shows a null reset time and its first window starts at the sweep instead
of one full duration after creation. Compute budget_reset_at at creation
like every other write path (/user/new, UI SSO, /key/generate, /team/new)
already does
Continues #29762. Response models (Message, Choices, Usage) delete unset
optional fields in __init__ so model_dump matches the OpenAI spec. Each delete
routed through pydantic's BaseModel.__delattr__, whose per-call
ModelMetaclass.__getattr__ lookup and _check_frozen dominate construction. When
the target is a declared field already present in __dict__ on a non-frozen
model, delete it with object.__delattr__ directly; that is exactly what pydantic
2.13 does for that case, minus the metaclass getattr and the frozen check. It
falls back to the previous super().__delattr__ path for extras, private
attributes, cached properties and missing names, so behavior is unchanged.
Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
Internal users seeded from default_internal_user_params (SSO/JWT first-login
upsert, or /user/new without an explicit budget_reset_at) get budget_duration
set but budget_reset_at = NULL. The ResetBudgetJob user/team queries filter on
{"budget_reset_at": {"lt": now}}, which never matches NULL, so these rows are
never reset: their spend accumulates for the lifetime of the row and silently
exceeds max_budget with no periodic reset.
The budget-table query already handles this by OR-ing in a
{budget_reset_at IS NULL AND budget_duration IS NOT NULL} branch. Apply the
same pattern to the user and team reset queries in PrismaClient.get_data.
Adds a regression test asserting both the user and team reset queries select
NULL-budget_reset_at rows that have a budget_duration.
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Ran the before/after proof live against Vertex Claude (global endpoint,
project vertex-check-481318): base transform 400s an unflagged model
(claude-opus-4-7) on a mid-conversation role:system reminder, the fix
hoists it to a 200, and a flagged model (claude-opus-4-8) keeps the
reminder in messages with cache_read held at 15615 across the reminder
turn. Flip both vertex.mid_conversation_system rows to fail_before_fix:
proven.
Azure AI Foundry and Vertex AI serve Claude on the first-party Anthropic
Messages contract, which was verified live to be byte-identical to
api.anthropic.com: a leading role:"system" entry in messages is rejected on
every model ("messages.0: use the top-level 'system' parameter"), and a
mid-conversation role:"system" reminder is accepted in place on Claude 4.8+/5
but 400s on Claude 4.7 and older ("role 'system' is not supported on this
model"). This is the same contract Bedrock Invoke already handles model-aware
(PRs #32578/#32831/#32882); Vertex and Azure did no hoisting at all, so a Claude
Code session on an older Vertex/Azure Claude model hard-400s on its reminder
turns, and the only thing sparing 4.8+/5 was that nothing was hoisted
Extract Bedrock's model-gated normalization into the shared
AnthropicMessagesConfig base as _normalize_system_role_messages and call it from
the Vertex and Azure messages configs. Flagged models (4.8+/5) hoist only the
leading run of system entries and keep mid-conversation reminders in place so
the top-level system prefix stays byte-identical and the prompt cache is
preserved; unflagged models hoist every system entry so the request returns a
completion instead of a 400
Add supports_mid_conversation_system to the azure_ai and vertex_ai Claude 4.8+/5
cost-map entries. Exact cost-map hits win over the claude-mid-conversation-system
fallback rule, so without the explicit flag those models would be treated as
unsupported and hoist every reminder, collapsing the prompt cache (the exact
customer regression). A per-provider test guards this so future 4.8+/5 entries
cannot silently miss the flag
Closes the Vertex/Azure gap from the customer RCA
Continues #29761. Delta.__init__ set roughly ten attributes through pydantic's
__setattr__ and then deleted the five OpenAI omits on every chunk. Those keys
are extra fields (extra='allow'), so this builds __pydantic_extra__ and
__pydantic_fields_set__ directly after the parent init instead of round-tripping
each field through __setattr__/__delattr__. The resulting __dict__,
__pydantic_extra__, __pydantic_fields_set__ and model_dump output (including
exclude_unset, which the streaming path relies on) are byte-identical to the
previous behavior; a serialization-contract test locks that. A TYPE_CHECKING
block re-declares the extra attributes with their concrete types so type
checkers still see delta.content and friends.
Co-authored-by: Jay Gowdy <jgowdy@godaddy.com>
Staging now contains #33153, whose final rounds made _extract_upstream_auth_failure a thin delegate
to upstream_auth_challenge and introduced the response-level iterator this branch predates. The
resolution completes the consolidation both branches were converging on: iter_exception_tree
(faults/traversal.py) is the one tree walk, _iter_upstream_responses is rebuilt on top of it instead
of carrying a second copy of the traversal, the manager keeps the delegate, and the semantic filter
port from this branch stands. Test conflicts were append-append and both sides are kept
* test(e2e): cover passthrough headers, batch assume-role, gemini, vllm, bedrock guardrails, batch rate-limit mapping
Add parent-package e2e suites for the six feature gaps: pass-through header forwarding via /config/pass_through_endpoint, Bedrock batch STS assume-role, Gemini chat + files, hosted_vllm batch/files, Bedrock guardrail pre_call blocks (plus restored content-filter team opt-out), and OpenAI batch RPM 429 body mapping. Registry cells and LiteLLMParamsBody/TeamMetadata fields updated so markers collect cleanly.
* test(e2e): cover LIT-4587 gaps for redis, responses, tpm cache, apply_guardrail, langfuse
Adds customer-shaped live e2e for apply_guardrail, responses store+metadata TTL,
TPM excluding cached tokens, redis-backed RPM, redis circuit-breaker path,
Langfuse spend, Cohere chat, virtual-key auth, file content download, hosted_vllm
chat, and Nova Sonic realtime. Registry cells updated for the new markers.
* test(e2e): drive LIT-4587 gap suites on Anthropic to avoid Gemini quota flakes
Redis RPM, circuit-breaker path, virtual-key auth, responses metadata, and
Langfuse driver models now use Anthropic haiku so local runs stay green when
Gemini daily quota is exhausted.
* test(e2e): drop Langfuse spend suite; feature is being deprecated
Remove test_langfuse_e2e.py, logging.langfuse registry cells, and the
langfuse-only conftest driver/credentials fixtures.
* test(e2e): fold provider/batch feature tests into their endpoint suites
Keep the e2e layout endpoint- and suite-scoped instead of one file per
provider or feature
Move the virtual-key auth case into access_control/test_access_control_e2e.py
as TestVirtualKeyAuth (replacing an incomplete stub) and drop the standalone
test_virtual_key_auth_e2e.py
Fold the five per-file batch suites (file content, RPM 429 mapping, Bedrock
assume-role, Gemini files, hosted_vllm batch) into batches/test_batches_e2e.py.
The hosted_vllm batch case is skipped for now since it needs a live vLLM server
(HOSTED_VLLM_API_BASE) the e2e environment does not provision; it and the
gemini-files and RPM-mapping cases reference LIT-3382 / LIT-3266 where relevant
Merge the cohere, gemini and hosted_vllm chat cases into
llm_translation/test_chat_completions_regression_e2e.py so /chat/completions
coverage lives in one endpoint file, and repoint the coverage_registry source
fields to the new homes
Move the shared CacheControl / TextBlock / RichMessage request blocks into the
root models.py (re-exported from endpoints_client) so quota_management can use
them without a cross-suite import, which also clears the basedpyright errors in
test_tpm_excludes_cached_tokens_e2e.py; type the httpbin echo body in
test_passthrough_headers_e2e.py with a pydantic model to drop the Any-typed
json.loads path
* test(e2e): address review feedback and re-home virtual-key coverage
Replace the tautological Bedrock assume-role batch id assertion (`startswith(...)
or batch.id`, always true) with a managed-id shape check, since the unified
target_model_names path re-encodes the id rather than returning a raw ARN
Raise the batch RPM-mapping test's rpm_limit above one so the file upload can no
longer consume the key's sole request unit before batch create runs; the batch
create then clears the generic per-request limiter and the batch limiter is what
returns the "Batch rate limit exceeded" body the assertions check
Set exercised_on to [] on the pass-through header test; it drives a pass-through
endpoint, not /chat/completions
Move the virtual-key valid_allows / invalid_denied cells from other.yaml to
mgmt.yaml as mgmt.virtual_key.* so TestVirtualKeyAuth rolls up under Management,
and point its covers marker at the new ids
* test(e2e): add reliability suite covering fallback, timeout, and cache behavior
* test(e2e): move reliability suite under router and drive it with real deployments
* test(e2e): make the router complexity fixture opt-in so reliability tests can coexist
* fix(cache): make in-memory and disk increments atomic
* refactor(cache): narrow in-memory increment lock scope
* fix(cache): address follow-up review on increment tests/types
* fix(cache): refresh atomic increment coverage
* test(cache): widen increment race window with non-zero _SlowInt seed
The zero seed was falsy, so InMemoryCache.increment_cache's `get_cache(...) or 0`
and DiskCache.get_cache's truthiness guard both discarded the _SlowInt before
__add__ could run, leaving the sleep-based window-widening inert. Seed a non-zero
value and return _SlowInt from __add__ so the sleep fires on every read-modify-write
in both backends, making the concurrency regression deterministic.
* test(cache): cover InMemoryCache.async_increment delegation
Add a focused async test asserting async_increment accumulates through the
locked sync path, exercising the previously uncovered delegation line.
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
It is a non-binary latency SLO threshold rather than a deterministic pass/fail
behavior a single e2e test can assert, so it does not fit the coverage registry's
one-test-per-cell contract. The registry README already flagged the perf cells for
a support-check or prune, and throughput SLO under load is covered structurally by
the Locust load suite. Removing it keeps the denominator to behaviors an e2e test
can deterministically prove.
Caller-supplied bedrock_tags land as AWS resource tags under the proxy's
AWS identity, letting an authenticated caller forge ownership or
cost-allocation labels. Add bedrock_tags to _BANNED_REQUEST_BODY_PARAMS
so per-request tags need general_settings.allow_client_side_credentials
or configurable_clientside_auth_params on the deployment, matching the
aws_bedrock_project_id precedent. Deployment-level bedrock_tags in
litellm_params are unaffected.
Also stop an explicit empty bedrock_tags list in litellm_params from
falling through to optional_params
