2 commits

Author	SHA1	Message	Date
yuneng-jiang	6a0d03914c	test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) ... * test: drop the cwd-relative sys.path.insert calls from the test suite TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape: sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The argument resolves against the working directory rather than the file, so from the repo root, where every job runs pytest, it inserts the directory two levels above the checkout. It has never pointed at litellm. The package is installed into the environment anyway, which is what actually makes the import work, and what the rule's message has said all along. Removing them leaves 1,634 imports of sys and os with no remaining reference, and those go too, except where another test module imports the name back out of the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a variable, which are a different question and are left alone. Collection is identical either way: 45,871 tests and the same 51 pre-existing collection errors before and after, and ruff reports no new undefined name. * test: drop the duplicate imports the sys.path sweep exposed to F811 * test(pre-call-utils): restore the os import the new bedrock tests need	2026-08-22 09:25:58 -07:00
yucheng-berri	9ad8698aab	feat: add deepkeep as custom guardrail (#33844) ... * adding deepkeep as custom guardrail * adding deepkeep as a custom guardrail * adding deepkeep as a custom guardrail (hooks) * adding litellm/proxy/_experimental/out/ to .gitignore * adding deepkeep as custom guardrail in litellm * removing sentinel_fortress * comparing schema.prisma files * fix(deepkeep): address greptile review comments - extra_headers: fix type annotation (list -> Dict[str, str]) and actually merge them into _build_request_headers() so user-configured headers reach the DeepKeep API - user_api_key_hash: only fall back to user_api_key_token when no explicit hash is already set, avoiding silent overwrite - apply_guardrail: preserve tool_calls and structured_messages in the return value so downstream callers don't lose that content Adds tests for all four fixes. * fix(deepkeep): address greptile review comments - extra_headers: fix type annotation (list -> Dict[str, str]) and actually merge them into _build_request_headers() so user-configured headers reach the DeepKeep API - user_api_key_hash: only fall back to user_api_key_token when no explicit hash is already set, avoiding silent overwrite - apply_guardrail: preserve tool_calls and structured_messages in the return value so downstream callers don't lose that content Adds tests for all four fixes. * fix: add missing __init__.py and allowlist entries for upstream merge - tests/test_litellm/proxy/client/__init__.py: fixes pytest collection collision with tests/test_litellm/models/test_models.py (same basename) - tests/test_litellm/models/__init__.py: same fix - backend/routes/allowlist.py: add /config_overrides/ and /v1/unified_access_group prefixes for new routes added by upstream * fix(ui/tests): resolve frontend-lint failures in new test files - useLogDetails.test.ts: add Wrapper.displayName, replace 'null as any' with null, type resolveCall promise resolver properly - usePaginatedDailyActivity.test.ts: remove unused waitFor import, add Wrapper.displayName, change Record<string,any> to Record<string,unknown> - UsageViewSelect.adminFiltering.test.tsx: replace all props:any with explicit SelectProps/BadgeProps/SelectOption types, replace (X as any).displayName with direct X.displayName assignment no-explicit-any count: 2034 (budget: 2040). Prettier check: clean. * fix(ui): sync proxy/_experimental/out/ exactly to upstream 245 stale JS chunk files from earlier merges were left in the out/ directory but had been deleted in upstream. The Docker image in CI is built by copying this directory verbatim, so the stale artifacts caused the SERVER_ROOT_PATH redirect E2E to fail. Synced by: git checkout upstream/litellm_internal_staging -- out/ (adds new files) + git rm on every file present in HEAD but absent from upstream. * Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix(makefile): fall back to upstream/litellm_internal_staging for strict-budget gate origin/litellm_internal_staging exists on BerriAI's CI but not on forks that use a different remote name (e.g. Azure DevOps as origin). Fall back to upstream/litellm_internal_staging when the origin ref is absent. * linter reformat * fix(deepkeep): apply guardrail tool/tool_call redactions from API response When DeepKeep returns GUARDRAIL_INTERVENED with redacted tools or tool_calls, the previous code ignored those redactions and forwarded the original (potentially sensitive) values to the model — a guardrail bypass for content embedded in tool schemas or function arguments. Fix: prefer response_json["tools"] / response_json["tool_calls"] when present, falling back to the originals only when the guardrail did not return replacements — consistent with the existing pattern for texts and images. Refactor _build_return_inputs() into a private static helper to keep apply_guardrail() under the PLR0915 statement limit (50). Adds test_apply_guardrail_applies_tool_redactions_from_response to assert that redacted tool payloads from the API response are used. * Update litellm/proxy/guardrails/guardrail_hooks/deepkeep/deepkeep.py Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix(lint): move base-ref fallback into ruff_strict_gate.py; revert Makefile The previous Makefile fix had a shell bug: 'git rev-parse --verify' writes the resolved SHA to stdout, so the $$(...) substitution captured both the SHA and the echo output, handing '--base <sha>\norigin/...' as two tokens to the Python script, causing exit code 1 in CI. Fix: revert Makefile to its original single-line invocation and add _resolve_base() to ruff_strict_gate.py. The function checks whether the requested ref resolves; if not, it tries the 'upstream/' equivalent before falling back to the original ref (letting git emit a clear error). Behaviour in BerriAI CI: origin/litellm_internal_staging resolves → used as before, no change. Behaviour on forks with a different 'origin': falls back to upstream/litellm_internal_staging transparently. * fix(lint): fix UP006/UP045/F401 in changed files; add depth guard to check_any_discipline - Replace Dict/List/Optional/Tuple typing imports with built-in equivalents (UP006, UP045) across files touched in this PR diff, then clean up the now-unused typing imports (F401). - Add _MAX_CONTAINS_ANY_DEPTH guard to check_any_discipline.contains_any() to prevent RecursionError on deeply-nested mypy types. * fix(lint): resolve all three CI lint job failures 1. lint (ruff_strict_gate) — UP006/UP045/F401 violations introduced on changed lines. Fixed Dict/List/Optional/Tuple → built-in equivalents across every file in the PR diff; cleaned up now-unused typing imports. 2. any-discipline — RecursionError in check_any_discipline.contains_any() on deeply-nested mypy types. Upstream fixed this by converting to an iterative stack-based algorithm (merged). Also added deepkeep.py to any-discipline-budget.json via 'make lint-any-budget-update' so the new file's Any count is baselined instead of failing against the zero-baseline default. 3. basedpyright reportMissingParameterType — **kwargs in DeepKeepGuardrail __init__ lacked a type annotation. Added **kwargs: Any. * Update litellm/deepkeep_tilt_config.yaml Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> * fix(lint): black reformat after merge * fix(deepkeep): honour empty-list replacements in _build_return_inputs When DeepKeep returns GUARDRAIL_INTERVENED with an intentional empty replacement (e.g. texts:[], tool_calls:[]) the previous truthiness check treated [] as absent and forwarded the original content downstream — a guardrail bypass for any case where the firewall wants to fully clear a field. Fix: replace all response_json.get(field) truthiness checks with 'is not None' comparisons so that an empty list is respected as a deliberate replacement. Applies to texts, images, tools, tool_calls, and the original-input fallback guards. Adds test_apply_guardrail_honours_empty_list_replacements. * fix(test): replace live httpbin.org call with mocked transport in test_pass_through_with_httpbin_redirect Root cause of OOM: the test made a real HTTP request to https://httpbin.org inside a pytest-xdist worker. Under memory pressure the worker's httpx client and redirect-following logic allocated enough virtual memory to trip the OOM killer (confirmed by ulimit -v 16GB reproducing the crash with 'node down: Not properly terminated' on this exact test). Fix: replace the real network call with a custom httpx.AsyncBaseTransport that returns a pre-built 302 -> 200 response sequence in-memory. The test now runs hermetically with no network dependency and no excess memory allocation. ulimit -v 16GB: 24,284 passed (0 crashes) after this fix. * fix: merge upstream/litellm_internal_staging (197 commits), resolve conflicts 7 conflicts resolved: - 6 Python files: upstream added new code with old-style typing (Optional, Dict, List) on lines where we had ruff-fixed modern syntax (str | None, dict, list). Took upstream's version then re-ran ruff UP006/UP045/F401 --fix to keep both the new content and ruff compliance. - test_openapi_compliance.py: upstream replaced 'role' with 'steps' in output_fields and updated the spec comment. Took upstream's version. Also: added _resolve_base() fallback to type_check_gate.py and removed the hard 'git fetch origin litellm_internal_staging' from the Makefile's lint-basedpyright target (same pattern as ruff_strict_gate.py fix). * fix: merge upstream (41 commits), resolve .gitignore conflict, fix BLE001 - .gitignore: upstream removed package.json/out/ ignore entries; took theirs - deepkeep.py: added '# noqa: BLE001' on catch-all Exception handler (BLE001 rule newly enforced in ruff-strict-budget) - type_check_gate.py: added _resolve_base() fallback for basedpyright gate - Makefile: removed hard 'git fetch origin' from lint-basedpyright target * fix: merge upstream (57 commits), resolve conflicts - Makefile: upstream added lint-fetch-base target; made it tolerant of missing origin/litellm_internal_staging (git fetch || true) - test_websearch_chat_completion.py: took upstream's new assertions and skipif marker - anthropic_cache_control_hook.py: upstream added new code using List/Dict/Tuple which were undefined after our earlier UP006 cleanup; replaced with built-in list/dict/tuple * fix(coverage): revert ruff UP006/UP045 changes on upstream files The previous ruff fixes (Dict→dict, Optional→X|None) on 7 upstream files added ~500 changed lines of pure type-annotation no-ops to our PR diff. codecov/patch penalised these uncovered lines, dropping patch coverage to 51.35% (target 61.83%). Fix: revert these files to exactly match upstream/litellm_internal_staging. The ruff_strict_gate still passes because the violations exist equally in both the base and HEAD (total == base_count → no breach). * fix: merge upstream (130 commits), resolve Makefile + base_email conflicts - Makefile: upstream changed lint deps to $(LINT_DEP_INSTALL)/$(LINT_DEP_BASE); kept our --base removal (handled by _resolve_base in Python scripts) - base_email.py: took upstream's dedup cache addition - deepkeep.py: ruff format after merge * chore: remove lint/format-only changes and non-feature files Revert all lint-infra and black/ruff-reformat-only changes back to upstream/litellm_internal_staging so the PR diff shows only the DeepKeep guardrail feature: - Makefile, scripts/ruff_strict_gate.py, scripts/type_check_gate.py (lint-gate infra) - credential_migration.py + enterprise/* + assorted test files (black-reformat / xdist test-isolation drift) - backend/routes/allowlist.py (merge glue) Remove non-feature local artifacts: build-and-push.sh, deepkeep_tilt_config.yaml, stray __init__.py collision shims, and unrelated UI test files. * fix(lint): add reason to BLE001 noqa to satisfy type-discipline gate (LIT003) The type-discipline budget ratcheted LIT003's ceiling to 292 as upstream fixed reasonless suppressions, so our '# noqa: BLE001' (code but no reason) tipped the total to 293 and failed CI. Add a reason per the required '# noqa: CODE # <reason>' shape. * fix(deepkeep): apply structured_messages redactions returned by the guardrail API _build_return_inputs dropped any structured_messages the DeepKeep API returned and always forwarded the original input, so redactions on that field never took effect. Check the response first, same as texts/images/tools/tool_calls * chore(ui): drop redundant preserve prop from the guardrail form preserve defaults to true in rc-field-form (isMergedPreserve falls back to true when unset), so the explicit prop changed nothing and only widened this PR's blast radius to every guardrail provider in the shared form * fix(deepkeep): stop extra_headers list from crashing the guardrail call and name the real firewall id config key litellm_params.extra_headers is a list of header names to forward, so passing it straight into dict.update raised ValueError and, under fail_closed, took the request down with it. Only merge mapping values and warn otherwise The docstring example and the missing-secret error both said firewall_id, but initialize_guardrail only reads deepkeep_firewall_id, so anyone following them had their value silently ignored * refactor(proxy): drop normalize_callback change; split to its own PR (#33905) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yaniv Israel <yaniv@deepkeep.ai> Co-authored-by: DK-yaniv <164404355+DK-yaniv@users.noreply.github.com> Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-07-20 19:27:40 -07:00