The validation method now reads use_redis_transaction_buffer directly
from the passed general_settings dict rather than delegating to
RedisUpdateBuffer._should_commit_spend_updates_to_redis() which
imports the global. Tests simplified to remove unnecessary patching.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When `use_redis_transaction_buffer: true` is set in general_settings but no
Redis cache is configured in litellm_settings, the proxy starts successfully
but silently drops all spend tracking data. This adds a startup validation
that raises a clear error, preventing the proxy from running in a broken state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: Add Canadian PII protection patterns and PIPEDA-compliant policy template
Adds 6 new Canadian PII regex pattern detectors to patterns.json:
- ca_sin: Social Insurance Number (PIPEDA Privacy Act, Income Tax Act)
- ca_ohip: Ontario Health Insurance Plan Number (PHIPA)
- ca_on_drivers_licence: Ontario driver's licence (HTA, PIPEDA)
- ca_immigration_doc: IRCC immigration docs (UCI, work/study permits, IMM refs)
- ca_bank_account: Canadian bank account routing (transit-institution-account)
- ca_postal_code: Canadian postal code (Canada Post spec)
Adds comprehensive policy template 'canadian-pii-protection' (id: canadian-pii-protection)
with 5 sub-guardrails grouping patterns by data type. All patterns include contextual
keyword matching (English + French keywords where applicable) to reduce false positives.
Complements existing passport_canada pattern.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat: Add Canadian PII compliance dataset and tests (57 tests)
Adds:
- test_ca_patterns.py: 30 unit tests for regex pattern matching (SIN, OHIP,
driver's licence, immigration docs, bank account, postal code)
- test_ca_policy_e2e.py: 27 end-to-end tests running the full
ContentFilterGuardrail pipeline with MASK action — validates detection
of real PII and pass-through of clean prompts
- canadianPiiCompliancePrompts.ts: 21-prompt compliance dataset for UI
evaluation, wired into the main compliancePrompts framework
Fixes keyword_pattern alternation ordering in patterns.json — longer
alternatives (e.g. "social insurance number") now precede shorter ones
("social insurance") to avoid excessive gap-word count when the regex
engine selects the shorter match first.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat: Add University of Toronto FIPPA identifier patterns and tests (36 tests)
Add 3 UofT institutional identifiers (student/employee number, UTORid, TCard)
covered under Ontario FIPPA. Includes pattern definitions, policy template
sub-guardrail, compliance prompts, unit tests, and e2e tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Correct test assertion bug and inaccurate docstring
Fix test_utorid_masked checking `result` (dict) instead of `output` (string).
Update test_ca_policy_e2e.py docstring to clarify scope vs UofT tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Address Greptile review feedback
- Tighten ca_postal_code keyword_pattern: replace broad "address" with
specific compound terms (mailing/street/shipping/home address)
- Add missing "PIPEDA" tag to policy_templates.json for discoverability
- Add us_phone pattern to test_ca_policy_e2e.py setup to match deployed template
- Add phone number e2e test for complete coverage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Tighten patterns to reduce false positives and add missing test coverage
- ca_sin: reject leading-zero SINs ([1-9]\d{2}), set allow_word_numbers to false
- ca_immigration_doc: require separators in UCI pattern (prevent bare \d{10} match)
- uoft_utorid: qualify generic keywords (acorn -> acorn login, quercus -> quercus login)
- uoft_tcard: remove generic keywords (student card, id card, library card) that
overlap with credit card contexts; keep only UofT-specific terms (tcard, campus card)
- Add visa/mastercard/amex/iban patterns to test_ca_policy_e2e.py setup to match
deployed template; add Visa card masking test
- Add test verifying "student card" no longer triggers TCard redaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Internal users were blocked from accessing /key/aliases because the route
was missing from key_management_routes. Added the route and scoped query
results so non-admin users only see aliases for their own keys and their
teams' keys, matching /key/list behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use DeepSeekChatConfig instead of OpenAIConfig for deepseek provider
The deepseek provider was incorrectly using OpenAIConfig().map_openai_params()
instead of DeepSeekChatConfig().map_openai_params(), which meant DeepSeek-specific
parameter mappings were not being applied.
* test: add unit tests for deepseek DeepSeekChatConfig param mapping
Verify that get_optional_params uses DeepSeekChatConfig (not OpenAIConfig)
for the deepseek provider by testing thinking, reasoning_effort, and
budget_tokens stripping behavior.
* feat: add LITELLM_WORKER_STARTUP_HOOKS for per-worker initialization (gflags support)
Add support for running user-defined startup hooks in each worker process
during proxy_startup_event. This enables re-initialization of in-process
state (like gflags.FLAGS) that doesn't survive uvicorn worker spawning.
Usage:
export LITELLM_WORKER_STARTUP_HOOKS=mymodule:init_fn,other:setup_fn
Hooks run early in proxy_startup_event (before config/DB loading).
Supports both sync and async callables. Errors propagate to prevent
broken workers from serving traffic. No-op when env var is unset.
Includes 5 tests covering sync/async hooks, multiple hooks, error
propagation, and no-hooks-set scenarios.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add Worker Startup Hooks page with gflags usage example
- New docs page: docs/proxy/worker_startup_hooks.md
- Explains the problem (per-process state lost in multi-worker deployments)
- Full gflags example with wrapper module and startup script
- Covers multiple hooks, async hooks, error behavior
- Architecture diagram showing master→worker flow
- Added LITELLM_WORKER_STARTUP_HOOKS to config_settings.md env var table
- Added to sidebar under Setup & Deployment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mcp): resolve \$ref params and merge path-level params in OpenAPI tool registration
Real-world OpenAPI specs (e.g. GitHub's 11.8 MB official spec) use two
patterns that crashed tool registration:
1. \$ref parameters: params defined as {"$ref": "#/components/parameters/foo"}
instead of inline objects. Accessing param["name"] on a $ref raises KeyError.
Fix: resolve each param against components/parameters before processing.
2. Path-level parameters: params defined on the path object apply to all
HTTP methods on that path, but the operation object doesn't include them.
GitHub's spec uses this for owner/repo/etc. path params.
Fix: merge path-level params with operation-level params (op-level wins
when the same name+in combination appears in both).
With this fix the full GitHub REST API spec loads successfully:
720 paths → 1079 tools, all with correct parameter schemas.
* fix(mcp): resolve \$ref params in OpenAPI preview endpoint (test/tools/list)
The _preview_openapi_tools function (called by the UI add-server form to show
connection status and available tools) had the same bug as _register_openapi_tools:
it accessed param["name"] directly without resolving \$ref parameters or merging
path-level parameters from the path item.
This caused "Failed to load OpenAPI spec: 'name'" for any spec that uses
component-level parameter references (e.g. GitHub's official REST API spec).
Apply the same fix: resolve \$ref against components/parameters and merge
path-level params (with operation-level taking priority) before building schemas.
* refactor(openapi-mcp): extract resolve_operation_params, add tests
- Hoist _resolve_ref and _resolve_param_list to module level in
openapi_to_mcp_generator.py (were being redefined on every loop iteration)
- _resolve_ref now returns None for unresolvable $refs instead of
the stub dict, preventing (None, None) from poisoning deduplication
- Add resolve_operation_params() as a shared helper that handles both
$ref resolution and path-level param merging
- Replace duplicated inline logic in mcp_server_manager.py and
rest_endpoints.py with calls to resolve_operation_params()
- Add TestResolveRef, TestResolveParamList, TestResolveOperationParams
test classes covering $ref resolution, path-level merging, collision
semantics, unresolvable ref filtering, and a GitHub-style spec fixture
When Redis Cluster is configured via the REDIS_CLUSTER_NODES environment
variable, Cache.__init__() and Router._create_redis_cache() ignored the
env var and always created RedisCache instead of RedisClusterCache. This
caused the v3 rate limiter's cluster detection (_is_redis_cluster()) to
return False, skipping hash-slot key grouping. The resulting CROSSLOT
errors were silently caught, falling back to per-instance in-memory
counting — breaking RPM/TPM enforcement across multiple proxy instances.
Add REDIS_CLUSTER_NODES env var detection to both Cache.__init__() and
Router._create_redis_cache(), matching the existing pattern in
_redis.py:215-220. When the env var is set and no explicit startup_nodes
parameter is provided, parse it and create RedisClusterCache.
Fixes#22748
Related to #20836
Adds @ to the _KEY_ALIAS_PATTERN allowed character set so that
key aliases like user/user@example.com are accepted. Updates tests
to cover email-based alias formats.
Addresses 4 critical OpenTelemetry span issues in LiteLLM:
Issue #3: Remove redundant attributes from raw_gen_ai_request spans
- Removed self.set_attributes() call that was duplicating all parent span
attributes (gen_ai.*, metadata.*) onto the raw span
- Raw span now only contains provider-specific llm.{provider}.* attributes
- Reduces storage and eliminates search confusion from duplicate data
Issue #4: Prevent attribute duplication on litellm_proxy_request parent span
- When litellm_request child span exists, removed redundant
set_attributes() call on the parent proxy span
- Child span already carries all attributes; parent duplication doubles
storage and complicates search
Issue #5: Fix orphaned guardrail traces
- Guardrail spans were created with context=None when no parent proxy span
existed, resulting in orphaned root spans (separate trace_id)
- Added _resolve_guardrail_context() helper to ensure guardrails always
have a valid parent (litellm_request or proxy span)
- Applied fix to both _handle_success and _handle_failure paths
Issue #8: Add gen_ai.response.id for embeddings and image generation
- EmbeddingResponse and ImageResponse types don't have provider response IDs
- Added fallback to standard_logging_payload["id"] (litellm call ID) for
correlation across LiteLLM UI, Phoenix traces, and provider logs
- Completions still use provider ID (e.g. "chatcmpl-xxx") when available
Tests added:
- TestRawSpanAttributeIsolation: Verify raw span has no gen_ai/metadata attrs
- TestNoParentSpanDuplication: Verify parent span doesn't get duplicated attrs
- TestGuardrailSpanParenting: Verify guardrails are children (not orphaned)
- TestResponseIdFallback: Verify response ID set for all call types
All existing OTEL tests pass (73 passed, 14 pre-existing protocol failures).
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat(vertex_ai): support explicit AWS credentials for WIF auth
The current Vertex AI AWS Workload Identity Federation implementation
exclusively uses google.auth.aws.Credentials.from_info(), which requires
EC2 instance metadata access to obtain AWS credentials. In environments
where the metadata service is blocked for security reasons, this makes
WIF unusable.
Add support for explicit AWS credentials by implementing a custom
AwsSecurityCredentialsSupplier (google-auth >= 2.29.0). When aws_* keys
(e.g. aws_role_name, aws_region_name) are present in the WIF credential
JSON, LiteLLM uses BaseAWSLLM.get_credentials() to obtain AWS creds via
STS AssumeRole (or any other supported AWS auth flow), wraps them in the
custom supplier, and passes them to aws.Credentials() — bypassing the
metadata service entirely.
When no aws_* keys are present, the existing from_info() flow is used
unchanged, preserving full backward compatibility.
* refactor(vertex_ai): extract AWS WIF auth to own class + add docs
Address PR review feedback:
- Move _AWS_CREDENTIAL_KEYS, _extract_aws_params(), and
_credentials_from_aws_with_explicit_auth() from VertexBase into
new VertexAIAwsWifAuth class in vertex_ai_aws_wif.py
- Add documentation for explicit AWS credentials WIF auth method
in vertex.md (supported params, JSON example, SDK/Proxy tabs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(vertex_ai): use lazy credentials provider to prevent stale STS tokens
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused `completed_jobs` list (dead code after per-job update refactor)
- Wrap DB update in try/except to prevent one failed update from aborting remaining jobs
- Add test assertions verifying batch_processed, status, and file_object are written to DB
CheckBatchCost poller updated the status column but not the file_object
JSON column. The list_batches endpoint reads status from file_object,
so batches appeared stuck in "validating" even after Azure reported
them as completed. Now update file_object alongside status in the
per-job DB write.
Function calls not supported with reasoning_effort != 'none' on gpt-5.4.
Drop reasoning_effort when tools are in the request (small minority of volume).
Made-with: Cursor
The OpenAI chat completion API expects reasoning_effort as a string
('none', 'low', 'medium', 'high', 'xhigh'). Config/deployments may pass
the Responses API format: {'effort': 'high', 'summary': 'detailed'}.
Fix BadRequestError when model config uses dict format by extracting
the 'effort' value before passing to the API.
Made-with: Cursor
When forward_llm_provider_auth_headers=true, Authorization: Bearer <litellm-key> was
being forwarded to Anthropic if it looked like an OAuth key, causing auth failures.
Now checked against authenticated_with_header: if Authorization was used to authenticate
with the proxy, it is always stripped before forwarding to the LLM provider.
Made-with: Cursor
- Add forward_llm_provider_auth_headers support from litellm_settings
- When enabled, client x-api-key takes precedence over deployment keys
- Forward x-api-key when x-litellm-api-key or Authorization used for auth
- Fix duplicate patch lines in test_byok_oauth_endpoints.py
- Add Claude Code BYOK documentation with /login and ANTHROPIC_CUSTOM_HEADERS
- Add unit tests for clean_headers x-api-key forwarding logic
- Sync model_prices backup (pre-commit hook)
Made-with: Cursor
- Pass request_model to Azure AI cost calculator to detect router requests
- Add router flat cost ($0.14/M input tokens) even when Azure returns actual model in response
- Add test for router flat cost with response containing actual model
- Update docs with cost calculation flow and configuration requirements
Made-with: Cursor
- Shift from hardcoded model checks to dynamic lookup via _supports_factory
- Add supports_none_reasoning_effort for gpt-5.1/5.2/5.4 chat variants
- Add supports_xhigh_reasoning_effort for gpt-5.1-codex-max, gpt-5.2, gpt-5.4+
- Update model_prices_and_context_window.json and backup
- Add ProviderSpecificModelInfo types for new fields
- Fix Azure: use _supports_reasoning_effort_level instead of removed is_model_gpt_5_1_model
Made-with: Cursor
Support passing duration=null on /key/update to reset a key's expiry to never expires, alongside the existing "-1" magic string (kept for backward compat).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The /team/daily/activity endpoint used Prisma pagination (page_size=1000)
but the UI only fetched page 1. Teams with many keys/models easily exceed
1000 rows in LiteLLM_DailyTeamSpend, causing truncated totals.
Switches the endpoint to use SQL GROUP BY via get_daily_activity_aggregated
with include_entity_breakdown=True, returning all data in a single response
while preserving per-team breakdown. Also adds timezone parameter support.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When the messages or response JSON fields in spend logs are truncated
before being written to the database, the truncation marker now includes
a note explaining:
- This is a DB storage safeguard
- Full, untruncated data is still sent to logging callbacks (OTEL, Datadog, etc.)
- The MAX_STRING_LENGTH_PROMPT_IN_DB env var can be used to increase the limit
Also emits a verbose_proxy_logger.info message when truncation occurs in
the request body or response spend log paths.
Adds 3 new tests:
- test_truncation_includes_db_safeguard_note
- test_response_truncation_logs_info_message
- test_request_body_truncation_logs_info_message
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Add CRUD endpoints for managing Hashicorp Vault configuration via the
proxy admin API, with background sync, env var management, and
connection testing. Fix pre-existing bug where premium check ran after
global state mutation, and guard DELETE against clearing non-Vault
secret managers.
Previously, model_dump(exclude_none=True) included all bool fields (since
False != None), causing a partial PATCH to overwrite every other setting to
its default. Fix uses exclude_unset=True and reads the existing DB record
before merging, giving proper PATCH semantics.
This was a pre-existing bug but is fixed here since we're touching this code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Claude Code v2.1.69+ sends `custom: {defer_loading: true}` on tool
definitions. Anthropic's API accepts this field, but Bedrock rejects it
with "Extra inputs are not permitted", causing ~90% of requests to fail.
Strip the `custom` field from each tool in the request body before
sending to Bedrock, in both the Messages API and Chat API invoke paths.
Fixes#22847
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
* fix(proxy): readiness check returns 200 when database is unreachable
_db_health_readiness_check() catches health_check() exceptions but
never updates db_health_cache to "disconnected" and never re-raises.
The caller health_readiness() always returns 200 with "db": "connected"
hardcoded, regardless of actual DB state.
In Kubernetes, this means pods with dead database connections stay in
the Service endpoints and continue receiving traffic they cannot serve.
Changes:
- Set db_health_cache to "disconnected" and re-raise the exception on
health_check failure so health_readiness() returns 503
- Use actual db_health_status["status"] in the response instead of
hardcoding "db": "connected"
- Reduce cache TTL from 2 minutes to 15 seconds. The 2-minute window
is too wide for readiness probes (typically 10-15s intervals) and
means a pod can report healthy for up to 2 minutes after the DB dies
- Only serve cached results when status is "connected". The previous
condition (status != "unknown") would also cache "disconnected" for
2 minutes, delaying recovery detection after a DB comes back
* fix(proxy): add DB connection self-healing to readiness check
When the Prisma query engine's internal TCP connection pool holds dead
connections (caused by network blips, Cloud SQL proxy restarts, or
node-level issues), health_check() fails with httpx.ConnectError.
The engine never recovers on its own because nothing triggers a
disconnect/connect cycle to restart the subprocess with fresh
connections.
This leaves pods permanently failing readiness checks until they are
manually restarted, even after the underlying DB becomes reachable
again.
Add a reconnect attempt to _db_health_readiness_check() when
health_check() fails:
1. disconnect() - kills the query engine subprocess and closes all
connections (has built-in backoff retry: 3 tries, 10s max)
2. connect() - starts a new engine with fresh TCP connections (has
built-in backoff retry: 3 tries, 10s max)
3. health_check() - verifies the new connection works (has built-in
backoff retry: 3 tries, 10s max)
If reconnect succeeds, the pod immediately returns to service (200).
If it fails, the original exception is re-raised (503). Reconnect
attempts are rate-limited by probe frequency (~10-15s), so a
permanently unreachable DB gets one attempt per cycle with no retry
loops.
This uses the same disconnect/connect mechanism that
PrismaWrapper.recreate_prisma_client() uses for IAM token refresh,
and aligns with the community-documented pattern for Prisma connection
recovery in long-running processes (prisma/prisma#24718, #27024).
* Add poetry lock and modify test_health_endpoints
* Address allow_requests_on_db_unavailable regression
* Address comments
* resolve greptile issue
* Restore accidentally deleted UI HTML files
These were removed in an earlier commit but still exist on main.
Restoring to keep the PR diff clean.
* Guard reconnect with is_database_transport_error
Only attempt disconnect/connect/health_check cycle for transport-level
failures (unreachable DB, dropped connection). Data-layer errors like
UniqueViolationError indicate the DB is reachable, so reconnecting
would be pointless churn.
* Address greptile's comments
* Fix module alias after rebase and add adversarial test coverage
- Unify module alias to _health_endpoints_module after rebase conflict
- Add test for non-transport error with flag on (exercises is_database_transport_error guard)
- Add test for disconnect() failure during reconnect cycle
- Split non-transport error test into flag-off (re-raises) and flag-on (skips reconnect) variants
* Remove stale UI HTML files reintroduced during rebase
* fix: don't close HTTP/SDK clients on LLMClientCache eviction
Removing the _remove_key override that eagerly called aclose()/close()
on evicted clients. Evicted clients may still be held by in-flight
streaming requests; closing them causes:
RuntimeError: Cannot send a request, as the client has been closed.
This is a regression from commit fb72979432. Clients that are no longer
referenced will be garbage-collected naturally. Explicit shutdown cleanup
happens via close_litellm_async_clients().
Fixes production crashes after the 1-hour cache TTL expires.
* test: update LLMClientCache unit tests for no-close-on-eviction behavior
Flip the assertions: evicted clients must NOT be closed. Replace
test_remove_key_closes_async_client → test_remove_key_does_not_close_async_client
and equivalents for sync/eviction paths.
Add test_remove_key_removes_plain_values for non-client cache entries.
Remove test_background_tasks_cleaned_up_after_completion (no more _background_tasks).
Remove test_remove_key_no_event_loop variant that depended on old behavior.
* test: add e2e tests for OpenAI SDK client surviving cache eviction
Add two new e2e tests using real AsyncOpenAI clients:
- test_evicted_openai_sdk_client_stays_usable: verifies size-based eviction
doesn't close the client
- test_ttl_expired_openai_sdk_client_stays_usable: verifies TTL expiry
eviction doesn't close the client
Both tests sleep after eviction so any create_task()-based close would
have time to run, making the regression detectable.
Also expand the module docstring to explain why the sleep is required.
* docs(AGENTS.md): add rule — never close HTTP/SDK clients on cache eviction
* docs(CLAUDE.md): add HTTP client cache safety guideline
