When use_sidecar is set in the YAML config (general_settings), ensure the
USE_SIDECAR and SIDECAR_PORT env vars are set so that
AsyncHTTPHandler._should_use_sidecar_transport() picks them up.
Also add None guard for prisma_client in _sync_ui_settings_to_general_settings.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix connection leak: use release() instead of close() in SidecarResponseStream.aclose()
- Fix concurrent session leak: add asyncio.Lock with double-check in _get_session()
- Fix TOCTOU race: use DashMap entry() API in Rust sidecar get_or_create_client()
- Fix broken streaming: detect stream from request body JSON instead of Accept header
- Fix event loop stall: replace blocking subprocess.wait() with async run_in_executor()
- Fix type contract: decode orjson.dumps() bytes to str for consistency with json.dumps()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* perf: address GIL contention and hot-path bottlenecks from profiling data
Based on py-spy GIL profiling (38,800 samples, 2000 concurrent users) and
pyinstrument per-request timing, this commit addresses the top performance
bottlenecks identified:
1. _sanitize_request_body_for_spend_logs_payload (2.9% GIL):
- Remove redundant inner import (constants already imported at top-level)
- Remove dead-code branch (len check after already confirmed len > max)
- Pre-compute truncation ratios outside inner function
- Reorder isinstance checks: str first (most common leaf type)
2. Pydantic repr in logging (2.3% GIL):
- Guard print_deployment calls behind isEnabledFor(logging.INFO)
- Replace copy.deepcopy with shallow dict() copy in print_deployment
- Use %-style lazy formatting instead of f-strings for logger calls
- Remove kwargs from prometheus debug log message
3. Prometheus label_factory overhead (1.5% + 0.7% GIL):
- Cache model_dump() on UserAPIKeyLabelValues via get_label_dict()
- Convert supported_enum_labels to frozenset for O(1) membership tests
- Called 37 times per success event; caching avoids 36 redundant dumps
4. pre_call_utils header lookup (1.9% GIL):
- Replace dict comprehension over all headers with early-exit loop
- Only lowercase and compare the two target header names
5. safe_json_dumps (0.7% GIL):
- Replace stdlib json.dumps with orjson.dumps for final serialization
6. Hot-path debug logging:
- Convert f-string debug logs to %-style in litellm_logging.py
- Simplify prometheus print_verbose call
7. Cost calculator annotation checks:
- Optimize response_includes_annotation_type to handle both dict
and object annotation types without repeated __getattr__ calls
Estimated GIL time reduction: ~11-12% under concurrency.
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* test: add locust benchmark comparison tooling for perf analysis
Adds:
- loadtest_config_perf.yaml: proxy config with spend_logs enabled
- locustfile_perf.py: locust scenario for perf comparison
- compare_perf_results.py: CSV parser + comparison report generator
- run_perf_comparison.sh: automated baseline vs optimized runner
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* perf: guard orjson import with fallback, pre-parse httpx URLs, fix docstring
Addresses three review concerns and adds httpx URL caching:
1. safe_json_dumps.py: Guard orjson import with try/except fallback to
stdlib json. This module is on the core SDK import path via
_logging.py — unconditional orjson import would break plain
'pip install litellm' (non-proxy) users.
2. router.py print_deployment: Update docstring to accurately describe
the reduced return shape (model_name + litellm_params only).
3. run_perf_comparison.sh: Fix locustfile reference to use the correct
locustfile_perf.py instead of locustfile.py.
4. httpx URL pre-parsing (~7.8us -> ~0.4us per request, 19x speedup):
Add _parse_url() with LRU cache (maxsize=64) that pre-parses URL
strings into httpx.URL objects. Applied to all HTTP methods (GET,
POST, PUT, PATCH, DELETE) in both AsyncHTTPHandler and HTTPHandler.
Eliminates regex-heavy re.finditer inside httpx._urlparse on every
request — confirmed as a GIL hotspot in py-spy thread dumps.
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
* fix: address review - document _parse_url safety, guard print_verbose
1. http_handler.py _parse_url: Add docstring documenting why the LRU
cache is safe with query-string URLs. When params= is non-None,
httpx replaces the query string entirely; when params= is None,
the cached URL preserves the original query string. Both match
pre-optimization behavior (verified with httpx.URL vs str tests).
2. prometheus.py print_verbose: Guard the call behind litellm.set_verbose
check so the string formatting is truly lazy. The previous % formatting
was eagerly evaluated (same cost as f-string) since print_verbose takes
a pre-formatted string.
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The Chat button in the navbar and the "back to Developer Console" link
in ChatPage used the `serverRootPath` module variable directly, which is
initialized to "/" and only updated after `getUiConfig` resolves. Since
React does not re-render on module variable changes, the links computed
their hrefs with the stale default, ignoring any configured
SERVER_ROOT_PATH.
Both components now call `useUIConfig()` (React Query, cached) to
reactively read `server_root_path`, matching the pattern used elsewhere
in the UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The validation method now reads use_redis_transaction_buffer directly
from the passed general_settings dict rather than delegating to
RedisUpdateBuffer._should_commit_spend_updates_to_redis() which
imports the global. Tests simplified to remove unnecessary patching.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When `use_redis_transaction_buffer: true` is set in general_settings but no
Redis cache is configured in litellm_settings, the proxy starts successfully
but silently drops all spend tracking data. This adds a startup validation
that raises a clear error, preventing the proxy from running in a broken state.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: Add Canadian PII protection patterns and PIPEDA-compliant policy template
Adds 6 new Canadian PII regex pattern detectors to patterns.json:
- ca_sin: Social Insurance Number (PIPEDA Privacy Act, Income Tax Act)
- ca_ohip: Ontario Health Insurance Plan Number (PHIPA)
- ca_on_drivers_licence: Ontario driver's licence (HTA, PIPEDA)
- ca_immigration_doc: IRCC immigration docs (UCI, work/study permits, IMM refs)
- ca_bank_account: Canadian bank account routing (transit-institution-account)
- ca_postal_code: Canadian postal code (Canada Post spec)
Adds comprehensive policy template 'canadian-pii-protection' (id: canadian-pii-protection)
with 5 sub-guardrails grouping patterns by data type. All patterns include contextual
keyword matching (English + French keywords where applicable) to reduce false positives.
Complements existing passport_canada pattern.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat: Add Canadian PII compliance dataset and tests (57 tests)
Adds:
- test_ca_patterns.py: 30 unit tests for regex pattern matching (SIN, OHIP,
driver's licence, immigration docs, bank account, postal code)
- test_ca_policy_e2e.py: 27 end-to-end tests running the full
ContentFilterGuardrail pipeline with MASK action — validates detection
of real PII and pass-through of clean prompts
- canadianPiiCompliancePrompts.ts: 21-prompt compliance dataset for UI
evaluation, wired into the main compliancePrompts framework
Fixes keyword_pattern alternation ordering in patterns.json — longer
alternatives (e.g. "social insurance number") now precede shorter ones
("social insurance") to avoid excessive gap-word count when the regex
engine selects the shorter match first.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat: Add University of Toronto FIPPA identifier patterns and tests (36 tests)
Add 3 UofT institutional identifiers (student/employee number, UTORid, TCard)
covered under Ontario FIPPA. Includes pattern definitions, policy template
sub-guardrail, compliance prompts, unit tests, and e2e tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Correct test assertion bug and inaccurate docstring
Fix test_utorid_masked checking `result` (dict) instead of `output` (string).
Update test_ca_policy_e2e.py docstring to clarify scope vs UofT tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Address Greptile review feedback
- Tighten ca_postal_code keyword_pattern: replace broad "address" with
specific compound terms (mailing/street/shipping/home address)
- Add missing "PIPEDA" tag to policy_templates.json for discoverability
- Add us_phone pattern to test_ca_policy_e2e.py setup to match deployed template
- Add phone number e2e test for complete coverage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: Tighten patterns to reduce false positives and add missing test coverage
- ca_sin: reject leading-zero SINs ([1-9]\d{2}), set allow_word_numbers to false
- ca_immigration_doc: require separators in UCI pattern (prevent bare \d{10} match)
- uoft_utorid: qualify generic keywords (acorn -> acorn login, quercus -> quercus login)
- uoft_tcard: remove generic keywords (student card, id card, library card) that
overlap with credit card contexts; keep only UofT-specific terms (tcard, campus card)
- Add visa/mastercard/amex/iban patterns to test_ca_policy_e2e.py setup to match
deployed template; add Visa card masking test
- Add test verifying "student card" no longer triggers TCard redaction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Internal users were blocked from accessing /key/aliases because the route
was missing from key_management_routes. Added the route and scoped query
results so non-admin users only see aliases for their own keys and their
teams' keys, matching /key/list behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: use DeepSeekChatConfig instead of OpenAIConfig for deepseek provider
The deepseek provider was incorrectly using OpenAIConfig().map_openai_params()
instead of DeepSeekChatConfig().map_openai_params(), which meant DeepSeek-specific
parameter mappings were not being applied.
* test: add unit tests for deepseek DeepSeekChatConfig param mapping
Verify that get_optional_params uses DeepSeekChatConfig (not OpenAIConfig)
for the deepseek provider by testing thinking, reasoning_effort, and
budget_tokens stripping behavior.
* feat: add LITELLM_WORKER_STARTUP_HOOKS for per-worker initialization (gflags support)
Add support for running user-defined startup hooks in each worker process
during proxy_startup_event. This enables re-initialization of in-process
state (like gflags.FLAGS) that doesn't survive uvicorn worker spawning.
Usage:
export LITELLM_WORKER_STARTUP_HOOKS=mymodule:init_fn,other:setup_fn
Hooks run early in proxy_startup_event (before config/DB loading).
Supports both sync and async callables. Errors propagate to prevent
broken workers from serving traffic. No-op when env var is unset.
Includes 5 tests covering sync/async hooks, multiple hooks, error
propagation, and no-hooks-set scenarios.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: add Worker Startup Hooks page with gflags usage example
- New docs page: docs/proxy/worker_startup_hooks.md
- Explains the problem (per-process state lost in multi-worker deployments)
- Full gflags example with wrapper module and startup script
- Covers multiple hooks, async hooks, error behavior
- Architecture diagram showing master→worker flow
- Added LITELLM_WORKER_STARTUP_HOOKS to config_settings.md env var table
- Added to sidebar under Setup & Deployment
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update litellm/proxy/proxy_server.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mcp): resolve \$ref params and merge path-level params in OpenAPI tool registration
Real-world OpenAPI specs (e.g. GitHub's 11.8 MB official spec) use two
patterns that crashed tool registration:
1. \$ref parameters: params defined as {"$ref": "#/components/parameters/foo"}
instead of inline objects. Accessing param["name"] on a $ref raises KeyError.
Fix: resolve each param against components/parameters before processing.
2. Path-level parameters: params defined on the path object apply to all
HTTP methods on that path, but the operation object doesn't include them.
GitHub's spec uses this for owner/repo/etc. path params.
Fix: merge path-level params with operation-level params (op-level wins
when the same name+in combination appears in both).
With this fix the full GitHub REST API spec loads successfully:
720 paths → 1079 tools, all with correct parameter schemas.
* fix(mcp): resolve \$ref params in OpenAPI preview endpoint (test/tools/list)
The _preview_openapi_tools function (called by the UI add-server form to show
connection status and available tools) had the same bug as _register_openapi_tools:
it accessed param["name"] directly without resolving \$ref parameters or merging
path-level parameters from the path item.
This caused "Failed to load OpenAPI spec: 'name'" for any spec that uses
component-level parameter references (e.g. GitHub's official REST API spec).
Apply the same fix: resolve \$ref against components/parameters and merge
path-level params (with operation-level taking priority) before building schemas.
* refactor(openapi-mcp): extract resolve_operation_params, add tests
- Hoist _resolve_ref and _resolve_param_list to module level in
openapi_to_mcp_generator.py (were being redefined on every loop iteration)
- _resolve_ref now returns None for unresolvable $refs instead of
the stub dict, preventing (None, None) from poisoning deduplication
- Add resolve_operation_params() as a shared helper that handles both
$ref resolution and path-level param merging
- Replace duplicated inline logic in mcp_server_manager.py and
rest_endpoints.py with calls to resolve_operation_params()
- Add TestResolveRef, TestResolveParamList, TestResolveOperationParams
test classes covering $ref resolution, path-level merging, collision
semantics, unresolvable ref filtering, and a GitHub-style spec fixture
When Redis Cluster is configured via the REDIS_CLUSTER_NODES environment
variable, Cache.__init__() and Router._create_redis_cache() ignored the
env var and always created RedisCache instead of RedisClusterCache. This
caused the v3 rate limiter's cluster detection (_is_redis_cluster()) to
return False, skipping hash-slot key grouping. The resulting CROSSLOT
errors were silently caught, falling back to per-instance in-memory
counting — breaking RPM/TPM enforcement across multiple proxy instances.
Add REDIS_CLUSTER_NODES env var detection to both Cache.__init__() and
Router._create_redis_cache(), matching the existing pattern in
_redis.py:215-220. When the env var is set and no explicit startup_nodes
parameter is provided, parse it and create RedisClusterCache.
Fixes#22748
Related to #20836
