mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
72 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sameer Kankute
|
1ccc1e5b23
|
chore: litellm oss staging160626 (#30527)
* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit |
||
|
Yassin Kortam
|
a992ed18df
|
feat(spend_logs): opt-in native Postgres partitioning for SpendLogs retention (#29466)
High-volume deployments see LiteLLM_SpendLogs grow unbounded because retention via DELETE leaves dead tuples that autovacuum cannot reclaim fast enough. With a range-partitioned table, retention drops whole partitions instead: an instant metadata operation that returns disk to the OS immediately. The feature is gated behind general_settings.use_spend_logs_partitioning (default false). With the flag off, the cleanup job never queries the catalog and behaves exactly as today. With it on, the job verifies the table is partitioned, pre-creates upcoming partitions, and drops expired ones; expired rows the drops cannot reach (DEFAULT partition, partitions spanning the cutoff) are still deleted row-wise so retention is never bypassed. If the table is not partitioned it falls back to batched DELETE only. Converting an existing table is a manual, documented operation in db_scripts/partition_spend_logs.sql; db_scripts/unpartition_spend_logs.sql rolls it back. Both scripts rename the old table's indexes aside before recreating them, since a table rename keeps the schema-unique index names and would otherwise silently skip the CREATE INDEX IF NOT EXISTS block. Granularity and pre-create lookahead are tunable via SPEND_LOG_PARTITION_INTERVAL (day/week/month, invalid values fall back to day) and SPEND_LOG_PARTITION_PRECREATE_AHEAD. |
||
|
Yassin Kortam
|
da9d64b4de
|
fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures (#29986) | ||
|
michelligabriele
|
f9293d40c4
|
fix(proxy): self-heal startup/reload prisma reads on engine disconnect (#28803) | ||
|
Sameer Kankute
|
95e3d136e1
|
test(google): add google-genai SDK proxy integration tests (#29781)
* test(google): add google-genai SDK proxy integration tests for Gemini and Vertex Pin google-genai in the CI dependency group and exercise streaming/non-streaming generate_content through the LiteLLM proxy in the existing unified_google_tests suite. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(test): address Greptile review for google-genai proxy SDK tests Restore GOOGLE_APPLICATION_CREDENTIALS after the module proxy fixture tears down, initialize temp-file tracking on the proxy SDK base class, and skip litellm reload for proxy_genai_sdk tests so the module-scoped proxy server stays consistent. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(test): only load Vertex credentials when keys exist for proxy SDK tests Avoid writing empty GOOGLE_APPLICATION_CREDENTIALS temp files so Vertex tests skip cleanly without credentials, use a session-scoped proxy fixture, and clean up per-test credential temp files. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(test): scope google-genai pin to unified_google_tests only Remove google-genai from the ci dependency group and pin it in tests/unified_google_tests/requirements.txt for local test installs. Co-authored-by: Cursor <cursoragent@cursor.com> * test(google): tie litellm reload skip to proxy fixture dependency Replace the name-based reload guard with a check on whether the test requests the google_genai_proxy_url fixture, so the skip stays correct if the proxy SDK tests are renamed. * fix(test): stop DatabaseURLSettings tests leaking DATABASE_URL into os.environ The autouse env scrubber relied on monkeypatch.delenv, but apply_to_env writes DATABASE_URL straight into os.environ, which monkeypatch never tracks and therefore never undoes. The synthesized writer.example.com URL leaked past the last test in this module and into proxy-infra tests that read DATABASE_URL to decide whether to hit a real database, e.g. test_deprecated_key_grace_period_cache_hit_path, turning an intended skip into a ConnectError. Snapshot and restore the managed vars directly so the original environment is reinstated regardless of how it was mutated. * test(google): drop redundant per-test vertex credential setup The session-scoped google_genai_proxy_url fixture already configures GOOGLE_APPLICATION_CREDENTIALS before the proxy starts, and _require_proxy_sdk skips when credentials are missing, so the per-test _setup_vertex_credentials_if_needed helper and its temp-file tracking never did any work. Remove it to keep the ABC self-contained. * test(google): declare model_config contract on proxy SDK ABC _skip_reason_if_credentials_missing reads self.model_config to pick the provider, but that property was only declared on the sibling BaseGoogleGenAITest. Make the dependency explicit by adding model_config as an abstract property on BaseGoogleGenAIProxySDKTest so the ABC is self-contained and a standalone subclass fails fast instead of hitting an AttributeError. * test(google): narrow streaming error catch to Exception Catching BaseException in the streaming assertion swallowed KeyboardInterrupt and SystemExit, turning a Ctrl-C into a test failure message instead of letting pytest interrupt cleanly. Only genuine runtime errors should be recorded as stream failures, so catch Exception. * test(google): initialize proxy on the same loop that serves it The proxy was initialized via asyncio.run() on the main thread, which creates and tears down a throwaway event loop, while requests were served on a separate loop in the worker thread. Any asyncio primitive bound to the init loop would be unusable once serving started. Run initialize() on the worker thread's loop right before server.serve() so setup and request handling share a single event loop. * test(google): drop redundant google-genai requirements pin google-genai>=1.37.0,<2.0 is already declared in the proxy-runtime extra, which the google_generate_content_endpoint_testing CI job installs via uv sync --all-extras. The standalone tests/unified_google_tests/requirements.txt duplicated that pin with a narrower ==1.37.0 specifier and was never installed by CI, so it added a second source of truth without changing what gets installed. Drop it and rely on the proxy-runtime extra. * chore: revert incidental uv.lock exclude-newer bump The google-genai ci pin was added and then dropped (it is already provided by the proxy-runtime group), but each uv lock recomputed the relative exclude-newer span, leaving only a timestamp bump in uv.lock. Restore it to the base value so this test-only PR carries no lockfile change. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yassin Kortam
|
014cb8fa9d
|
feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557)
Split the monolithic LiteLLM proxy into independently scalable Kubernetes components to allow separate horizontal scaling of the LLM data plane and management API surfaces - Add DatabaseURLSettings pydantic-settings model that assembles DATABASE_URL (and optional DATABASE_URL_READ_REPLICA) from discrete DATABASE_* env vars before Prisma initializes, supporting both IAM token auth (minting short-lived RDS tokens) and password auth; replaces the CLI-only path that componentized entrypoints bypass - Add gateway component (port 4000) that trims the proxy route table to the LLM data-plane surface (chat, embeddings, completions, audio, realtime, provider passthroughs, health/metrics) via an allowlist applied inside the lifespan context so plugin-registered routes are captured - Add backend component (port 4001) that exposes the management/admin surface (keys, users, teams, orgs, spend analytics, model management, SSO, audit logs) with a complementary allowlist - Add ui component — Next.js static export served by nginx (port 3000) with RSC payload routing, asset prefix aliasing, and SPA fallback for dashboard routes - Add migrations component with dedicated Dockerfile that runs prisma migrate deploy via a Helm pre-install/pre-upgrade Job, eliminating per-pod schema contention on the Prisma advisory lock - Add Helm chart (helm/litellm) with separate Deployments, Services, HPAs, and ConfigMap for each component; shared _helpers.tpl emits DATABASE_*, IAM_TOKEN_DB_AUTH, REDIS_*, and DISABLE_SCHEMA_UPDATE env vars from chart values; ingress template routes traffic to the correct component by path prefix - Add comprehensive tests for DatabaseURLSettings covering IAM auth, password auth, read replica fallbacks, operator-pinned URL preservation, and percent-encoding; add coverage test asserting gateway + backend allowlist union equals the full proxy route set - Add pydantic-settings>=2.14.1 as a proxy extra dependency and update liccheck allowlist Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|
Sameer Kankute
|
ce17e9490f
|
Merge branch 'litellm_internal_staging' into litellm_agent_oss_staging_05_06_2026 | ||
|
Yassin Kortam
|
b5d3a5fc85
|
feat: add read-replica routing for Prisma DB via DATABASE_URL_READ_REPLICA (#27493)
Some checks are pending
Unit Tests: Caching (Redis) / caching-redis (push) Waiting to run
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / schema-migration (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Security / security (push) Waiting to run
- Introduce RoutingPrismaWrapper that transparently routes read operations (find_*, count, group_by, query_raw, query_first) to a reader endpoint while writes remain on the writer, enabling Aurora-style reader/writer endpoint splits - Add IAMEndpoint dataclass and parse_iam_endpoint_from_url() to capture static connection fields from a reader URL so only the IAM token needs to rotate, avoiding the need for separate DATABASE_HOST_READ_REPLICA/etc. env vars - Enhance PrismaWrapper with per-instance knobs (db_url_env_var, iam_endpoint, recreate_uses_datasource, log_prefix) so writer and reader wrappers are independent: the reader writes its fresh URL to DATABASE_URL_READ_REPLICA and passes datasource override to Prisma since Prisma only auto-reads DATABASE_URL - Fix deadlock in PrismaWrapper.__getattr__: when called from inside a running event loop, schedule the token refresh as a background task instead of blocking with run_coroutine_threadsafe + future.result(), which would deadlock the loop thread waiting for a coroutine that needs the loop to run - Fix botocore crash when DATABASE_PORT is unset by defaulting to "5432" in both proxy_cli.py and PrismaWrapper.get_rds_iam_token(); passing None caused botocore to embed the literal string "None" in the presigned URL - Implement graceful reader degradation: reader connect/recreate failures are non-fatal; wrapper sets _reader_unavailable=True and silently routes reads to the writer to keep the proxy serving traffic during transient reader outages - Add PrismaClient.writer_db property so the reconnect smoke-test always validates the writer engine specifically; query_raw on the routing wrapper would route to the reader and not verify the newly-recreated writer - Expose DATABASE_URL_READ_REPLICA in Helm chart (values.yaml + deployment.yaml) via both plain value and secret key reference, and document the field in docker-compose.yml - Add 887-line test suite covering routing logic, IAM token refresh paths, reader degradation scenarios, datasource override behavior, and the deadlock regression Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu> |
||
|
oss-pr-review-agent-shin[bot]
|
ed7ded9e77
|
Merge PR #27221 into agent staging branch | ||
|
pnookala-godaddy
|
209bd0b906 |
fix(proxy): sort spend updates to prevent DB deadlocks
Iterate user/key/team/team_member/org/end_user/tag spend dicts in sorted order inside each Prisma transaction so concurrent pods acquire row locks in the same order, avoiding PostgreSQL deadlocks under load. |
||
|
Cursor Agent
|
1cbe744114
|
feat(spend-logs): opt-in suppression of stack traces in spend-tracking error logs
Adds LITELLM_SUPPRESS_SPEND_LOG_TRACEBACKS env var. When set to true and the
proxy log level is INFO or above, spend-tracking error paths emit a single
ERROR line without the full traceback. Stack traces are preserved at DEBUG
and the Sentry / proxy_logging_obj.failure_handler path is unchanged.
The new spend_log_error helper is wired through the spend write hot path:
- DBSpendUpdateWriter (update_database, _update_*_db, batch upsert,
redis-commit fallbacks)
- _ProxyDBLogger._PROXY_track_cost_callback
- get_logging_payload exception path
- update_spend / update_daily_tag_spend / spend logs queue monitor
Resolves LIT-2704.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
|
||
|
Yuneng Jiang
|
4f6192a49e |
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_dbReconnectNonBlocking_local
# Conflicts: # tests/test_litellm/proxy/db/test_prisma_self_heal.py |
||
|
Yuneng Jiang
|
aa2ef41200 |
fix(proxy): preserve original transport error if reconnect itself raises
Greptile review on #26756 (P2): if `attempt_db_reconnect` itself raises (e.g. lock cancellation, timer error, unexpected internal failure), the original `httpx.ReadError` / transport error was lost — `failure_handler` and `db_exceptions` alerts then logged the reconnect exception instead of the actual DB transport problem, masking the root cause. Wrap the reconnect call in a try/except. On reconnect failure, re-raise the *original* `first_exc` and chain the reconnect error as `__cause__` so it remains visible for debuggability without becoming the primary exception observers see. Adds `test_call_with_db_reconnect_retry_preserves_original_error_when_reconnect_raises` asserting (a) the propagated exception is the original transport error and (b) the reconnect exception is attached as `__cause__`. |
||
|
Yuneng Jiang
|
1c9c219a74 |
fix(proxy): self-heal Prisma read paths + harden reconnect state machine
Two related fixes layered on top of the existing reconnect plumbing: 1. Restore reconnect-and-retry on `PrismaClient.get_generic_data` (issue #25143). 1.83.x lost the transport-reconnect-and-retry-once branch that 1.82.6 had on this method, so transient `httpx.ReadError` flaps now surface immediately as `db_exceptions` alerts. `_update_config_from_db` fans out four concurrent `get_generic_data` reads, so a single transport blip used to mark four alerts and a stale config window. Adds `call_with_db_reconnect_retry` to `litellm/proxy/db/exception_handler.py` — a single canonical "try DB read, on transport error reconnect once and retry once" wrapper. Mirrors the inline pattern in `auth_checks._fetch_key_object_from_db_with_reconnect` so we have one implementation rather than three drifting copies, and gives future read paths a clean opt-in. 2. Fix the `_engine_confirmed_dead` flag-reset bug in `_run_reconnect_cycle`. The flag was cleared before `_do_heavy_reconnect()` ran, so any failure inside the heavy reconnect (timeout, missing DATABASE_URL, recreate failure) left the flag False — and the next attempt could silently demote to the lightweight path even though the engine was genuinely dead. Move the reset into the success branch so the flag stays True across heavy-reconnect failures and the next attempt re-enters the heavy branch. Tests: - `tests/test_litellm/proxy/db/test_exception_handler_reconnect_retry.py` (new) — 9 tests covering the helper's contract: happy path, retry on transport error, no retry on data-layer errors, propagation when reconnect fails, propagation after second transport error, `hasattr` guard for partial mocks, fresh-coroutine-per-call invariant, explicit timeout override, default timeouts read off the prisma_client. - `tests/test_litellm/proxy/db/test_prisma_self_heal.py` — adds: - `test_get_generic_data_retries_on_transport_error_for_config_table` - `test_get_generic_data_propagates_when_reconnect_fails` - `test_engine_confirmed_dead_persists_across_failed_heavy_reconnect` (regression test for the flag-reset bug). All 16 self-heal tests + 9 helper tests + 535 auth/exception-handler tests pass locally. |
||
|
Yuneng Jiang
|
8c91c8b2c4 | Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_dbReconnectNonBlocking | ||
|
ryan-crabbe-berri
|
c4c1861389
|
Merge pull request #26195 from BerriAI/litellm_team_member_total_spend
Track per-member total spend on team memberships |
||
|
Yuneng Jiang
|
288d403529
|
[Fix] Preserve in-memory spend updates when Redis rpush fails
store_in_memory_spend_updates_in_redis drained the in-memory queues into local variables before the rpush pipeline. If rpush raised (cloud Redis hiccup, timeout, connection blip), those already-drained transactions were garbage-collected with the scheduler job, silently losing all spend aggregated during that tick. Wrap the rpush in try/except. On failure, re-enqueue the aggregated transactions into their respective in-memory queues so the next scheduler tick retries. Add a unit test that seeds real queues, simulates an rpush failure, and asserts the transactions land back in-memory. |
||
|
Yuneng Jiang
|
fbcdacc446
|
[Fix] Proxy: reconnect Prisma DB without blocking the event loop
When the DB becomes unreachable the reconnect path calls
`prisma.disconnect()`, which ultimately invokes prisma-client-py's
synchronous `subprocess.Popen.wait()` on the query engine subprocess.
That call does not yield to asyncio, so the event loop freezes for
however long the Rust engine takes to shut down (30-120+ seconds in
production when the engine is stuck on TCP close). During the freeze
`/health/liveliness` becomes unresponsive, and in Kubernetes the
liveness probe fails and the pod is SIGKILL'd.
Replace `disconnect()` in the reconnect paths with a direct, non-blocking
kill of the engine subprocess (SIGTERM -> 0.5s asyncio-yielding sleep ->
SIGKILL) followed by a fresh Prisma client and a new `connect()`. Both
`recreate_prisma_client` and the formerly-separate "direct reconnect"
path go through the same kill-then-recreate flow.
Also validate `_get_engine_pid` returns an int (defensive; prevents a
MagicMock leak under unit-test mocking).
Tests that encoded the old blocking behavior are updated or removed;
the deleted `test_lightweight_reconnect_skips_kill_on_successful_disconnect`
invariant ("don't kill on successful disconnect") was part of the bug.
|
||
|
Ryan Crabbe
|
e5f3e15969
|
Track per-member total spend on team memberships
Adds total_spend column to LiteLLM_TeamMembership that accumulates continuously and is not zeroed by the budget cycle reset job. This enables UI surfaces to distinguish current-cycle spend (the existing spend column, which resets) from lifetime spend per team member. Also exposes budget_reset_at on LiteLLM_BudgetTable so /team/info callers can see when a member's budget window next resets. The field was already stored in the DB but stripped by the response Pydantic model. Includes regression tests that: - Guard the reset job against ever writing total_spend: 0 - Verify the spend writer increments both spend and total_spend in one UPDATE statement. |
||
|
Yuneng Jiang
|
9c0b73e5f4
|
[Fix] should_create_missing_views returns False for reltuples=0 (falsy zero bug)
`should_create_missing_views()` had `and result[0]["reltuples"]` which is falsy when reltuples=0. On a fresh empty PostgreSQL table, CREATE INDEX sets reltuples=0, causing the guard to return False and skip view creation entirely. Views like MonthlyGlobalSpendPerKey are never created, and the /global/spend/logs endpoint returns 500. Fix: change to `and result[0]["reltuples"] is not None` so reltuples=0 (empty table) and reltuples=-1 (unanalyzed table) both correctly return True. Also harden test_vertex_ai.py to return None instead of crashing with JSONDecodeError when the spend-logs endpoint returns a non-JSON 500 response, and add unit tests covering all three reltuples branches (0, -1, positive). |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Joe Reyna
|
f92490c308
|
fix: make PodLockManager.release_lock atomic compare-and-delete (re-land #21226) (#24466)
Some checks are pending
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (key-generation, tests/proxy_unit_tests/test_key_generate_prisma.py, 30, 0) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 30, 8) (push) Waiting to run
Unit Tests: Security / security (push) Waiting to run
* fix: make PodLockManager.release_lock atomic compare-and-delete Re-lands #21226 (reverted in #21469). release_lock() previously did GET + compare + DEL in separate calls, leaving a window where another pod could reacquire the lock between the GET and DEL, causing a stale owner to delete a live lock. Fix: use a Redis Lua script for atomic compare-and-delete. Script registration is cached per PodLockManager instance. Falls back to the old GET+DEL path for cache backends that don't expose async_register_script. Original revert was due to e2e tests running in CI without Redis. Those tests now carry @pytest.mark.skip(reason="Requires Redis connection.") so this re-land is safe. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: add Lua fallback on execution error + test coverage gaps Address Greptile review feedback on #24466: 1. Wrap Lua script execution in try/except — if Redis clears loaded scripts (restart) or scripting is disabled, fall back to GET+DEL rather than letting the exception propagate and leave the lock held until TTL. Reset cached script handle so the next call re-registers. 2. Add test_release_lock_lua_path_emits_released_event — verifies _emit_released_lock_event is called when Lua path returns 1. 3. Add test_release_lock_falls_back_to_get_del_when_lua_execution_fails — verifies the fallback path is taken and script handle is reset. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
b0a40fde6d
|
Merge pull request #25559 from shreyescodes/fix/cors-and-db-safety-bugs
fix: harden CORS credentials, create_views exception handling, and spend log cleanup loop |
||
|
shreyes19
|
e01fe01d35 | fix: address Greptile P1 review comments | ||
|
shreyes19
|
e079ee779f |
fix: harden CORS, create_views exception handling, and spend log cleanup loop
- proxy_server.py: disable allow_credentials when allow_origins=['*'] (wildcard + credentials is a browser security misconfiguration). Add LITELLM_CORS_ORIGINS env var to configure explicit allowed origins. - create_views.py: narrow broad 'except Exception' to only catch genuine 'view does not exist' errors; re-raise all other DB errors (auth, connection, etc.) that were previously silently swallowed. - spend_log_cleanup.py: validate execute_raw() return type is int before using it as a deletion count; break loop safely on unexpected types to prevent infinite deletion loops. |
||
|
stuxf
|
a6c30b30bf
|
build: migrate packaging, CI, and Docker from Poetry to uv (#25007)
* build: migrate packaging metadata to uv * ci: move automation and local tooling to uv * docker: migrate image builds and runtime setup to uv * docs: update install and deployment guidance for uv * chore: align auxiliary scripts and tests with uv * test: harden test_litellm isolation * fix: keep release and health check images self-contained * build: pin uv tooling and health check deps * test: isolate bedrock image request formatting from suite state * test: cover sandbox executor requirements flow * ci: fix circleci no-op command steps * ci: fix circleci publish workflow parsing * fix: stabilize remaining uv migration CI checks * ci: increase matrix test timeout headroom * fix: restore published docker and license coverage * fix: restore proxy runtime build parity * fix: restore proxy extras parity and venv migrations * ci: persist uv path across circleci steps * fix: keep psycopg binary in default test env * docker: preserve prisma cache across stages * test: run local proxy checks through uv python * build: restore runtime deps moved into ci * build: refresh uv lock after upstream merge * fix: restore module import in test_check_migration after merge The conflict resolution imported only the function but the test body references check_migration as a module throughout. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: revert dependency promotions, remove nodejs-wheel-binaries, fix Docker layer caching - Move google-generativeai, Pillow, tenacity back to ci group (they are lazily imported and bloat the base SDK install needlessly) - Remove nodejs-wheel-binaries from extra_proxy and proxy-dev (redundant in Docker where system Node.js is already installed via apk) - Remove all nodejs-wheel node replacement and venv npm patching blocks from Dockerfiles since the wheel is no longer installed - Add --no-default-groups to CodSpeed benchmark workflow so the benchmark environment matches the old minimal pip install footprint - Apply standard uv two-phase Docker pattern: copy metadata first, install deps (cached layer), then copy source and install project - Replace CircleCI enterprise no-op with proper uv sync command Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: regenerate uv.lock after removing nodejs-wheel-binaries Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): use cache/restore instead of cache to prevent cache poisoning The old workflow used actions/cache/restore (read-only). The uv migration changed it to actions/cache (read-write), which zizmor flags as a cache poisoning risk. Restore the safer read-only variant. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): disable setup-uv built-in cache to silence cache-poisoning alert The setup-uv action enables caching by default, which zizmor flags as a cache poisoning risk. Disable it since we already use a read-only cache/restore step. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): disable setup-uv cache in publish workflow Silences zizmor cache-poisoning alert. Publishing workflow runs infrequently on protected branches so caching adds no real benefit. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(test): remove duplicate verbose_logger mock in test_check_migration The logger was patched twice — first via mocker.patch() then via mocker.patch.object(autospec=True). The second call fails because autospec cannot inspect an already-mocked attribute. Remove the redundant first patch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(ci): free disk space before Docker build in test-server-root-path The Dockerfile.non_root build ran out of disk on the CI runner. Remove Android SDK, .NET, Boost, and GHC toolchains (~12GB) to free space. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
ishaan-berri
|
61b295238b
|
cherry-pick: tag query fix + MCP metadata support (#25145)
* added support for metadata (#24261) * added support for metadata * fix: PR review - meta truthiness, BlobResourceContents mimeType, add Blob+empty meta tests Made-with: Cursor * pyproject to .25 * feat(teams): resolve access group models/MCPs/agents in team endpoints Add access_group_models, access_group_mcp_server_ids, and access_group_agent_ids to /team/info and /v2/team/list responses. These fields contain resources inherited from access groups, kept separate from direct assignments so the UI can distinguish the source. Backend: _resolve_access_group_resources() helper resolves access group resources via existing _get_*_from_access_groups() functions. UI: Teams table and detail view show direct models as blue badges and access-group-sourced models as green badges. * perf(teams): single-pass access group resolution + asyncio.gather in list endpoint - Fetch each access group object once and extract all 3 resource fields in a single pass instead of 3 separate calls (3N → N lookups) - Use asyncio.gather to resolve access groups across teams concurrently in list_team_v2 instead of sequential awaits - Add 5 unit tests for _resolve_access_group_resources * docs: add default_team_params to config reference and update examples - Add default_team_params to litellm_settings reference table in config_settings.md with all sub-fields documented - Update self_serve.md and msft_sso.md examples to include team_member_permissions, tpm_limit, and rpm_limit - Fix misleading comment that implied default_team_params only applies to SSO auto-created teams — it applies to all /team/new calls * docs: clarify that models sub-field only applies to SSO auto-created teams * fix: lazy import get_access_object to break cyclic import + short-circuit all-proxy-models display - Remove get_access_object from module-level import in team_endpoints.py and use a lazy _get_access_object wrapper to avoid cyclic dependency - Add _prisma_client is None early-exit guard in _resolve_access_group_resources - Short-circuit UI to show "All Proxy Models" when team.models is empty or contains "all-proxy-models", skipping access group model resolution * add: making organizations a select instead of read only badges * fix(ui): only send organization_id when changed and use raw initial value * fix(ui): add paginated team search to usage page filter Replace the static team dropdown on the usage page with a new TeamMultiSelect component that uses the paginated v2/team/list endpoint with debounced server-side search and infinite scroll. * fix(ui): fix imports and update placeholder for team multi select * fix(ui): wire team_id filter to key alias dropdown on Virtual Keys tab The Key Alias dropdown on the Virtual Keys page was showing aliases from all teams regardless of which team was selected. The team_id was never passed through the frontend chain to the backend /key/aliases endpoint. - Backend: add optional team_id query param to /key/aliases endpoint - networking.tsx: add team_id param to keyAliasesCall - useKeyAliases: accept and forward team_id to API call and query key - filter.tsx: pass allFilters context to custom filter components - PaginatedKeyAliasSelect: read Team ID from allFilters and pass to hook * fix(tests): correct mock targets in TestResolveAccessGroupResources Three tests were patching the non-existent `get_access_object` instead of `_get_access_object` (the lazy-import wrapper), causing AttributeError. Also added missing `prisma_client` mock so tests get past the early-exit guard and actually exercise the resolution logic. * fix: use direct attribute access with or [] fallback in _resolve_access_group_resources Replace getattr(ag, "field", []) with ag.field or [] for cleaner access and safe handling if a field is None. * fix(ui): remove model source legend from team detail view The blue/green color distinction is self-explanatory; the legend added visual clutter without providing enough value. * fix(ui): add missing access_group fields to TeamData.team_info type The TeamData interface was missing access_group_models, access_group_mcp_server_ids, and access_group_agent_ids fields, causing a TypeScript build failure. * perf(teams): batch-fetch access groups in single DB query Replace per-ID _resolve_access_group_resources loop with a single find_many call that deduplicates IDs across all teams. Removes the N+1 query pattern on cold cache for the team list endpoint. * refactor(proxy): extract helpers to fix PLR0915 violations Extract `_apply_non_admin_alias_scope` from `key_aliases`, `_resolve_team_access_group_resources` from `team_info`, and `_enforce_list_team_v2_access` from `list_team_v2` to bring each function under ruff's 50-statement limit. No behavior changes. * test(ui): update tests to match new team_id / access-group signatures - useKeyAliases, PaginatedKeyAliasSelect: add trailing `undefined` to spy matchers for the new `team_id` param on `useInfiniteKeyAliases` and `keyAliasesCall`. - EntityUsage: mock new `TeamMultiSelect` child so QueryClientProvider is not required for team-entity tests. - ModelsCell: replace the overflow-accordion test with one that verifies the new collapse-on-`all-proxy-models` behavior (no accordion, single badge). * fix(ui): send null (not '') for cleared organization_id on team update AntD <Select allowClear> returns undefined when the user clears the selection. Coalescing to "" caused the team-update payload to carry organization_id: "" instead of null, relying on the backend to coerce it. Send null directly so the intent is explicit at the source. * poetry * chore: regen poetry.lock for litellm-proxy-extras 0.4.64 bump * chore: update Next.js build artifacts (2026-04-04 17:55 UTC, node v22.16.0) --------- Co-authored-by: shivam <shivam@uni.minerva.edu> Co-authored-by: Ryan Crabbe <ryan@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * Tag query fix (#25094) * feat(tag-spend): implement separate scheduler job for daily tag spend updates * fix(docker): add g++ to build dependencies in Dockerfile * initial test cases. TODO: check scheduler init and test cases in proxy_server related to it * resolved QPS issue when redis transaction buffer is enabled * resolving circular import error flagged by greptile * fix(mypy): use Optional[str] for api_base in PydanticAI provider to match superclass signature --------- Co-authored-by: Shivam Rawat <shivam@berri.ai> Co-authored-by: shivam <shivam@uni.minerva.edu> Co-authored-by: Ryan Crabbe <ryan@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Harish <harishgokul01@gmail.com> Co-authored-by: Ishaan Jaffer <ishaan@berri.ai> |
||
|
ishaan-berri
|
b53cfe729a
|
Litellm ishaan march30 (#24887) (#25151)
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry Missing unversioned entry causes cost tracking to return $0.00 for all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI Claude models have both versioned and unversioned entries. * fix(router): skip misleading tags error when no candidates (e.g. cooldown) Return early from get_deployments_for_tag when healthy_deployments is empty so tag-based routing does not raise no_deployments_with_tag_routing after cooldown filters all deployments. Adds regression test. Made-with: Cursor * feat(oci): add embedding support and update model catalog - Add OCIEmbeddingConfig for OCI GenAI embedding models - Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini) - Add 8 embedding models (Cohere embed v3.0, v4.0) - Update documentation with embedding examples - Update pricing for all new models * test(oci): add unit tests for OCI embedding support - 17 unit tests covering OCIEmbeddingConfig - Tests for URL generation, param mapping, request/response transform - Tests for model pricing JSON completeness * style(oci): format with black and ruff * fix(oci): correct embedding request body format OCI embedText API expects inputs, truncate, and inputType at the top level of the request body, not nested under embedTextDetails. Fixed transformation and updated tests accordingly. Verified with real OCI API: 3/3 embedding models working. * docs: clarify tag routing early return and test intent Made-with: Cursor * fix(oci): address code review findings from Greptile - P1: Fix signing URL mismatch with custom api_base by accepting api_base parameter in transform_embedding_request - P2: Remove encoding_format from supported params (OCI does not support it, was silently dropped) - P2: Raise ValueError for token-array inputs instead of silently converting to string representation - Add test for token-list rejection * fix(mcp): add STS AssumeRole support for MCP SigV4 authentication MCPSigV4Auth only supported static AWS credentials or the boto3 default credential chain. Production Kubernetes environments typically authenticate via IAM role assumption (sts:AssumeRole), which was not possible. Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole to obtain temporary credentials before signing requests. Explicit keys, if also provided, are used as the source identity for the STS call; otherwise ambient credentials (pod role, instance profile) are used. * fix: stop logging credential values and add missing redaction patterns Replaces raw credential values in debug/error log messages with boolean presence checks or type names. Adds PEM block, GCP token, JWT, SAS token, and service-account blob patterns to the redaction filter. Fixes private_key pattern to capture full PEM blocks instead of stopping at the first whitespace. Addresses: Vertex AI credential JSON (including RSA private key) being logged to stderr on health check failures. * fix: log only field names for UserAPIKeyAuth, not full object * style: apply black formatting to experimental_mcp_client/client.py * style: fix black/isort formatting and mypy error in proxy_server.py - Fix black formatting in experimental_mcp_client/client.py (done in prev commit) - Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py - Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py * fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue When test_proxy_cli.py tests run before test_check_migration.py in the same xdist worker, litellm.proxy.db.check_migration is already in sys.modules. Patching litellm._logging.verbose_logger has no effect on the already-bound reference. Patch the correct target (check_migration.verbose_logger) and import the module before patching so the order doesn't matter. * fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature --------- Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com> Co-authored-by: Milan <milan@berri.ai> Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
michelligabriele
|
92b8e1acf8 | address greptile review: async sleep, SIGKILL Windows guard, trailing newlines | ||
|
michelligabriele
|
1f04fa2461 | fix(proxy): kill orphaned prisma engine subprocess on failed disconnect | ||
|
yuneng-jiang
|
8ecac84789
|
Revert "feat(proxy): add Prisma DB pool and engine health metrics to Promethe…"
This reverts commit
|
||
|
ohadgur
|
0bb26c3f1b
|
feat(proxy): add Prisma DB pool and engine health metrics to Prometheus (#22655)
* feat(proxy): add Prisma DB pool and engine health metrics to Prometheus Add a PrismaMetricsCollector that periodically queries pg_stat_activity and the Prisma engine process to expose connection pool and engine health as Prometheus gauges/counters. Auto-enabled when prometheus_system is in service_callback. New metrics: - litellm_db_pool_active_connections (Gauge) - litellm_db_pool_idle_connections (Gauge) - litellm_db_pool_total_connections (Gauge) - litellm_db_pool_waiting_connections (Gauge) - litellm_db_engine_up (Gauge) - litellm_db_engine_restarts_total (Counter) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address Greptile review feedback - Only increment engine_restarts counter on heavy reconnects (engine actually dead), not lightweight network-blip reconnects - Fix potential KeyError in _get_or_create_gauge/counter fallback path when REGISTRY._names_to_collectors is absent - Rename litellm_db_pool_waiting_connections to litellm_db_pool_lock_waiting_connections to clarify it measures lock contention, not pool slot queuing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: warn when prometheus_system enabled but watchdog disabled Log a warning when users have prometheus_system in service_callback but PRISMA_HEALTH_WATCHDOG_ENABLED=false, since DB pool and engine metrics won't be collected in that configuration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * ci: retrigger CI checks Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: use labeled gauge for DB pool connection metrics Replace 3 separate pool gauges (active, idle, total) with a single `litellm_db_pool_connections` gauge using a `state` label. This is more Prometheus-idiomatic and exposes all pg_stat_activity states (active, idle, idle in transaction, etc.) without ambiguity about what "total" includes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address Greptile review — stale labels and fallback re-registration - Zero out known pg_stat_activity states that are absent from the current query result, preventing stale gauge values from persisting. - Simplify _get_or_create_gauge/counter by removing the fallback loop that could re-register an already-registered metric (ValueError). - Add test for stale label clearing across collection cycles. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: include "unknown" in _PG_STATES for stale label clearing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: collect immediately on start and consolidate into single query - Move sleep to end of loop so metrics appear on /metrics immediately after startup instead of after a 30s delay. - Combine pool state and lock waiting queries into a single SQL query using conditional aggregation, halving per-cycle DB overhead. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: prevent tight spin loop on collection error Move asyncio.sleep outside the try/except so it always executes even when _collect_engine_health() or _collect_pool_metrics() raises. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: add multiprocess_mode to _get_or_create_gauge initialization - Include `multiprocess_mode` parameter to properly support multiprocessing in Gauge creation. - Ensure consistent behavior for labeled and unlabeled Gauges. * fix: handle invalid env var and document watchdog prerequisite - Add try/except ValueError for PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS to prevent proxy startup crash on non-numeric values (e.g. "30s") - Document that DB metrics require both prometheus_system callback and PRISMA_HEALTH_WATCHDOG_ENABLED=true Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use defensive null coalescing for query_raw row values Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: add invalid env var fallback test and fix mock signature - Add test for non-numeric PRISMA_METRICS_COLLECTION_INTERVAL_SECONDS - Add **kwargs to mock _patched_get_or_create_gauge for forward compat Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Krish Dholakia
|
cf439c269c
|
Agents - add max budget + tpm/rpm limiting per agent AND per agent session (#22849)
* feat: enforce x-litellm-trace-id in header, if required * feat: update spend for agent * refactor: update agent table to follow similar format as other entities - also add a spend column - allows us to see spend of an agent * fix: cleanup ui * feat: return spend on agent endpoints * feat: scope pr * feat(agents/): support budgets + rate limiting on agents + agent sessions * fix: address PR review feedback - Add missing tpm_limit, rpm_limit, session_tpm_limit, session_rpm_limit columns to root schema.prisma to match proxy and extras schemas - Add backwards-compatible fallback to key metadata for max_iterations so existing users don't silently lose enforcement Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: qa'ed RPM limiting on agents --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
9d9a59190c |
Use passed general_settings parameter instead of global import
The validation method now reads use_redis_transaction_buffer directly from the passed general_settings dict rather than delegating to RedisUpdateBuffer._should_commit_spend_updates_to_redis() which imports the global. Tests simplified to remove unnecessary patching. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
3a15e1cc2e |
[Fix] Block proxy startup when use_redis_transaction_buffer is enabled without Redis cache
When `use_redis_transaction_buffer: true` is set in general_settings but no Redis cache is configured in litellm_settings, the proxy starts successfully but silently drops all spend tracking data. This adds a startup validation that raises a clear error, preventing the proxy from running in a broken state. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ishaan Jaff
|
1f412bc6d8
|
[Feat] Add Tool Policies for AI Gateway (#22732)
* fix: fix ui render * fix: fix minor bugs * refactor: use prisma functions instead of raw sql (safer) * fix(add-new-tiles-to-tool-policies): allow developer to see what's available * feat: ensure tool allowlist runs correctly for tool names + mcp's * refactor: more ui improvements * feat: working key tool blocking * feat(tools): show tool logs * refactor: backend code improvements * refactor: improve log viewer for tools * fix: address PR review feedback for tool access control - Add missing blocked_tools column to root schema.prisma (schema drift) - Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately - Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: race condition in permission resolution and remove duplicate allowlist check - Use atomic update_many with object_permission_id=None to prevent concurrent requests from creating orphaned permission rows and losing tool blocks - Remove duplicate allowed_tools enforcement from guardrail (already enforced in auth layer via check_tools_allowlist) - Move inline uuid import to module level Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * update to account for userAgent * UI - Add ToolDetails * input/output policy * LiteLLM_PolicyAttachmentTable * LiteLLM_PolicyAttachmentTable * fix: add _enqueue_tool_registry_upsert * fix: tool mgmt endpoints * tool mgmt endpoints * Update tests/test_litellm/proxy/db/test_tool_registry_writer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update tests/test_litellm/proxy/db/test_tool_registry_writer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Update tests/test_litellm/proxy/db/test_tool_registry_writer.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy - Migrate root schema.prisma LiteLLM_ToolTable from call_policy to input_policy/output_policy, add missing user_agent and last_used_at columns (now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras) - Fix SpendLogToolIndex comment across all three schema files - Fix all call_policy references in test_tool_registry_writer.py: swapped update_tool_policy arguments, wrong get_tools_by_names return type assertions, _mock_tool_row setting call_policy instead of input_policy Addresses Greptile review feedback on PR #22732. Made-with: Cursor --------- Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Ryan Crabbe
|
1c1b9cda42 |
merge: resolve conflict with origin/main in test_db_spend_update_writer.py
Keep both batch update tests (from this branch) and pipeline test (from main). |
||
|
Ryan Crabbe
|
e1e05d27f8 | Merge remote-tracking branch 'origin/main' into litellm_batch_update_database_tasks | ||
|
ryan-crabbe
|
9857643eb4
|
Merge pull request #22044 from ryan-crabbe/litellm_redis_pipeline_spend_updates
Litellm redis pipeline spend updates |
||
|
Ishaan Jaff
|
6ee50ff73e
|
feat(proxy): tool policies - auto-discover tools + policy enforcement guardrail (#22041)
* feat(proxy): tool policies - auto-discover tools, manage policies, guardrail enforcement
- New LiteLLM_ToolTable in schema.prisma to store discovered tools
- Auto-discovery: tools seen in LLM responses get upserted via ToolDiscoveryQueue
(hooks into DBSpendUpdateWriter, same pipeline as spend tracking)
- Management endpoints: GET /v1/tool/list, GET /v1/tool/{name}, POST /v1/tool/policy
- ToolPolicyGuardrail: blocks tool_calls in responses based on policy setting
- UI: Tool Policies page under Guardrails section with policy selector,
filters by policy/team/key, live tail, sortable table
- Unit tests for queue, writer, endpoints, guardrail
* feat(tool-policies): track call_count + discover tools from request body and /messages API
- Add call_count column to LiteLLM_ToolTable; incremented on every flush
- Extract tools from request body too (not just response tool_calls):
- OpenAI /chat/completions: tools[].function.name
- Anthropic /messages pass-through: request_body.tools[].name
- Show call_count column in UI table (sortable)
- UI: drop dual_llm option, keep only trusted/blocked
* fix: address greptile review feedback
- Remove redundant @@index([tool_name]) from schema.prisma (tool_name has @unique which already creates an index)
- Replace gen_random_uuid()::text with str(uuid.uuid4()) for portability
- Rewrite test_tool_registry_writer.py to mock execute_raw/query_raw (actual implementation) instead of Prisma model methods
- Fix test patches in test_tool_management_endpoints.py to target source modules since imports are inside function bodies
- Add "Tool Policies" page title to ToolPolicies.tsx
* fix: address greptile review round 2
- Replace NOW() with Python datetime parameter in tool_registry_writer (SQLite portability)
- Fix cache key collision in tool_policy_guardrail: use null-byte separator instead of colon
- Remove type==function filter from request-side tool extraction to match response-side behavior
- Clear seen_tool_names on flush so call_count increments per batch cycle not per pod lifetime
* fix: address greptile review round 3
- Fix test_seen_names_persist_across_flushes to match actual per-flush-cycle behavior
- Update module docstring in tool_discovery_queue.py to accurately describe flush behavior
- Add created_at/updated_at to raw SQL INSERT in batch_upsert_tools and update_tool_policy
* fix: cache tool policies per tool name not per combination
Previously the cache key was built from the full set of tool names in a
request, so each unique combination of tools got its own cold cache entry
and triggered a separate DB query. With N distinct tools across requests
this was effectively a DB hit on every request.
Now each tool name is cached individually. Cache hits are checked per
tool, only missing tools are fetched from DB in a single batch query,
and each result is cached separately. Once a tool's policy is warm,
any subsequent request using that tool benefits from the cache regardless
of what other tools are in the request.
* Update ui/litellm-dashboard/src/components/ToolPolicies.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
|
||
|
Ryan Crabbe
|
98b4964330 |
perf(proxy): pipeline Redis RPUSH/LPOP in spend update cycle
Replace 14 sequential Redis round-trips (7 RPUSH + 7 LPOP) per spend update cycle with 2 pipelined calls (1 RPUSH pipeline + 1 LPOP pipeline). This reduces connection pool contention at scale (50+ pods). - Add RedisPipelineRpushOperation and RedisPipelineLpopOperation TypedDicts - Add async_rpush_pipeline() and async_lpop_pipeline() to RedisCache - Refactor store_in_memory_spend_updates_in_redis() to use pipeline - Add get_all_transactions_from_redis_buffer_pipeline() for batched drain - Update _commit_spend_updates_to_db_with_redis() to use pipeline drain - Existing individual methods preserved for backward compatibility |
||
|
Ryan Crabbe
|
4c5963fdb9 | test: exercise production deepcopy path in agent payload test | ||
|
Ryan Crabbe
|
86ec2fbf84 |
perf(proxy): batch 11 create_task() calls into 1 in update_database()
Replace 11 separate asyncio.create_task() calls per request with a single batched task that runs all spend-update helpers sequentially. This reduces task scheduling overhead at high RPS (11,000 -> 1,000 tasks/sec at 1K RPS) and cuts 5 copy.deepcopy(payload) calls to 1 shared copy. Also fixes a mutation bug where the daily agent spend handler received the raw payload without deepcopy, unlike all other daily helpers. |
||
|
yuneng-jiang
|
a9c44d8530 | adjust default aggregation threshold | ||
|
Julio Quinteros Pro
|
fb8b11cc0a |
fix(tests): use counter-based mock for time.time in prisma self-heal test
The test used a fixed side_effect list for time.time(), but the number of calls varies by Python version, causing StopIteration on 3.12 and AssertionError on 3.14. Replace with an infinite counter-based callable and assert the timestamp was updated rather than checking for an exact value. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ishaan Jaff
|
a939fa37ae
|
fix(proxy): restore broad is_database_connection_error; add is_database_transport_error for reconnect (#21796)
Any PrismaError should be treated as a DB connection error for the allow_requests_on_db_unavailable feature and 503 responses. The narrow keyword-based check is now in is_database_transport_error, which is what the reconnect logic in auth_checks.py should use. Fixes test_delete_access_group_503_on_db_connection_error and test_handle_authentication_error_db_unavailable failures caused by PR #21706 narrowing is_database_connection_error. |
||
|
Ishaan Jaff
|
e0129710c8
|
fix(proxy): self-heal Prisma connection for auth and runtime (#21706)
* fix(proxy): add prisma reconnect primitive and db watchdog * fix(proxy): start and stop prisma watchdog in lifecycle * fix(auth): retry key lookup once after prisma reconnect * test(proxy): add prisma self-heal watchdog coverage * test(auth): cover reconnect-once behavior for key lookup * refactor(auth): extract db reconnect helper and remove inline import * fix(proxy): apply reconnect cooldown after attempt and add auth timeout path * fix(auth): bound reconnect latency on key lookup path * test(auth): assert reconnect timeout argument in key lookup * test(proxy): verify reconnect cooldown timestamp set after attempt * fix(proxy): harden prisma reconnect cycle semantics * test(proxy): cover watchdog reconnect + timeout budget * fix(proxy): bound watchdog probe and reconnect paths * test(proxy): cover watchdog timeout and probe behavior * fix(proxy): narrow prisma db connection error classification * fix(proxy): add auth reconnect lock timeout budget * fix(auth): pass lock timeout for db reconnect retries * test(proxy): cover narrow prisma connection error detection * test(proxy): add reconnect lock-timeout behavior coverage * test(auth): assert reconnect lock timeout argument * fix(proxy): avoid lock leak race in reconnect lock timeout path * test(proxy): cover reconnect lock-timeout race cleanup |
||
|
yuneng-jiang
|
6097905e55 |
[Feature] Track key last active timestamp
Virtual keys only track created_at and updated_at, which don't indicate when a key was last used. This adds a last_active field that gets updated during the async batch spend update, giving admins visibility into which keys are actively being used. Changes: - Add last_active DateTime? to VerificationToken and DeletedVerificationToken in all 3 schema files and Python types - Set last_active in the batch key spend update alongside spend increment - Add Last Active column to virtual keys UI table with info popover and hover tooltip showing full date/time with timezone Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Emerson Gomes
|
06e7bfce2e | perf(spend): avoid duplicate daily agent transaction computation (#21187) | ||
|
Ishaan Jaffer
|
9ce7871d8a | test_queue_flush_limit |