4151 commits

Author	SHA1	Message	Date
mateo	2fe152a1d2	fix(proxy): only schedule the deprecation loop when alerting is configured ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-11 00:05:24 +00:00
Deepanshu Lulla	363d56f917	feat(proxy): add per-deployment keepalive_seconds SSE heartbeat to prevent load-balancer timeout on long streams (#34423) ... * feat(proxy): add per-deployment keepalive_seconds SSE heartbeat for long-running streams Adds _iter_with_keepalive, _keepalive_from_deployment_config, and _resolve_keepalive_seconds helpers to proxy_server.py. When enabled (keepalive_seconds > 0 in request body or deployment litellm_params), async_data_generator emits ': ping\n\n' SSE comment frames every N seconds during idle upstream intervals, preventing load-balancer idle-timeout drops on long chain-of-thought reasoning streams. The hot path (keepalive_seconds absent or 0) is a plain async-for with no per-chunk Task wrapping — zero overhead. Includes 8 new unit tests covering sentinel emission, hot-path pass-through, early-close cleanup, priority resolution, deployment-config lookup, and end-to-end heartbeat emission through async_data_generator. Registers keepalive_seconds in all_litellm_params (types/utils.py) so the parameter is not stripped from request bodies. Adds the field to LiteLLMParamsTypedDict and GenericLiteLLMParams (types/router.py) so deployment YAML config is parsed and validated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(proxy): narrow BaseException to CancelledError to fix BLE001 strict lint gate * fix: use explicit None check instead of truthiness in keepalive_seconds extraction `float(raw or 0)` would treat any falsy value (including the integer 0) as absent and substitute 0.0 before float() saw it. Replace with `float(raw) if raw is not None else 0.0` so a caller-supplied zero is correctly passed through to the `value <= 0` guard that disables keepalive, rather than being silently overwritten. * fix(proxy): don't guess a deployment's keepalive_seconds when model_id is missing When a streaming response lacks _hidden_params.model_id, the fallback that looks up keepalive_seconds by model_name previously returned the first configured deployment's value, which could apply the wrong interval (or override an explicit disable) when multiple deployments share the same model_name with different keepalive_seconds settings. Only resolve the fallback when every deployment agrees; otherwise leave it unset. * fix(proxy): also treat an unset keepalive_seconds as disagreement in the fallback The model_name fallback for keepalive_seconds only compared configured values, filtering out deployments that leave the field unset entirely. That meant a deployment with no keepalive_seconds configured could still inherit a sibling deployment's interval when model_id is unavailable. Compare the raw per-deployment value (including None for unset) so an unconfigured deployment never silently adopts another's heartbeat. * fix(proxy): deployment-level keepalive_seconds: 0 is a hard disable clients can't override Previously an authenticated client's request-level keepalive_seconds always took precedence over the deployment default, including when a deployment operator explicitly set keepalive_seconds: 0 to disable heartbeats. That let any client re-enable heartbeats for a deployment the operator opted out of, using them to keep an idle-looking stream alive past a load balancer's idle timeout and hold a parallel-request slot open longer than intended. Treat an explicit deployment-level 0 as authoritative: resolve the deployment's configured value first, and short-circuit to disabled before ever looking at the request body if the deployment hard-disabled it. * fix(proxy): a stale (unresolvable) model_id must not fall through to model_name guessing A populated _hidden_params.model_id names the specific deployment that served a stream. If that ID no longer resolves (e.g. a deployment removed by a config reload mid-stream), the resolver was falling through to the model_name-based fallback, letting a currently-live sibling deployment's keepalive_seconds silently apply to a stream it never served. Return None once a populated model_id fails to resolve, rather than degrading to a guess. * fix(proxy): keepalive_seconds is operator-only by default; require deployment opt-in for client override A security review flagged that a client's request-level keepalive_seconds could unilaterally enable heartbeats for any deployment, even one that never configured keepalive_seconds at all, letting an authenticated client defeat load-balancer idle timeouts and hold a parallel-request slot open for longer than the deployment operator ever intended, with no way for the operator to prevent it short of explicitly setting keepalive_seconds: 0. Add allow_client_keepalive_override (default False) to LiteLLMParamsTypedDict and GenericLiteLLMParams. _resolve_keepalive_seconds now ignores the request body's keepalive_seconds entirely unless the resolved deployment explicitly grants override permission; only the deployment's own configured value (or disabled, if unset) applies otherwise. An explicit deployment-level 0 still takes priority over everything, including a grant of override permission. * fix(proxy): register allow_client_keepalive_override in all_litellm_params Caught during live proxy verification against the real Anthropic API: allow_client_keepalive_override was added to LiteLLMParamsTypedDict and GenericLiteLLMParams but never registered in all_litellm_params, so it leaked straight through into the provider request body as an unrecognized field. Anthropic rejected every call on a deployment that had this field configured with a 400 ("Extra inputs are not permitted"), regardless of its value. Register it alongside keepalive_seconds so it's stripped before reaching the provider, matching what keepalive_seconds already does. * feat(proxy): support keepalive_seconds via x-litellm-keepalive-seconds header Some clients (e.g. the Vercel AI SDK) can set custom headers more easily than extra JSON body fields. Add x-litellm-keepalive-seconds, following the existing x-litellm-timeout/x-litellm-stream-timeout/x-litellm-num-retries convention in LiteLLMProxyRequestSetup: the header merges into the same data["keepalive_seconds"] field the request body already populates, so it goes through the exact same _resolve_keepalive_seconds precedence and the allow_client_keepalive_override gate -- a header can't enable heartbeats for a deployment that hasn't opted in any more than the body field can. Verified live against the real Anthropic API: the header produces real heartbeats on an opt-in deployment (88 pings over a genuine long-reasoning stall) and is silently ignored on a deployment without override permission (0 pings), matching the existing body-field behavior exactly. * chore: rebase onto litellm_internal_staging, drop unrelated credential_migration.py reformat, fix budget-ratchet drift Rebased onto the current litellm_internal_staging (merge-base was 5 days stale). Dropped the now-redundant schema.d.ts-only regen commit entirely (the new base's own schema.d.ts already supersedes it) and regenerated schema.d.ts fresh against the new base. Reverted litellm/proxy/management_endpoints/credential_migration.py to exactly match litellm_internal_staging: it was a pure reformat with no semantic change, unrelated to this PR, flagged by review as unnecessary noise in an encryption-migration file. Fixed two lint-budget-ratchet failures caused by the base's ceilings tightening since this branch last synced (other merged work lowered ANN401/LIT001 budgets; this code was previously under budget and didn't change): - _iter_with_keepalive's aiter param: Any -> AsyncIterator[Any], a real narrowing (it's always the result of .__aiter__()). - _keepalive_from_deployment_config/_resolve_keepalive_seconds's request_data param: dict[str, Any] -> Mapping[str, Any], matching the existing read-only-dict convention already used elsewhere in this file (_apply_ssrf_general_settings, _build_redis_usage_cache, etc.) for params that are only ever read, never mutated. - response/raw params: dropped the explicit `Any` annotation to match async_data_generator's own (deliberately unannotated) `response` param, its actual caller. - litellm_pre_call_utils.py's new headers param: dict -> Mapping[str, str], same read-only-dict rationale. * fix(proxy): freeze the transient collections in the keepalive helpers _iter_with_keepalive and _keepalive_from_deployment_config built a set literal for asyncio.wait, a set comprehension for the per-deployment config-agreement check, and two dict-literal fallbacks, all flagged by the LIT002 mutable-collection-construction gate. Switched to a tuple for asyncio.wait, a frozenset-wrapped generator plus next(iter(...)) for the config check, and a shared MappingProxyType({}) empty mapping for the fallbacks. * fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback Greptile P1: when a streaming request falls back from model group A to group B and the response's _hidden_params carries no model_id, _keepalive_from_deployment_config fell straight through to guessing via request_data["model"], which still names the pre-fallback group A since the fallback handler mutates its own local **kwargs copy, not this dict. request_data[metadata|litellm_metadata]["model_info"]["id"], by contrast, is mutated on this same dict by Router._update_kwargs_with_deployment on every attempt including fallbacks (the same source ProxyLogging._build_litellm_call_info uses for logging), so check it before falling through to the model-name guess. Added two regression tests that fail on the prior code (assert get_model_list is never called once metadata.model_info.id resolves) and pass with the fix. * Revert "fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback" This reverts commit d7790678645695b20f25880315238b49c31a9143. * fix(proxy): satisfy the new LIT010/ANN001 gates in the keepalive helpers litellm_internal_staging picked up a LIT010 (every local/module variable must be declared Final unless it's genuinely rebound) and tightened ANN001 (missing parameter annotations) since this branch last synced. Annotated every single-assignment local and module constant with Final, suppressed pending's loop-carried reassignment with # rebind-ok, and typed the previously-bare response/raw parameters as object with isinstance narrowing at their use sites instead of cast (LIT006 discourages cast; validate into a concrete type instead). Also swapped the hand-rolled getattr(response, "_hidden_params", None) + isinstance(hidden, dict) check for the existing get_hidden_params_dict() helper already used for this exact purpose elsewhere in this file and in common_request_processing.py. * fix(proxy): re-resolve keepalive_seconds per chunk to track mid-stream fallback Greptile P1: the router can perform a mid-stream fallback to a different deployment partway through a stream (MidStreamFallbackError in router.py), and Router._apply_fallback_hidden_params_to_item merges the fallback deployment's hidden params onto every subsequent chunk. But _resolve_keepalive_seconds was only ever called once, before iteration started, against the pre-fallback response wrapper, so a stream that fell back to a deployment with a different (or disabled) keepalive policy kept using the original deployment's interval for the rest of the stream. _iter_with_keepalive now takes a resolve_keepalive_seconds(item) callback and re-resolves after every real chunk using that chunk's own _hidden_params (which do carry the fallback deployment's identity), rather than trusting the value picked before iteration began. Updated the three existing timing tests to inject a constant-returning resolver, since they pin the sentinel/cancellation mechanics rather than re-resolution, and added two regression tests (interval lowered and raised mid-stream) that fail against the prior static-resolve signature and pass with the fix. * fix(proxy): keep re-resolving keepalive even when a stream starts disabled Greptile P1: a stream that starts on a deployment with keepalive off (or unset) skipped _iter_with_keepalive entirely at the call site, so a mid-stream fallback to a deployment that enables it never got a chance to activate heartbeats for the rest of that stream, risking the exact load-balancer idle-timeout this feature exists to prevent. _iter_with_keepalive now has an internal fast path for keepalive_seconds <= 0 that still re-resolves after every chunk (no asyncio.create_task/wait overhead while inactive, same cost as a bare async for), so activation from a disabled start works the same way deactivation and interval changes already do. The caller now only skips wrapping entirely when there's no router to ever fall back through in the first place (llm_router is None), rather than whenever the first chunk's deployment happens to start with keepalive off. Added a regression test that starts keepalive_seconds=0, has the resolver enable a short interval on a later chunk, and asserts sentinels appear afterward; it fails against the prior call-site-gated code and passes with the fix. * perf(proxy): memoize keepalive resolution per chunk's model_id _resolve_keepalive_seconds ran a full llm_router.get_deployment() Pydantic rebuild after every streamed chunk, even when keepalive was unconfigured anywhere in the deployment list, since async_data_generator wraps every stream once a router exists. Caching the result by model_id keeps mid-stream fallback re-resolution correct while paying the router lookup once per deployment instead of once per token. * fix(proxy): expire cached keepalive resolution after a bounded TTL veria-ai flagged that caching by model_id alone lets an already-in-flight stream keep evading a live config reload (deployment removed, keepalive disabled, or client override revoked) for the rest of the stream. Expiring the memo after _KEEPALIVE_CACHE_TTL_SECONDS bounds that window instead of freezing the resolved value for the stream's full lifetime, while still avoiding a full deployment rebuild on every chunk in the steady state. Also fixes add_litellm_data_for_backend_llm_call's now-required request_data kwarg in the header-merge test, picked up by rebasing onto litellm_internal_staging. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-08-10 16:47:17 -07:00
mateo	8f1aea5e0a	refactor(proxy): tighten model deprecation typing and cover the endpoint ... Drops Any-typed router plumbing, immutable bucketing, generated dashboard API types, and adds endpoint plus resolution-fallback tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 22:58:35 +00:00
Cursor Agent	f249356e16	feat(proxy): proactive model deprecation alerts and /model/deprecations endpoint ... Surfaces deprecation_date metadata that is already shipped in model_prices_and_context_window.json so operators get lead time to migrate before a provider sunsets a model. - New helper litellm.proxy.common_utils.model_deprecation classifies the router's configured models into deprecated / imminent / upcoming buckets. Resolution order: explicit model_info.deprecation_date > model_info.base_model > litellm_params.model. - New GET /model/deprecations (and /v1/model/deprecations) endpoint returns a ModelDeprecationResponse, gated by user_api_key_auth. - New AlertType.model_deprecation_warnings (in DEFAULT_ALERT_TYPES) plus SlackAlerting.send_model_deprecation_alert dispatches a Slack message for deprecated/imminent models. Severity is High when any model is already past its date, Medium when only imminent. - ProxyLogging.startup_event schedules a daily background task (_run_scheduled_deprecation_check) when the alert type is enabled. The interval is configurable via LITELLM_MODEL_DEPRECATION_CHECK_INTERVAL and the warn window via LITELLM_MODEL_DEPRECATION_WARN_DAYS. - Tests: 16 unit tests for the helper plus 4 for the Slack hook in tests/test_litellm/. Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>	2026-08-10 22:42:25 +00:00
fancybear-dev	f1ed4690bb	fix(proxy): treat SAML as configured in UI SSO detection (#36196) ... * fix(proxy): treat SAML as configured in UI SSO detection _has_user_setup_sso only checked OAuth client IDs, so SAML-only setups left /.well-known/litellm-ui-config sso_configured=false and the login button gray even when SAML IdP metadata was set. Include SAML_IDP_METADATA_URL / SAML_IDP_METADATA_XML so UI discovery matches the login redirect path. * chore: adhere to comment policy Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: apply suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>	2026-08-10 15:41:27 -07:00
ryan-crabbe-berri	76ad1c319d	feat(proxy): add GET /v1/indexes to list vector store indexes (#36289) ... * fix(scripts): stop type-discipline checker reading Literal strings as forward refs The checker re-parsed every string constant inside an annotation as a forward reference, so Literal["list"] was counted as the mutable list type. Skip Literal subtrees and ratchet the LIT001 ceiling down to the corrected count. * fix(proxy): keep lazy openapi snapshot fragments for transitively imported features generate_snapshot skipped register_fn for any feature module already in sys.modules, so a module pulled in transitively by an earlier feature never mounted its routes and its fragment silently vanished on regen (vector_store_management). Route collection also matched path_prefixes only, dropping suffix-matched routes from fragments. Register every feature and collect routes with feat.matches, mirroring the runtime loader. * feat(proxy): add GET /v1/indexes to list vector store indexes /v1/indexes was POST-only, so indexes created through it could never be viewed again. Add an admin-only list endpoint returning the stored index rows newest first, fix the stale index_create docstring curl, and regenerate the lazy openapi snapshot and dashboard schema types. * chore(proxy): defer lazy openapi snapshot catch-up regen to a follow-up Reverts _lazy_openapi_snapshot.json and schema.d.ts to the staging versions. The snapshot was months stale, so regenerating it here buried the actual change under ten thousand generated lines. A follow-up will land the regen together with CI enforcement that keeps the snapshot current. Until then GET /v1/indexes is served but absent from the dashboard's generated types, which the UI step needs anyway. * fix(proxy): use Annotated dependency to avoid new B008 violation	2026-08-10 15:09:59 -07:00
ryan-crabbe-berri	c40828509b	fix(reset_budget_job): atomic budget cascade with chunked reset scans (#36287) ... * fix(reset_budget_job): advance budget_reset_at atomically with the spend cascade A postgres timeout mid-cascade previously left LiteLLM_BudgetTable rows stamped for the next window while team member, enduser, org and tag spend stayed at cap, so every later tick skipped them until the window rolled over. All cascade writes and the budget_reset_at advance now share one prisma batch transaction; a failed run persists nothing and the rows stay due for the next ~10 minute tick. Cache and counter invalidation runs only after commit, and the catch-all enduser log line now names the cascade. * fix(reset_budget_job): elect one runner per tick and chunk the reset scans Every pod and worker previously ran the reset job every ~10 minutes, each fetching every expired row with no limit and writing one giant transaction at the same calendar-aligned boundary; that concurrency is what piled up postgres lock contention and timeouts. The job now takes the shared PodLockManager redis lock (no redis keeps the old behavior), and each phase walks its due rows in 500-row chunks, one transaction per chunk, stopping when a chunk is short, makes no forward progress, or hits the per-run cap; leftovers wait for the next tick. * chore(lint): ratchet budget ceilings down for fixed violations * fix(reset_budget_job): harden chunk loop, fail open on redis errors, heartbeat the lock Review fixes on the two prior commits. Reset scans now skip rows with no budget_duration, so permanently due rows can neither starve a phase nor have a lifetime cap zeroed every tick. Chunk progress counts rows whose new budget_reset_at actually cleared the cutoff, so a zero-length duration cannot burn the per-run chunk cap. A failed lock acquire only skips the run when another pod verifiably holds the lock; a broken redis runs unguarded instead of silently disabling resets fleet-wide. Partial row failures report real progress and fire the failure hook without killing the phase. The leader re-asserts the lock between phases and stops if another pod took over, and the budget window advance uses update_many so a tier deleted mid-chunk cannot abort the transaction. Lint budget ceilings re-ratcheted for the net-fixed violations. * fix(reset_budget_job): renew the leader lease and reject non-positive budget durations Bot review follow-ups. PodLockManager now extends the lock TTL when the holding pod re-acquires, via an atomic compare-and-expire script with a plain SET fallback, so a run longer than the TTL keeps its lease instead of silently sharing the job with another pod. The positive-duration validation that team member endpoints already had is hoisted to management common_utils and applied to key, internal user, budget, customer and team intake, so a tenant can no longer create zero-duration budgets whose permanently due rows starve other tenants' resets. Such durations now return 400 at intake; existing rows are untouched. * refactor(reset_budget_job): defer leader election to a follow-up PR * fix(reset_budget_job): satisfy strict lint gates String defaults for the two getenv calls (PLW1508) and the chunk outcome returns moved to try/else (TRY300).	2026-08-10 14:42:36 -07:00
yucheng-berri	ade5a425e8	fix(proxy): isolate guardrail load failures per row (#36432) ... * fix(proxy): isolate guardrail load failures per row One DB guardrail row that fails to initialize aborted the whole _init_guardrails_in_db loop, so a single typo'd guardrail type or a missing required param left the proxy running with zero DB guardrails registered and requests that should have been blocked reaching the provider. Catch per row around sync_guardrail_from_db, log the guardrail name, id and error, and continue with the remaining rows. The failing row's id is still added to db_guardrail_ids before the attempt so reconcile_db_guardrails cannot mistake a live row for a deleted one. * test(proxy): drop inline note and record reconcile via a handler double Replaces the patched bound method with an InMemoryGuardrailHandler subclass that records what reconcile_db_guardrails received, so the test injects a double instead of swapping a method on a live object.	2026-08-10 14:37:09 -07:00
yuneng-jiang	3726bceb53	Merge pull request #36336 from BerriAI/litellm_/standard-lists-api-d1dc4a ... test(proxy): guard management_v1 against fastapi names removed in supported releases	2026-08-10 13:39:59 -07:00
Devin AI	584e88073d	chore: merge litellm_internal_staging	2026-08-10 19:53:28 +00:00
Yassin Kortam	ade805ef0c	feat(rate limiting): configurable estimated output tokens per key, team and model (#36143)	2026-08-10 12:51:14 -07:00
Devin AI	f0c3d8dcda	chore: merge litellm_internal_staging into litellm_anthropic_fast_mode_speed_usage ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-10 19:28:09 +00:00
yucheng-berri	e014b341c8	feat(ptu): gate PTU flat-cost attribution behind an opt-in env var (#36138) ... LITELLM_ENABLE_PTU_COST_ATTRIBUTION, read through get_secret_bool and defaulting to false, makes the whole PTU flat-cost feature inert unless an operator opts in. The daily rollup cron is not registered at all, so no sentinel row is ever written; /model/new and /model/{id}/update reject a request that carries any PTU model_info field with a 400 naming the fields and the env var rather than dropping them; the daily activity read path reports zero flat cost; and the model add and edit forms hide the four PTU inputs. The read gate lives where flat cost enters SpendMetrics rather than in the aggregated SQL select. /team/daily/activity, the endpoint the Usage page reads, is served by the paginated find_many path and never runs that query, so forcing the select to a constant zero would have left the reporting surface that matters still showing flat cost. Sentinel row filtering is deliberately not gated. An operator can enable the flag, accrue rows under the __ptu_flat_cost__ api_key, then disable it, and those rows stay in LiteLLM_DailyTeamSpend; gating the filter too would surface the sentinel as a bogus api_key and mint a provider bucket for its empty provider. Response fields keep their shape and report 0.0, so typed clients are unaffected, and the migration and the ModelInfo field declarations are untouched. The write gate reads the incoming request rather than the merged deployment, so a model configured during an earlier opt-in stays editable, and the edit form drops the PTU keys from the payload instead of sending nulls that would clear stored config. The dashboard reads the flag from a read-only enable_ptu_cost_attribution key on /get/ui_settings, computed from the environment on every read. It is deliberately not an allowlisted persisted setting, and PATCH /update/ui_settings rejects it with a 400, so an admin cannot flip an env-gated feature from the UI. Two review findings on the gate itself. The PTU clear loop now runs only when the feature is enabled: the write gate rejects a value but lets an explicit null through, and a client round-tripping a model_info blob sends the PTU keys as nulls, so a disabled proxy would have quietly erased a billing configuration set up during an earlier opt-in. Disabling pauses PTU rather than discarding its setup. And the dashboard flag is re-read every thirty seconds instead of the hour the other UI settings use, since those are persisted records while this one tracks the proxy process; a restart that flips the variable would otherwise leave the model form offering inputs the backend now rejects. The flag is polled rather than only marked stale, since a form that stays mounted and focused never refetches on its own. The read gate checks the row before the flag. It runs once per metric accumulation and a record fans out across roughly a dozen breakdowns, while the flag reads through the secret manager uncached, so consulting it for every accumulation put thousands of lookups on a shared endpoint that made none before. Only a row actually carrying flat cost reaches it.	2026-08-10 12:23:20 -07:00
Deepanshu Lulla	0580465384	feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416) ... * feat(router): add per-deployment allowed_fails_policy and cooldown_time override support Three bugs fixed in the router cooldown system: (1) deployment-level allowed_fails and allowed_fails_policy in model_info now take precedence over router-level settings in _should_cooldown_deployment; (2) failed fallback deployments now get evaluated for cooldown via _trigger_cooldown_for_failed_deployment, bypassing the Logging dedup gate; (3) DualCache promotes Redis cooldown entries using default 600s TTL instead of true remaining cooldown time -- _corrected_active_cooldown now evicts expired entries and corrects stale in-memory TTLs on backfill. Adds ServiceUnavailableError, BadGatewayError, and NotFoundError fields to AllowedFailsPolicy and cooldown_time to LiteLLMParamsTypedDict. * fix(router): gate fallback cooldown trigger on has_logged_async_failure; use only litellm_metadata for deployment ID * fix(router): use X | Y union syntax to fix UP007 strict lint gate * test(router_utils): add coverage for _trigger_cooldown_for_failed_deployment and has_logged_async_failure gate * test(router_utils): cover deployment cooldown override and exception swallow paths * fix(router): add InternalServerError/ServiceUnavailableError/BadGatewayError/NotFoundError to router-level get_allowed_fails_from_policy * fix(router): format router.py and add router-level policy tests * test(router): add CI-visible coverage for per-deployment cooldown policy Tests for `_get_deployment_cooldown_policy`, `_resolve_allowed_fails_from_policy`, and `_should_cooldown_based_on_deployment_policy` (cooldown_handlers.py), the `_corrected_active_cooldown` branches in CooldownCache, and the four new exception-type branches in `Router.get_allowed_fails_from_policy` (router.py) -- all in `tests/test_litellm/` which the enterprise-routing CI job runs. * fix(router): use is not None guard for cooldown_time_override in should_cooldown_based_on_allowed_fails_policy A cooldown_time_override of 0 was previously treated as falsy and silently fell through to the router-level cooldown_time value. Switched to an explicit is not None check so that zero is honored as a valid override. Added a regression test covering the zero case. * fix(router): honor has_logged_async_failure and metadata for fallback cooldown; support both model_info and litellm_params locations Manual verification against a live proxy surfaced that the fallback-cooldown-gap trigger never actually fired: the has_logged_async_failure check read a plain attribute that Logging never sets (the real flag lives in model_call_details), and the deployment_id lookup only trusted litellm_metadata, which regular chat completions never populate (only batch/thread/file endpoints do). Router overwrites model_info on whichever key is present before every attempt, so metadata is equally authoritative there, not caller-controlled as previously assumed. Also let allowed_fails/allowed_fails_policy/cooldown_time be set under either model_info or litellm_params, each preferring its own canonical location. * fix(router): fix ContentPolicyViolationError policy shadowing and partial-policy zero-threshold Two bugs from Greptile review on PR #34416: - ContentPolicyViolationError subclasses BadRequestError, so listing BadRequestError first in _EXCEPTION_POLICY_FIELDS made the isinstance check always match BadRequestError for content-policy errors, using the wrong allowed_fails threshold. Reordered so the subclass is checked first. - A deployment with a partial allowed_fails_policy and no deployment-wide allowed_fails forced allowed_fails_override=0 for any exception type its policy didn't cover, cooling the deployment down on the first unrelated failure. Now defers to router-level behavior for uncovered exception types instead of forcing an immediate cooldown. * fix(router): only trust a metadata/litellm_metadata bucket the router itself wrote deployment info into veria-ai flagged that preferring litellm_metadata whenever present could pick up a caller-supplied litellm_metadata.model_info.id (preserved via allow_client_pricing_override) instead of the metadata bucket the router actually populated for a regular completion's fallback attempt, naming an arbitrary "victim" deployment for cooldown. Router._update_kwargs_with_deployment() always writes model_info and deployment_model_name into the same bucket together. Only trust a bucket that carries deployment_model_name alongside model_info, since that marker is only ever set by the router itself, not by request-body metadata. * test(router): add regression coverage for ContentPolicyViolationError policy shadowing The subclass-ordering fix in commit 38fe4e4490 had no regression test. Verified the new test fails on the pre-fix ordering (asserts 2, got 10) before restoring the fix, and confirmed the same behavior through the full _should_cooldown_deployment call path against a real Router instance. * fix(router): let explicit allowed_fails_policy entries override the generic 4XX cooldown exclusion _is_cooldown_required skips cooldown evaluation for any 4XX status outside {429, 401, 408, 404} by default, since a generic client error is usually not the deployment's fault. BadRequestError and ContentPolicyViolationError both carry status 400, so their AllowedFailsPolicy fields (BadRequestErrorAllowedFails, ContentPolicyViolationErrorAllowedFails, both router-level pre-existing and the new deployment-level ones) were silently unreachable: an operator could set them to any value with no effect, since _is_cooldown_required blocked cooldown evaluation before that policy was ever consulted. _should_run_cooldown_logic now also checks whether an explicit allowed_fails_policy entry (deployment-level or router-level) covers the exception's type, and if so, proceeds with cooldown evaluation regardless of the generic status-code exclusion. The exclusion remains the default for exception types with no explicit policy. Verified live against a mock-triggered ContentPolicyViolationError (config-level mock_response, azure/gpt-4.1-mini deployment) with BadRequestErrorAllowedFails=100 and ContentPolicyViolationErrorAllowedFails=0 on the same deployment: it now cools down after exactly one ContentPolicyViolationError instead of never cooling down. * fix(router): use the router-stamped failed_deployment_id for fallback cooldown targeting Greptile flagged a real gap in the metadata-bucket-based deployment lookup: for a generic-API-call fallback, the router writes the current attempt into litellm_metadata, but a stale "metadata" bucket carrying the same deployment_model_name marker (from an earlier point) would be picked first, cooling the wrong deployment. Router already has a more robust, pre-existing mechanism for this exact problem: _set_failed_deployment_id_on_exception stamps the failing deployment's id directly onto the exception at the point of failure, immune to metadata-bucket ambiguity since a caller can't influence it and it doesn't depend on which bucket the current call type happens to use. It just wasn't called from _ageneric_api_call_with_fallbacks_helper's except block, unlike _completion/_acompletion. Added the missing call there (matching the existing pattern exactly), and changed _trigger_cooldown_for_failed_deployment to prefer exception.failed_deployment_id when present, falling back to metadata-bucket inspection only for call paths that don't stamp it yet. Verified live: the standard fallback-cooldown-gap scenario (two bad-key deployments in a fallback chain) still correctly cools down both the originally-called and fallback deployment. * fix(router): address human review on per-deployment cooldown overrides Scope allowed_fails_policy override to deployment-level only (a router-level policy predates this feature and must keep its existing behavior), exempt advisor-orchestration failures from the fallback cooldown trigger, keep the single-deployment model group protection intact against a generic deployment-level allowed_fails, make cooldown_time precedence consistent across resolution paths, fix a falsy-zero swallowing bug in the router-level allowed_fails fallback, and make allowed_fails_policy resolution fall through to the next matching exception type instead of stopping at the first unset field. Also restrict allowed_fails/allowed_fails_policy/cooldown_time to model_info: litellm_params gets copied into the actual provider request, so a router-only setting placed there would leak into that request. * test(router): update test_cooldown_handlers.py for the deployment-policy signature change Surfaced by the rebase: this mirrored test file (tests/test_litellm/ mirrors litellm/) predates the router_unit_tests/ coverage added earlier in this PR and was still calling _should_cooldown_based_on_deployment_policy with its old 4-argument signature and asserting the now-removed litellm_params cooldown_time location. * test(router): update test_fallback_event_handlers.py for model_info-only cooldown_time Another mirrored test file surfaced by the rebase that still asserted the now-removed litellm_params.cooldown_time location. * fix(router): match cooldown-duration precedence in the fallback path to the primary path _trigger_cooldown_for_failed_deployment only checked deployment config before falling back to the router default, skipping the response Retry-After header step that Router.deployment_callback_on_failure applies on the primary path. * fix(router): restore litellm_params.cooldown_time as a pre-existing fallback cooldown_time already had litellm_params support on Router.deployment_callback_on_failure before this PR; the earlier model_info-only restriction (aimed at the leak concern for the genuinely new allowed_fails/allowed_fails_policy fields) incorrectly dropped that pre-existing capability too. model_info still takes priority when both are set. * fix(router): keep the fallback-cooldown trigger in sync with #35104's review fixes Applies the same two fixes landed on the split-out PR #35104 (which #34416 still duplicates until it's rebased onto the merged base): increment the deployment's per-minute failure counter before evaluating cooldown, and require the server-stamped failed_deployment_id instead of trusting a metadata bucket, since neither "metadata" nor "litellm_metadata" can be told apart from a caller-supplied one without knowing the call's function_name. * fix(router): freeze the model_info fallback mapping to satisfy the type-discipline gate * fix(router): defer f-string interpolation in fallback-cooldown debug logs * fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget * fix(router): suppress reportPrivateUsage for cross-module cooldown helpers * fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks * fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one * fix(router): keep up with upstream typing modernization and Final-annotation ratchet * fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout * fix(router): stamp dynamic client-side-credential id in completion fallback paths too The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential) deployment id on exceptions, but the regular _completion/_acompletion exception handlers still stamped the static shared deployment's id. A tenant using invalid forwarded credentials could generate repeated failures attributed to, and eventually cooling down, the shared deployment other tenants rely on. Extracted the stamping logic into one shared helper used by all three call sites (generic API, sync completion, async completion) so the fix and future changes to it stay in one place. * fix(proxy): recognize body-supplied timeout/request_timeout/stream_timeout as caller-controlled client_side_timeout was only set when the caller used the x-litellm-timeout header, but Router._get_timeout also resolves the effective timeout from kwargs["timeout"], kwargs["request_timeout"], and kwargs["stream_timeout"], all settable directly in the request body (and x-litellm-stream-timeout wasn't marked either). A caller could set any of those to a near-zero value, force a 408 on every deployment in a fallback chain, and cool down deployments other tenants rely on without the guard in _trigger_cooldown_for_failed_deployment recognizing it as caller-controlled. Also strip any client-forged client_side_timeout from the request body so the marker is always server-computed. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>	2026-08-10 11:02:06 -07:00
yucheng-berri	457be8f00a	feat(ptu): surface PTU flat cost on the daily activity read path (#35391) ... Aggregate the ptu_flat_cost written by the rollup into SpendMetrics.flat_cost and DailySpendMetadata.total_flat_cost, so /team/daily/activity returns flat cost alongside per-request spend. The aggregated SQL path selects ptu_flat_cost only for LiteLLM_DailyTeamSpend and a constant zero for the other daily tables, keeping the response shape uniform. Rows written under the PTU sentinel api_key add their flat cost to every parent bucket (per-model, per-day, per-team totals) but never appear as an api_key row in any breakdown, and are excluded from the per-request provider breakdown; the sentinel string is not a real key alias. Both flat_cost and total_flat_cost default to zero, so a read of any entity without PTU config is unchanged. The sentinel row now keys on the deployment id, so the per-model breakdown keys it on model_group instead. That breakdown key is rendered directly as a label by the Usage page and the daily_with_models export, and a deployment id there would read as a UUID. Two deployments sharing a public name merge under it, which is the collapse the write path used to do by summing them into one row. Request rows are untouched and still key on model, since their model_group is a routing concept rather than a display name.	2026-08-10 10:55:55 -07:00
Deepanshu Lulla	9ce96c2d34	feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418) ... * feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both setters after assigning litellm_trace_id so every JSON log record emitted within the async request context carries trace_id and, when provided, session_id — enabling log correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without any changes to individual log call sites. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields * fix(logging): always reset session_id_var to empty string when no session_id provided * feat: gate request correlation IDs in logs behind request_correlation_in_logs flag * refactor: move correlation ID injection into CorrelationContextFilter * feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload Plaintext log lines (json_logs off) now get the same trace_id/session_id suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a visible effect regardless of log format. StandardLoggingPayload gets a new independent session_id field, populated from litellm_session_id. trace_id's existing session_id-first fallback is preserved when request_correlation_in_logs is off; with the flag on, an explicit litellm_trace_id now takes priority over litellm_session_id so the two fields carry genuinely independent values. * fix(logging): restore correlation context after nested calls; sanitize correlation ids Addresses two review findings on this PR. CorrelationContextFilter's trace_id/session_id contextvars were set on every Logging.__init__ but never reset, so a nested LiteLLM call sharing the same asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call, an MCP sampling call) would leave the outer request's subsequent log lines stamped with the nested call's ids instead of its own. set_trace_id/ set_session_id now return their contextvars.Token, and Logging stores them and resets both once its own success/failure handler actually completes, via a new idempotent _restore_correlation_context() called from all four terminal handlers. set_trace_id/set_session_id also now strip control characters and bound length before storing a caller-controlled trace_id/session_id, since these values can originate from request input (litellm_session_id, x-litellm- trace-id) and get interpolated into plain-text log lines - without this, a caller could embed \r/\n or escape sequences to forge fake log entries. * fix(logging): restore correlation context after nested calls, not before The previous commit called _restore_correlation_context() as the first line of each terminal handler, before that handler's own callback dispatch loop runs. That's backwards: a nested LiteLLM call triggered from within a callback (e.g. a guardrail's own LLM-as-judge call) would then capture the *already-reset* value as its own pre-call baseline, and its own reset would restore to that instead of the true outer value - verified live to still leak. success_handler/async_success_handler/failure_handler/async_failure_handler are now thin wrappers: the original bodies move to _success_handler_body/etc, called inside a try/finally that restores context only once the full body - including any nested calls its own callback dispatch triggers - has actually finished, mirroring proper stack-scoped nesting semantics. * test(logging): cover async_failure_handler's correlation-context restore Codecov flagged the new async_failure_handler wrapper (try/finally around _async_failure_handler_body) as uncovered - the method had no direct test at all before this PR's refactor split it into a wrapper. Adds a test that awaits it directly and asserts both that async_log_failure_event still fires and that _restore_correlation_context() puts the pre-call trace_id/session_id back. * fix(logging): restore correlation context by value, not by contextvars.Token veria-ai correctly flagged that contextvars.Token.reset() only works in the exact Context it was created in, and litellm's async success path (and streaming failure path) dispatch async_success_handler/async_failure_handler via asyncio.create_task and the global logging worker - a different Context than Logging.__init__ ran in. reset_trace_id/reset_session_id silently swallowed the resulting ValueError, so the restore was a no-op for exactly those paths. Verified independently: reproduced the raw contextvars behavior, then confirmed litellm's async success dispatch really does go through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py). Logging now captures the pre-call *value* (not a Token) and restores via a plain set_trace_id()/set_session_id() call, which works regardless of which Task/Context calls it. reset_trace_id/reset_session_id are removed as dead/unreliable code. Added a regression test that spawns __init__ and the restore in different asyncio Tasks - confirmed it fails against the prior Token-based commit and passes here. * fix(logging): restore correlation context in the originating task too Greptile's re-review correctly identified a remaining gap: for a successful acompletion(), async_success_handler is dispatched via asyncio.create_task + the global logging worker into a *different* Task than the one wrapper_async/Logging.__init__ ran in. The prior fix (43c164a) only restored the handler's own (detached, throwaway) Task - it never touched the originating request Task, which keeps this call's trace_id/ session_id set for the rest of its own execution (e.g. nested calls made via the same Task). wrapper()/wrapper_async() in litellm/utils.py now restore the originating Task's correlation context in a finally block once the whole call is done, regardless of what detached logging tasks it spawned along the way. Since the wrapped body rebinds its own `kwargs` local via function_setup(), sharing the dict object doesn't work here; a small mutable holder carries the constructed Logging instance back out to the outer wrapper instead. _restore_correlation_context() is no longer guarded against repeat calls: with value-based (not Token-based) restoration, each distinct Task that calls it needs its own restore to take effect in that Task's own view of the contextvars, so multiple calls (once per Task involved in an attempt) are required, not just tolerated. Added a regression test using mock_response to exercise the real success dispatch path (asyncio.create_task + GLOBAL_LOGGING_WORKER) without a live provider call, asserting the *test's own* (originating) task context is restored after the call - this is exactly the case Greptile flagged and the prior commit didn't cover. * fix(logging): restore correlation context when function_setup itself fails Greptile's 4th finding: if function_setup() constructs Logging() (whose __init__ already mutates trace_id_var/session_id_var) and then raises before returning - e.g. update_environment_variables() throws - the caller's wrapper()/wrapper_async() never receives a logging_obj reference, so its own restore-on-finally never fires. The correlation ids leak into every subsequent log line on that thread/task with no way to clear them. function_setup()'s own except block now restores the context itself in that case, using whatever logging_obj it managed to construct before failing (locals().get(), safe against the earlier failure modes where logging_obj was never assigned at all). Added a regression test that monkeypatches Logging.update_environment_variables to raise after construction, confirmed it fails without this fix (the leaked ids show up directly in the raised exception's own log line) and passes with it. Broader sweep (test_utils.py, test_router.py, test_main_module_header.py, streaming handler tests, plus all logging-specific tests): 722 passed. * fix(logging): don't assume every litellm_logging_obj is a real Logging instance CI caught a real regression from the last commit: tests/test_litellm/llms/xai/test_xai_key_fallback.py injects a minimal FakeLogging stand-in (only implementing update_from_kwargs) as litellm_logging_obj for a narrow realtime-config unit test, bypassing the real Logging class entirely. wrapper()/ wrapper_async()'s finally block and function_setup()'s except block both unconditionally called _restore_correlation_context() on whatever ended up in the holder, which doesn't exist on that stand-in. _restore_correlation_context is new plumbing specific to this PR's feature, not part of any pre-existing stand-in's expected interface, so callers of it can't assume every object playing the litellm_logging_obj role implements it. Added _restore_correlation_context_if_supported(), a small getattr-guarded helper, and used it at all three call sites. * fix(logging): don't restore context too early on setup failure or streaming Two more findings from Greptile's 5th review round. 1. function_setup()'s except block restored correlation context *after* logging the "Error in function_setup" exception, so that diagnostic log line itself was stamped with the doomed call's ids instead of the outer ids - misleading, since the failed call never produces anything else to attribute those ids to. Restore now happens before the log call. 2. wrapper()/wrapper_async() restored the originating task's context as soon as a streaming call returned, before the caller ever starts iterating the CustomStreamWrapper it just got back. Any log lines emitted while iterating (in the same thread/task) incorrectly showed the pre-call ids instead of this call's own ones. The wrapper finally block now skips the restore when the return value is a stream wrapper, deferring to the terminal handler that already fires once the stream is actually assembled/exhausted. Both verified with tests that fail against the prior commit and pass against this one. Broader sweep unchanged at 829 passing. * fix(logging): best-effort correlation cleanup on abandoned streams Greptile's 7th finding: if a caller returns a streaming response and never fully consumes it - stops iterating early, drops the reference, cancels it - the terminal handler that normally restores the originating task's trace_id/session_id never fires, since it only runs once the stream is actually assembled/exhausted. The ids leak into every subsequent log line in that thread/task with no bound. There's no reliable Python hook for "this was abandoned without being closed" - CustomStreamWrapper has no close()/__aexit__/context-manager convention today, and the only automatic option is __del__, whose timing is inherently unpredictable (delayed by cyclic GC, not guaranteed at interpreter shutdown, can run on a different thread). This is a best-effort safety net, not a guarantee, and is documented as such in the docstring. Testing this via real garbage collection proved unreliable in practice: per-chunk logging submits work to a thread pool executor whose worker thread transiently holds its own bound-method reference to the wrapper until that task completes, so refcount doesn't hit zero on a deterministic schedule even with polling. Tests call __del__ directly instead - a plain method, safe to invoke early - which exercises exactly the restore logic real garbage collection would eventually trigger, plus a case confirming a broken logging_obj can never make __del__ raise. * fix(logging): restore consumer's context at every real stream exit point Two more findings from this round. Veria AI: even a *fully consumed* stream never restored the actual consuming thread/task's correlation context. The terminal success dispatch (dispatch_success_handlers via asyncio.create_task for async, or success_handler via the shared executor for sync) only restores whatever detached context it runs in - never the caller's own thread/task that's running the for/async for loop. Same root cause as the wrapper-level fix two rounds ago, just missed for the streaming-completion path. Greptile: explicit aclose() (client disconnect, router fallback aborting a partial stream) closed the underlying stream without restoring correlation context either, since request wrappers intentionally skip restoration for returned streams and no terminal handler runs on this path. Added CustomStreamWrapper._restore_consumer_correlation_context(), called from every point control genuinely returns to the consumer: the final raise StopIteration/StopAsyncIteration on natural exhaustion (both sync branches, both async branches), _handle_stream_fallback_error (the shared choke point for all three failure-raising call sites), and aclose(). __del__ now delegates to the same helper instead of duplicating it. Verified with tests extending the existing streaming-exhaustion cases to assert the consuming context is restored after the loop completes (fails against the prior commit, passes now), plus a dedicated aclose() test. Broader sweep: 832 passing. * fix(logging): don't let a delayed __del__ finalizer clobber a newer active call If an abandoned stream's __del__ fires late (after cyclic GC delay), a different call may have already taken over the correlation contextvars in the same Task/thread. Restoring unconditionally would stomp that active call's trace_id/session_id with the abandoned stream's stale pre-call snapshot. __del__ now only restores when the contextvars still hold the ids this call itself set. * fix(logging): compare sanitized ids in the __del__ ownership guard set_trace_id()/set_session_id() sanitize (strip control chars, bound length) before storing, so the contextvar's value can differ from the raw litellm_trace_id/litellm_session_id. The __del__ ownership guard was comparing against the raw values, so a caller-supplied id containing control characters or exceeding 256 chars would never match, permanently skipping cleanup. Capture what set_trace_id()/set_session_id() actually stored and compare against that instead. * fix(logging): restore consumer context on the synthesized finish_reason chunk Both __next__ and _finalize_completed_stream() have a branch that fires when the underlying stream ends without ever emitting an explicit finish_reason chunk: they synthesize one via finish_reason_handler() and return it. A consumer that stops as soon as it sees finish_reason - a common pattern - never calls __next__()/__anext__() again, so the existing restore in the sent_last_chunk-is-True StopIteration branch never runs for them. The underlying stream is already exhausted at this point regardless of whether the caller keeps iterating, so restoring here is safe. * fix(logging): don't restore correlation context before the caller receives the final chunk The previous fix (5147c69186) restored context immediately before returning the synthesized finish_reason chunk from __next__/_finalize_completed_stream, reasoning that completion_stream was already exhausted. But that chunk is still this call's own data, and the caller's own application-level log statements processing it run in the same synchronous frame right after the return - restoring first made those lines carry the wrong (outer) ids, exactly what wrapper()/wrapper_async() deliberately avoid by not restoring while a stream is being iterated. Revert to not restoring there. A caller that keeps iterating still gets a correct, deterministic restore on its very next __next__()/__anext__() call (completion_stream is exhausted, so that immediately re-raises StopIteration/StopAsyncIteration through the already-restoring branch). A caller that stops right after finish_reason relies on aclose() or the best-effort __del__ guard, same as any other stream the caller doesn't fully exhaust. * refactor(logging): hoist a safely-hoistable function-body import to module top CorrelationContextFilter.filter()'s `import litellm` was a function-body import; verified it can move to module top without a circular-import failure (litellm/__init__.py already imports from litellm._logging before setting request_correlation_in_logs, but a bare `import litellm` only binds the already-in-sys.modules module object - the attribute itself isn't read until filter() actually runs, by which point litellm is fully initialized). * test(logging): move correlation tests into their conventionally-mapped files tests/test_litellm/ mirrors litellm/ in a parallel path. Correlation tests for the Logging class (litellm_logging.py), function_setup/wrapper_async (utils.py), and CustomStreamWrapper (streaming_handler.py) had all landed in test_logging.py, which only maps to litellm/_logging.py itself. Moving each group to its correctly-mapped file: test_litellm_logging.py (Logging class init/restore), test_utils.py (function_setup, wrapper_async), and test_streaming_handler.py (CustomStreamWrapper) in the next commit. test_logging.py keeps only what actually exercises _logging.py's own contextvars/filters/formatters/sanitization. No behavior change - same assertions, same coverage, just relocated. * fix(logging): restore correlation context unconditionally in wrapper()'s sync path Blocking finding from review: a caller-visible correlation feature was silently misattributing one request's logs to a different, unrelated one on the sync/threaded path. wrapper()/wrapper_async() both left trace_id/session_id "open" across a stream's entire iteration so the caller's own log lines while consuming it would carry the right ids. That's safe for wrapper_async(): each async call gets its own asyncio Task with its own copy of the contextvars, and Tasks are never recycled across requests, so a leftover value can only ever affect that one already-abandoned Task. It is not safe for wrapper() (sync): a plain OS thread has no such per-call isolation, and a thread pool's worker threads *are* recycled across unrelated requests. If a sync stream was abandoned (client disconnect, early break, an uncaught exception) without ever being exhausted or closed, nothing restored its contextvars, and a pool could later hand that same thread to a completely different call, which would inherit the abandoned request's ids as its own "pre-call" baseline and then restore back to that poison when it finished - permanently misattributing every subsequent log line on that thread, including its own, to the abandoned request. Strengthening the __del__ finalizer already added for this can't fix it: finalizer timing is exactly what a permanently-reused thread can't rely on. wrapper() now restores unconditionally in its own finally, before a sync stream is ever handed back to the caller. The trade-off: a sync stream consumer's own application-level log statements while iterating no longer automatically carry this call's ids (litellm's own internal per-chunk logging is unaffected, since it's dispatched separately). That's an acceptable cost for eliminating a silent cross-request misattribution bug. wrapper_async() keeps the existing conditional (skip-if-streaming) behavior, justified by the Task-isolation argument above; CustomStreamWrapper's __del__/aclose()/next-iteration restore machinery remains meaningful and necessary there. This also simplifies wrapper()/wrapper_async() back toward their original shape: both previously used a mutable-dict-holder split into a separate _body function to smuggle logging_obj/result out to an outer finally, working around function_setup() rebinding its own local `kwargs`. That restructuring is no longer needed - `logging_obj` (and, for wrapper_async(), `result`) were already function-level locals in scope for a plain try/finally; three of wrapper_async()'s retry-return statements now assign through `result` first so it accurately reflects what's actually returned even on a retry path. Regression test: test_abandoned_sync_stream_does_not_contaminate_a_later_call_on_the_same_thread in test_streaming_handler.py reproduces the exact reported scenario with a real single-worker ThreadPoolExecutor - confirmed it fails with the prior (skip-restore-on-stream) wrapper() and passes with this fix. * refactor(logging): use Mapping instead of bare dict for read-only params _get_standard_logging_payload_trace_id/_session_id only read litellm_params (.get() calls, no mutation) - annotate it as Mapping[str, Any] rather than a bare mutable dict, per the repo's no-mutable-collection-in-annotation rule. * fix(logging): scope request_correlation_in_logs to the async/proxy path only Blocking review finding: wrapper() (the sync entry point) used the same skip-restore-on-stream design as wrapper_async(), but a plain OS thread has no per-call context isolation the way an asyncio Task does, and a thread pool's worker threads are recycled across unrelated requests - an abandoned sync stream could leave its ids stuck on a thread a pool later hands to a completely different request, misattributing that request's logs. A fix existed and was tested (restore unconditionally in wrapper()'s own finally), but it doesn't benefit this feature's primary consumer - the proxy only ever calls the async entry point - and carries sync-specific complexity this PR doesn't need. Scope the feature to async only instead: Logging.__init__() takes a new supports_correlation_logging parameter (default True), threaded down from a new function_setup(..., is_async_call: bool = True) parameter. wrapper() is the one caller that passes is_async_call=False; every other function_setup() call site (wrapper_async(), the router, and proxy/MCP-internal call sites) is already async and keeps the default. With supports_correlation_logging=False, Logging.__init__() never calls set_trace_id()/set_session_id() at all, so a sync call has nothing to leak in the first place. wrapper() reverts to its pre-review shape with no correlation-specific code at all. StandardLoggingPayload's own trace_id/session_id fields are unaffected either way - they're a deterministic per-call read of self.litellm_trace_id/self.litellm_session_id, not ambient contextvar state, so they were never exposed to the cross-request bug. Full sync/direct-SDK support (stamping + its own safe-restore mechanism) is deferred to a follow-up PR; the fix and its regression test already exist in this branch's history at commit 9f3a20f4b2 and can be resurrected there. Tests: replaced the two wrapper()-level tests with ones proving the new invariant (sync calls, streaming and non-streaming, never touch trace_id_var/session_id_var even when the caller explicitly passes litellm_trace_id/litellm_session_id), and added a direct unit test for the supports_correlation_logging=False gate on Logging.__init__ itself. Verified live: a real proxy (Postgres-backed, real OpenAI calls) shows clean trace_id/session_id isolation across two concurrent sessions with no cross-contamination; a standalone script confirms real sync SDK calls against a real model never touch the correlation contextvars. * feat(logging): fall back to W3C traceparent/baggage for trace_id/session_id request_correlation_in_logs previously only resolved trace_id/session_id from litellm-specific sources: x-litellm-trace-id/x-litellm-session-id headers, a generic x-<vendor>-session-id header, or Anthropic-style metadata.user_id. If none were present, trace_id fell back to an auto-generated UUID unrelated to anything else, and session_id stayed empty - even when the caller already had real distributed-tracing instrumentation sending the actual industry-standard headers for this. Add a fallback to the W3C Trace Context traceparent header (trace-id component) and W3C Baggage header (session.id entry), so a request already carrying real OpenTelemetry trace context correlates litellm's own logs with the same trace in the caller's observability backend (Datadog, Honeycomb, Tempo, etc.) instead of getting an unrelated generated id. Precedence is unchanged for existing sources: explicit litellm headers and the Anthropic metadata path both still win over this new fallback, which only fires when neither found anything. trace_id and session_id are resolved independently here (unlike the existing chain_id mechanism, which uses one shared value for both), since traceparent and baggage are semantically distinct W3C concepts. New helpers _trace_id_from_traceparent/_session_id_from_baggage in litellm_pre_call_utils.py parse the header formats directly (no new dependency - both are simple fixed-width/delimited strings), wired into LiteLLMProxyRequestSetup.add_litellm_metadata_from_request_headers() only when the corresponding litellm_trace_id/litellm_session_id key isn't already set by the existing paths. Verified live against a real proxy: a bare traceparent header produces a log trace_id exactly matching its trace-id component; a traceparent alongside an explicit x-litellm-trace-id header (different value) produces a log showing the explicit header's value, proving precedence. * fix(logging): reserve trace_id/session_id in JsonFormatter against message-content spoofing JsonFormatter merges keys parsed from the message body before applying extra record attributes, and the extra-attributes loop skips a key that's already present. A caller-controlled log message that happens to parse as JSON/dict with a "trace_id"/"session_id" key (e.g. the proxy logging a raw request-header dict) could therefore make the JSON record carry the attacker-supplied value instead of the real correlation context set via CorrelationContextFilter. trace_id/session_id are now applied from the LogRecord's own attributes after message-content parsing, unconditionally overwriting anything the message body claimed for those two keys. * style(logging): fix import order (ruff I001) in _logging.py and litellm_logging.py - _logging.py: import litellm belongs after the stdlib from-imports, grouped with the other litellm.* imports, not before them. - litellm_logging.py: the refactor to Mapping introduced a second, separate `from collections.abc import Mapping` instead of merging it into the existing `from collections.abc import Callable` import. Caught by the strict-rule budget gate (ruff-strict-budget.json caps I001 at 0 new violations); both auto-fixed with `ruff check --fix --select I001`. * style(logging): freeze mutable-collection constructions flagged by LIT002 Five sites in this PR's diff built a mutable list/dict literal instead of a frozen value: a plain list of optional strings in CorrelationPlainFormatter, a `kwargs or {}` fallback, a `metadata or {}` fallback, two `[...]` candidate orderings, and a `dict(headers)` copy feeding a dict comprehension. Each is build-once/read-only, so this rewrites them as tuples, MappingProxyType, or a plain conditional `.get()` instead of seeding then reading a fresh mutable collection - no behavior change, confirmed by the existing test suite. Caught by the type-discipline budget gate (LIT002 capped at 0 new violations). * fix(logging): reserve trace_id/session_id even when no correlation context is active Live-proxy verification surfaced a gap in the earlier message-content-spoofing fix (7f390a57fc): that fix only overwrites trace_id/session_id from the LogRecord's own attribute, so it does nothing for a log line emitted before CorrelationContextFilter has stamped anything on this record (e.g. the "Request Headers" debug line, which fires before Logging.__init__() runs for the request). On such a record, a caller-supplied header literally named trace_id/session_id still got promoted into the JSON output via the embedded JSON/dict-repr parser, since there was no genuine value to protect. Fixed at the source: trace_id/session_id are now excluded unconditionally from the message-content-parsing promotion step, not just superseded afterward. Verified live against a real proxy - the exact adversarial request (headers literally named trace_id/session_id) no longer leaks into any JSON log record. Added a regression test for this no-active-context variant specifically, confirmed it fails against the prior commit and passes now. Also fixes an unrelated basedpyright regression from an earlier rebase's conflict resolution: litellm/utils.py's `logging_obj` was incorrectly re-annotated `Final` at its second assignment in function_setup() (it's first declared `None` a few lines earlier), which basedpyright correctly rejects. * fix(proxy): stop logging the raw W3C baggage session_id value _session_id_from_baggage() extracts the caller-controlled session.id entry verbatim - it isn't sanitized until set_session_id() runs later in Logging.__init__(). The debug log line for this extraction interpolated the raw value directly, so a caller could embed terminal control characters or ANSI escape sequences that forge/alter plaintext log output for anyone tailing the proxy's logs. Verified live: a baggage header with an embedded ANSI escape reached the terminal as a real, unescaped control sequence before this fix. Drops the value from the log line entirely (the extraction succeeding is enough signal on its own) rather than sanitizing-then-logging, matching veria-ai's suggestion. Added a regression test using caplog that fails against the prior commit and passes now. * fix(logging): restore consumer context only after stream-failure exception mapping _map_anthropic_exception/_map_aleph_alpha_exception synchronously log a debug diagnostic (the raw status code) as part of exception_type()'s mapping. _handle_stream_fallback_error restored the consumer's outer correlation context before calling exception_type(), so that diagnostic log line carried the outer (or empty) trace_id/session_id instead of the failing stream's own - flagged by Greptile. Moved the restore to run after mapping completes, matching the same restore-after-not-before pattern already applied elsewhere in this file for success/finish_reason handling. Added a regression test that captures the correlation context live during a mocked exception_type() call; fails against the prior commit, passes now. * fix(logging): restore consumer context only after aclose()'s stream close completes aclose() restored the consumer's outer correlation context as its first statement, before awaiting the underlying provider stream's own aclose()/ close(). If that close attempt raises, the except branch's debug diagnostic ran under the already-restored outer context instead of the closing stream's own trace_id/session_id - flagged by Greptile, same restore-too-early pattern as the stream-failure fix in f1cf9589d6. Moved the restore to the end of aclose(), after the close attempt (and its diagnostic logging) completes. Added a regression test with a fake stream whose aclose() raises, capturing the correlation context live during the diagnostic log call; fails against the prior commit, passes now. * style(logging): satisfy new strict-lint budgets introduced upstream (Final, ANN401, S110, TRY300, kwargs typing) Rebasing onto litellm_internal_staging pulled in 116 upstream commits that introduced/tightened several lint gates this PR's own code now trips: - LIT010 (every local/module-level variable must be Final): added Final annotations across _logging.py, litellm_logging.py, streaming_handler.py, litellm_pre_call_utils.py, and utils.py. Where a name is genuinely reassigned (logging_obj: starts None, later set to the real object) or branch-assigned, either restructured into a single ternary expression (ordered_candidates) or suppressed with `# rebind-ok: <reason>` matching this repo's documented escape hatch. - LIT011 (parameter mutation): suppressed the two new `data[key] = value` writes in litellm_pre_call_utils.py with `# rebind-ok`, matching the unsuppressed precedent already used for every other `data[...]` write in the same function - `data` is an intentional out-param there. - ANN001/ANN003/ANN202 (missing parameter/return type annotations): fully typed success_handler/_success_handler_body, their async twins, and failure_handler/_failure_handler_body/async variants in litellm_logging.py, plus function_setup in utils.py (added Rules to its existing TYPE_CHECKING block for the rules_obj: Rules annotation). - ANN401 (explicit Any disallowed): suppressed with `# noqa: ANN401` on the handful of genuinely-heterogeneous result/*args/**kwargs parameters, since ordinary suppression is this repo's documented path. - S110 (try/except/pass): added to the existing BLE001 noqa on the one best-effort correlation-cleanup try/except this PR added. - TRY300 (return inside try): moved two `return result` statements into `else:` blocks in the retry-fallback paths this PR's own diff touched. - reportPrivateUsage (basedpyright): renamed the two new StandardLoggingPayloadSetup static methods (get_standard_logging_payload_ trace_id/session_id) to drop their leading underscore, since they're genuinely called from a sibling module-level function in the same file. No behavior change - confirmed by the full existing test suite (819 passed) plus all four lint gates (ruff format, ruff-strict, type-discipline, basedpyright) passing clean. * fix(lint): stop RUF100 flagging noqa suppressions the strict gate needs CI's plain "ruff check" job uses the default ruff.toml, a narrower config than ruff-strict.toml (used only by the strict-rule budget gate). ANN401 and S110 aren't enabled in the default config, so RUF100 (unused-noqa) flagged the `# noqa: ANN401`/`# noqa: ...,S110` suppressions this PR added as pointless under that config, even though they're genuinely needed under ruff-strict.toml. - ANN401: added to ruff.toml's existing `lint.external` list (same mechanism already used for C901/TID251, enforced by the strict gate but not by this config) - these Any usages are genuinely dynamic/forwarded, so the suppression itself is correct and just needed registering. - S110: fixed the underlying code instead of registering another external code - the try/except/pass in CustomStreamWrapper._restore_consumer_correlation_context now logs at debug level on failure (matching the existing best-effort-cleanup pattern in _record_partial_usage_for_failure elsewhere in this file), which satisfies S110's own suggestion directly and needs no suppression at all. Verified against both ruff.toml and ruff-strict.toml directly, plus all three other gates (ruff format, type-discipline, basedpyright) and the full test suite (821 passed). * fix(lint): scope the ANN401 exemption to file level instead of a repo-wide noqa Ruff has no per-line-scoped way to register a noqa code across configs (that requires the default ruff.toml's lint.external list, which is repo-wide in scope even though the noqa itself is per-line). Since ruff does support file-level exemptions via per-file-ignores, and ANN401 only needed exempting in exactly two files, moved the exemption there instead: - ruff-strict.toml: added [lint.per-file-ignores] disabling ANN401 for litellm_logging.py and utils.py specifically, with a comment explaining why (heterogeneous response/forwarded-args parameters with no fitting concrete type - already verified by trying CostResponseTypes and hitting a real basedpyright mismatch). - ruff.toml: reverted the ANN401 entry from lint.external - no longer needed, since there's no `# noqa: ANN401` left anywhere for RUF100 to second-guess. - Removed the now-redundant `# noqa: ANN401` from the 10 affected parameters in both files, keeping the existing kwargs-ok reasons and adding a short inline comment on the `result`/`*args` lines pointing at the ruff-strict.toml exemption for context. Verified against both configs directly (ANN401 clean under ruff-strict.toml for these files, RUF100 clean under the default config), all four gates (ruff format, ruff-strict, type-discipline, basedpyright), and the full test suite (821 passed). * fix(logging): redact credential-shaped trace_id/session_id before stamping log records CorrelationContextFilter stamps trace_id/session_id onto a LogRecord after SecretRedactionFilter has already run, so a caller-controlled value (e.g. via x-litellm-trace-id or a W3C baggage header) that happens to look like a real credential reached JSON and plaintext logs unredacted. Apply the same credential redaction already used elsewhere in this module at _sanitize_correlation_id(), the single choke point both set_trace_id() and set_session_id() route through, so every caller-facing entry point is covered without depending on filter ordering. * fix(logging): restore correlation context when a stream's max-duration timeout fires CustomStreamWrapper.__anext__() called _check_max_streaming_duration() before entering its try block, so the litellm.Timeout it raises bypassed the except Exception -> _handle_stream_fallback_error path entirely, leaking the timed-out stream's own trace_id/session_id into whatever the consumer's task logs next. Move the check inside the try so it flows through the same restoration path every other stream failure already uses. * test(streaming): make dispatch_failure_handlers mock awaitable for the async max-duration test Moving _check_max_streaming_duration() inside __anext__()'s try block (prior commit) means a max-duration Timeout now dispatches failure handlers through the same path every other stream failure already uses, instead of bypassing it entirely. dispatch_failure_handlers is async on the real Logging class; the test's plain MagicMock logging_obj made asyncio.create_task() choke on a non-coroutine return value once that path actually got exercised. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-08-10 10:40:13 -07:00
yucheng-berri	61218f5f9f	feat(ptu): daily rollup writes per-model PTU flat cost by active hour (#35343) ... Add the daily rollup that reads PTU config off model deployments and writes flat cost to LiteLLM_DailyTeamSpend. For each UTC day a deployment carrying ptu_count and cost_per_ptu_per_hour accrues ptu_count * cost_per_ptu_per_hour * active_hours, where active_hours is the overlap between the day and the optional [ptu_effective_from, ptu_effective_to) window clamped to 24; a window opening at 23:00 charges one hour that day. Rows use a sentinel api_key so they stay distinguishable from per-request rows and share the existing unique constraint, and the write is idempotent so re-runs never double count. The cron is registered at proxy startup and runs at 00:15 UTC. Pricing one day per fire leaves two ways for a day to end up unpriced and stay that way: a window backdated at configuration time, which no fire ever revisits, and a fire that is missed or lands late, which the next one does not replay because the billed day comes from the wall clock rather than the scheduled time. Both are silent, since the failure alert only fires for a charge that was attempted. Each scheduled run therefore follows the day's reconcile with a catch-up pass that prices the (team, model, date) charges inside every declared window that carry no row yet, bounded at the earliest ptu_effective_from and floored at PTU_ROLLUP_MAX_BACKFILL_DAYS. It writes only what is missing: a day already priced keeps the amount it was billed whatever the config says now, and it runs no prune, so deciding a row is stale stays the single-day path's job. Zero-cost days write nothing, which leaves an out-of-window day reconsidered each run rather than recorded as done. A catch-up pass that fails cannot take the day's own result with it, and an explicit target_date still means reconcile exactly that day. The sentinel row keys on the deployment id, with the operator-facing name alongside it in model_group, which sits outside the table's unique key. The name is what a usage view displays, but a deployment can be renamed, and two runs holding config views from either side of a rename then wrote the same day under two different keys, so nothing collided and both charges survived. A multi-pod rig reproduced that as a permanent double charge that no later run repaired. Keyed on the id both writes land on one key and the upsert collapses them; when the rate changed too, last writer wins on the amount rather than adding a row. Deployments sharing a public name inside a team therefore no longer need collapsing into a single charge: each keys its own row, and the read path merges them back under the shared name. The read path that surfaces the amount lands in a follow-up PR. The prune is the one destructive step, so it only runs when the pod took the cross-pod lock. The upserts stay unguarded, since they are idempotent and no lock problem may cost a day, but the delete compares a cutoff and an updated_at stamped on different hosts, and a live rig showed a pod whose clock ran ten minutes ahead sweeping the charge a concurrent pod had just written, leaving the day at zero. Its cutoff also allows PTU_PRUNE_SKEW_GRACE_SECONDS of slack, which separates the two populations without requiring clocks to agree: a stale row is hours old and a concurrently written one is seconds old. The catch-up deletes nothing. Removing a deployment or narrowing its window stops it accruing new charges and leaves the days it was already billed for standing, since those days were incurred and a usage view has to keep reporting them. A deployment carrying no ptu_effective_from is skipped rather than treated as open ended. The endpoints require a start, and substituting the cap floor for a missing one meant a windowless deployment accrued the whole ninety day window on its first run, billing days it did not exist while the result still reported a single row written.	2026-08-10 10:19:58 -07:00
yucheng-berri	e5386c10a7	feat(ptu): configure provisioned-throughput flat cost on a model deployment (#35341) ... Add ptu_count, cost_per_ptu_per_hour, ptu_effective_from and ptu_effective_to to ModelInfo so a model deployment can carry the inputs for provisioned-throughput flat-cost attribution. ModelInfo validates per-field bounds (positive count, non-negative rate, effective_to after effective_from); model/new and model/{id}/update enforce the cross-field invariant (count and rate set together, team_id required) on the effective model_info so partial updates validate the merged result, and v1/model/info returns the fields. LiteLLM_DailyTeamSpend gains ptu_flat_cost and ptu_source_model_id columns plus a sentinel api_key constant; the daily rollup that writes them lands in a follow-up PR. Adding the optional model_info fields is backward compatible; models without them are unaffected. ptu_effective_from is required alongside the count and rate rather than optional. Flat cost accrues from that instant, so an absent start has to be inferred, and inferring it let a deployment configured today be billed for days it did not exist. Both PTU validators also run over the merged view before any write on the update path, beside the premium check the create path already runs there: the team ACL update below autocommits, so a validator raising further down left the team mutated and the deployment row never written. The update path validates the model_info a patch would store rather than the patch alone. An invariant holds over the deployment as it will exist, not over whichever subset of fields a caller sent, and validating the patch rejected raising the rate on an already configured model because that patch carries no start of its own.	2026-08-10 09:51:16 -07:00
Mateo Wang	f6b9518ddb	Merge pull request #36092 from BerriAI/devin_ai_fix_openai_passthrough_files_route_36086 ... fix(proxy): stop /{provider}/v1/files from capturing /openai_passthrough	2026-08-09 12:05:20 -07:00
mateo-berri	bd719c21dc	test(passthrough): annotate route match scope as Final	2026-08-09 11:49:20 -07:00
Yuneng Jiang	00600c1af7	test(proxy): guard management_v1 against fastapi names removed in supported releases	2026-08-08 21:15:56 -07:00
yuneng-jiang	ecba48dd7c	Merge pull request #35773 from HuanQian571/litellm_fix_management_v1_get_flat_params ... fix(proxy): restore management_v1 query-param validation under fastapi>=0.140.7	2026-08-08 21:09:13 -07:00
mateo-berri	85c1b5d04a	Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_fix_openai_passthrough_files_route_36086	2026-08-08 18:01:59 -07:00
Mateo Wang	6eaeab8eae	Merge pull request #36273 from BerriAI/litellm_dbless_hook_registration ... fix(proxy): skip prisma-dependent hooks when no database is attached	2026-08-08 17:25:23 -07:00
yucheng-berri	efc4e6f28c	fix(batches): keep batch state in sync on a poll without claiming attribution (#34456) ... A poll of a Vertex passthrough batch wrote nothing to the managed-object row, so status and file_object stayed frozen at the create-time snapshot and GET /v1/batches served a stale status and an empty output file id for the life of the batch. Only the create may claim a batch, but every observation of one may refresh its state. store_unified_object_id takes create_if_missing, which the poll clears: it refreshes status and file_object through update_many, and leaves a row that is absent absent rather than creating one owned by the observer, since created_by and team_id are written by whoever reaches the create branch. The update payload is now shared with the upsert so it cannot drift into writing api_key, request_tags, created_by or team_id. The passthrough identity re-assertion that was previously part of this PR ships separately in #36121, so this PR keeps only the batch attribution work. The creating key owns user_api_key_alias only when it actually has one. Guarding the overwrite on the presence of a key rather than on a resolved alias nulled the field out for every key generated without key_alias, and for any key rotated or deleted before its batch finished, losing the creating user's alias that the spend row previously carried. The guard now matches the team-alias line below it.	2026-08-08 16:01:47 -07:00
mateo	2d1ee3aab2	fix(proxy): keep the reservation when a disconnect happens while provider output is held back ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-08 20:39:55 +00:00
mateo	bb0bb48da8	fix(proxy): do not let held-back keepalive pings block the budget reservation refund on client disconnect ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-08 20:05:45 +00:00
tin-berri	e35ee4e5fa	feat(router): independent, default-on deployment affinity for the auto-router (#36146)	2026-08-08 13:02:29 -07:00
devin-ai-integration[bot]	12aeb53aec	fix(otel): mark v2 server spans as failed for pre-call errors (#34546) ... * fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): cover malformed-body rejection when auth error is recovered Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): skip authorization for a request whose body never parsed Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-08 12:40:00 -07:00
mateo-berri	8c0556abf6	fix(proxy): authenticate managed ids before routing	2026-08-08 02:42:34 -07:00
mateo-berri	855c49d0ef	fix(proxy): skip prisma-dependent hooks when no database is attached	2026-08-08 01:44:56 -07:00
mateo-berri	5f7a663005	fix(proxy): enforce require_managed_files on every raw provider id route ... require_managed_files was only checked on upload, so raw provider ids still reached the batch, fine-tuning and vector store file routes. Ownership rows exist only for managed ids, so those requests were forwarded under shared credentials with no tenant check: knowing another tenant's id was enough to read, run against, cancel or delete their object. Generalise the file-id guard to validate_managed_id_requirement(resource_id, resource_kind) and call it on batch create/retrieve/cancel, fine-tuning create/retrieve/cancel (training_file and validation_file both) and the shared vector store file id resolver. Behaviour is unchanged when the setting is off.	2026-08-08 01:33:32 -07:00
mateo-berri	292161f766	fix(proxy): read through to the DB on registry misses so just-created models, guardrails, and agents resolve on sibling replicas	2026-08-07 23:25:10 -07:00
mateo-berri	10209a8f91	Merge branch 'litellm_internal_staging' into devin_ai_require_managed_files_read_paths_35530	2026-08-07 23:17:28 -07:00
Yuneng Jiang	ee0c0cc7e8	feat(proxy): let USE_V2_MIGRATION_RESOLVER select the v2 migration resolver ... --use_v2_migration_resolver had no env var, and the Helm migrations job runs prisma_migration.py, which calls run_server with a fixed argv. There was no seam to pass the flag, so a Helm install could not reach the v2 resolver at all. Reading it from the environment makes the existing flag configurable from a deployment.	2026-08-07 22:18:59 -07:00
Mateo Wang	f32d89f828	Merge pull request #36244 from BerriAI/litellm_fix_stale_team_budget_assertion	2026-08-07 20:22:54 -07:00
yucheng-berri	d4dc2c39e7	fix(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing (#36119) ... * feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing AWS's ApplyGuardrail API rejects requests whose content exceeds the account's per-request "maximum input size in text units" quota with a 400 ValidationException. That cap is account/region/policy-dependent and cannot be predicted from config, so it can only be reacted to. _make_apply_guardrail_request now tries the whole-content call first (no behavior change for requests that already fit). On a too-large ValidationException it bisects the flat content list and retries each half sequentially, recursing until every piece fits or cannot be split further, then merges the per-chunk responses (action, assessments, outputs, usage) into one so callers cannot tell chunking happened. A real guardrail block on any (sub-)chunk still raises immediately. Contextual-grounding requests are never chunked: grounding scores the response holistically against the whole reference source, so fragmenting it would produce misleading scores. Each chunk call also gets a small exponential backoff retry on AWS ThrottlingException (429), since chunking increases the number of per-second API calls and can trade a 400 for a 429. All new state is local to a single request's call stack (no shared cache, no cross-process coordination), so this is safe for single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike. * fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback Fixes three issues flagged in review of the chunking fallback: a single oversized content item couldn't be split (only list-length bisection was supported), a chunked request that got recovered still logged a stray failure telemetry entry alongside the real outcome, and flattening chunk outputs without positional bookkeeping could misalign masked text onto the wrong original message once a chunk had nothing to mask. * test(guardrails): add regression test for multi-level Bedrock guardrail chunking Confirms the too-large bisection recursion isn't capped at a single split: a payload that is still oversized after the first halving keeps splitting until every piece fits, converging on however many chunks it takes rather than only ever producing two. * fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a hybrid strategy: bin-pack content into fixed-budget batches up front as the fast path, falling back to the existing recursive bisection only for a batch AWS still rejects as too large. Avoids paying O(log n) round trips on every oversized request when a single pass would do. Also switch single-item text splitting from a raw character midpoint to the nearest whitespace boundary, so a fragment never starts or ends mid-word. Closes the accidental-severing case from review; the residual gap (a multi-word denied phrase deliberately straddling the boundary) is documented as an accepted limitation, since fixing it would require an overlap window reconciled against masked output with no documented length-preservation guarantee from AWS. * chore(ui): regenerate dashboard API types * fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle AWS reports an ApplyGuardrail request that exceeds the per-request text-unit cap as a ThrottlingException (429), not only as the documented ValidationException (400). Verified against a live guardrail with an active content-filter policy: a 3273-text-unit request comes back as "Input text size (3273 text units) exceeds the maximum allowed (1000 text units) for the content filter policy (Classic tier)". The throttle retry keyed off status 429 alone, so every oversized chunk burned the full backoff-retry budget - each attempt a billed AWS call preceded by a sleep - before the bisection fallback got a chance, at every level of the recursion. A size error is not transient; re-posting the same content can never succeed. It now short-circuits straight to bisection. Also rename _is_input_too_large_validation_error to _is_input_too_large_error (it never keyed off the status code, and the error is not always a ValidationException), correct the docstrings that asserted a 400, and log at warning level when a split happens so the recovery is visible without --detailed_debug. * Revert "chore(ui): regenerate dashboard API types" This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9. * fix(guardrails): group all fragments of one item and stop double-logging Two defects found in review, both invisible to the existing tests. Fragment grouping assumed a split content item always produces exactly two adjacent fragments. That holds for one bisection level but not two: an item split twice yields four fragments, which were regrouped in fixed pairs into two output entries for a single message. Since masking walks the merged outputs by a running index across the original, unchunked message list, that message was written back truncated to its first half and every later message shifted. Fragments now carry the size of the group they belong to, so any number of them collapse back into exactly one output entry. Telemetry was also double-counted. AsyncHTTPHandler.post calls raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's error path, which logged guardrail_failed_to_respond before re-raising as an HTTPException that the consolidating caller then logged again. A request recovered by chunking reported one failure per rejected attempt plus a success. The ApplyGuardrail path now opts out of that per-attempt logging, since it owns consolidated per-request logging; the connection-level branch still logs, as nothing else records it. The existing tests missed both because their mocks return a non-200 response object, while the real client raises. Added a helper that raises a genuine httpx.HTTPStatusError so these paths are covered the way production hits them, plus a case asserting an unrecoverable failure still logs exactly once rather than zero times. * refactor(guardrails): move Bedrock chunking rationale into docstrings The chunking work explained itself with inline comment blocks, which this repo's conventions do not want. Folded that reasoning into the docstrings of the functions it describes and dropped the comments, including the module-level constant blocks and the test-file banner. No behavior change. The banner also claimed AWS rejects an oversized request with a 400 ValidationException, which live testing disproved, so removing it drops a stale claim as well as an internal ticket reference from a public repo. * feat(guardrails): match AWS default chunk budget and make it configurable ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters, per second. Chunking has to respect that throughput limit rather than just the per-request size, otherwise splitting an oversized request trades a size error for a throttle. The budget now defaults to 25,000 to match that default for every user, up from an arbitrary 20,000. Accounts with raised quotas can spend fewer calls by setting chunk_budget_chars on the guardrail. A value AWS still rejects as too large is bisected automatically, so an over-large setting costs an extra round trip rather than failing the request. * fix(guardrails): never split a Bedrock text into an empty fragment _nearest_whitespace_split_index could return len(text) when the only space at or after the midpoint was the final character, so the first fragment came back identical to the text AWS had just rejected as too large and the second came back empty. AWS rejects the unchanged fragment again, and each retry re-splits it into the same fragment, so an oversized single item shaped like a long unbroken token with one trailing space exhausted the stack with a RecursionError instead of scanning or surfacing Bedrock's error. Candidate boundaries that would leave either side empty are now discarded, and the raw midpoint is used when none remain. The midpoint is always safe because _split_bedrock_content only calls this for text of at least two characters. * style(guardrails): move chunking rationale out of comments and into docstrings * fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body Also types the credentials parameter on the new chunking helpers and rebuilds fragment grouping without mutating a list or rebinding an index * fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body Restores the source changes intended for a08e4cf309, which landed with only the test. Also types the credentials parameter on the new chunking helpers and rebuilds fragment grouping without mutating a list or rebinding an index * style(guardrails): sort the constants import into the first-party block * refactor(guardrails): bring the Bedrock chunking path under the LIT lint budgets Annotates never-rebound locals with Final, replaces the retry counter and the two branch-assigned locals with single bindings, and moves the internal chunking chain to Sequence parameters and tuple returns. Collections that reach the logged payload stay lists on purpose: redact_nested_match_and_regex_keys only traverses dict and list, so a tuple would carry PII past redaction. The remaining constructions are contract-bound and carry inline reasons * fix(guardrails): keep the pre-chunking contract for failures reported inside a 200 Reverts the 500 this branch introduced for an AWS 200 whose body carries an Output.__type exception marker: the request proceeds as it did before chunking existed. The logged status is now derived from the merged response instead of being hardcoded to success, so that shape is still reported as guardrail_failed_to_respond. The consolidated failure logger also goes back to logging a dict rather than a bare string, matching both the pre-chunking code and the InvokeGuardrailChecks path in this file * docs(guardrails): correct the docstring for failures reported inside a 200 body The raise was reverted, so the docstring no longer describes the code. Records that the request proceeds by design and points at LIT-5338 for closing the fail-open path behind the existing unreachable_fallback setting --------- Co-authored-by: spencer-burridge <265588760+spencer-burridge@users.noreply.github.com>	2026-08-07 19:44:24 -07:00
mateo	0791dd941b	test(proxy): assert the copy _add_team_member_budget_table returns ... Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-08 02:19:57 +00:00
Mateo Wang	1bafdb3c93	Merge pull request #36049 from BerriAI/litellm_list_batches_resolves_unified_ids ... fix(managed_files): return unified output file ids from GET /batches	2026-08-07 17:59:09 -07:00
ryan-crabbe-berri	2a9aac7004	fix(ui): let access groups be a team's only model source, with hover provenance (#36234) ... * feat(proxy): return per-group model provenance on /team/info /team/info now carries access_group_details, one entry per resolved access group with its id, name, and model list, so the UI can attribute each inherited model to the group granting it. The batch resolver returns the access group rows keyed by id instead of a stringly dict of lists, and the team member budget helper returns a copy instead of mutating its parameter. Type discipline and basedpyright budgets ratchet down accordingly. * feat(ui): allow group-only teams and show model provenance on hover Team create and edit no longer require a model selection: an empty selection is saved as the no-default-models sentinel, never as a bare empty list, since an empty team model list means unrestricted access. The team info Models card now renders every badge with a hover tooltip naming how the team got that model: directly, via named access groups, or both, and group-granted badges stay visible when the direct list is empty or a sentinel. * refactor(proxy): dedupe access group ids and return copies instead of mutating Duplicate access_group_ids no longer amplify the /team/info response: ids collapse order-preserving before provenance is built, pinned by a regression test. The resolver returns a model_copy rather than mutating its parameter, and the team create call sends a new object instead of reassigning formValues.models. Budgets ratchet down further with the mutation removal.	2026-08-07 17:45:50 -07:00
devin-ai-integration[bot]	1a45bf9afe	fix(proxy): resolve entity access groups in the model listing endpoints (#36230) ... * fix(proxy): resolve entity access groups in the model listing endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse the fetched team object when listing models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover key-level access group resolution in model listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-07 17:45:30 -07:00
tin-berri	3238ce8406	feat(auto-router): track turns per complexity tier (LIT-5302) (#36209) ... * feat(auto-router): track turns per complexity tier (LIT-5302) Stamps complexity tier at decision time (rollup never re-derives from routed model, since tier->model mapping is mutable config). Records per-tier turn counts in LiteLLM_AutoRouterSession.tier_turns (jsonb), rolls up per router in benchmarks SQL via jsonb_object_agg, returns on AutoRouterBenchmarkGroup for dashboard turns/share metrics. Addresses Greptile/Bugbot findings: - Missing _SessionAggRow.tier_turns field: added with field_validator to parse jsonb text cast and handle NULL. Would 500 every benchmarks read. - Missing ::text cast on tier parameter: Postgres fails type inference on parameterized CASE/IS NULL without explicit cast. Added to all usages. - Docstring false claim (only complexity routers produce tiers): quality router stamps numeric tier '1'/'2'/'3'. Per-type grouping in SQL prevents cross-contamination. Rewrote docstring to clarify isolation. - Comment convention violations: stripped per CLAUDE.md rule. - Test gaps: 8 unit tests for extraction/validation/aggregation, 7 behavior tests for SQL semantics against real Postgres. 12 mutations killed. Fixed fragile complexity_router test that broke on nested function calls. No API change; extends existing GET /auto_router/benchmarks response only. Co-Authored-By: Claude <noreply@anthropic.com> * fix(auto-router): address review findings on tier turns tracking - Guard router_type update so a mid-session reconfigure can't pool foreign tier names into tier_turns - Keep pinned turns attributed to the tier that actually serves them - Drop stray -- AlterTable comment from hand-written migration - Drop the now-unnecessary ::text/json.loads round-trip; prisma already returns tier_turns as a parsed dict Co-Authored-By: Claude <noreply@anthropic.com> * fix(auto-router): satisfy type-discipline lint gate - tier_turns fields: dict[str, int] -> Mapping[str, int] (LIT001, mutable collection in annotation); these are read-only after construction - _summed_agg_row: {} -> MappingProxyType({}) (LIT002, mutable dict literal) - default-fallback branch: replace the reassigned-without-Final fallback_tier with a Final default_model_first flag and a single ternary assignment (LIT010) Verified locally: type_discipline_gate.py, ruff_strict_gate.py, and type_check_gate.py all pass against the litellm_internal_staging merge-base; full test_complexity_router.py (374), auto_router management-endpoint tests (26), db-layer rollup tests (31), and the live-Postgres proxy_behavior rollup suite (17) all pass. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-08-07 17:03:34 -07:00
mateo	860e37597f	fix(a2a): align agent list annotation and test with the tuple return type ... PR #36020 changed AgentRegistry.get_agent_list to return tuple[AgentResponse, ...], and PR #35163 added a test asserting the result equals []. Both were green on their own branches and only collided once they were both on litellm_internal_staging, so proxy-server has been failing on every PR since with 'assert () == []'. Nothing user-facing was wrong: get_agents only iterates the result and rebuilds it with comprehensions, and FastAPI serializes a tuple to the same JSON array. The test expectation was simply stale, so it now compares against (). The get_agents local was still annotated list[AgentResponse] while two branches assign the registry tuple straight into it, so it widens to Sequence[AgentResponse]. That covers both the tuple and the list branches without pretending the value is mutable. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-08-07 14:08:58 -07:00
devin-ai-integration[bot]	c19ab70d96	fix(proxy): forward resolved provider and deployment pricing in /cost/estimate ... estimate_cost resolved on-prem aliases (e.g. nvidia/zai-org/glm-5.2) to their underlying model and custom_llm_provider via the router, then called completion_cost without either, so provider inference ran on the bare model and raised "LLM Provider NOT provided"; deployment-configured per-token pricing was dropped too, so priced on-prem deployments estimated 0. The resolver now returns a frozen ResolvedCostModel(model, provider, custom_cost_per_token) and estimate_cost forwards both into completion_cost and surfaces the configured per-token pricing in the response, deriving that pricing as single Final values. Resolves LIT-5210	2026-08-07 20:51:17 +00:00
ryan-crabbe-berri	78addb230b	fix(proxy): deny agent access when key and team grants resolve to nothing (#36221) ... * fix(proxy): deny when agent grants resolve to nothing `get_allowed_agents` returned a plain list where the empty value meant both "this caller was never restricted" and "this caller's grants resolved to nothing". Downstream read either as allow-all, so a key restricted to one agent inside a team restricted to another reached every agent on the proxy, and an access group that resolved to no agents did the same. Replace it with `resolve_agent_access`, returning a tagged UnrestrictedAgentAccess | RestrictedAgentAccess. Only a caller with no grant anywhere is unrestricted; an empty restricted set denies. Access group lookup failures now propagate to the key/team resolvers so a DB error still fails open exactly as before, while a group that genuinely resolves to nothing denies. * style(proxy): drop redundant comments from the agent access match	2026-08-07 20:44:11 +00:00
ryan-crabbe-berri	eb3c8c168f	fix(proxy): derive config agent ids from agent_name so grants survive secret rotation (#36020) ... * fix(proxy): derive config agent ids from agent_name so grants survive secret rotation Config-defined A2A agents were identified by a sha256 of the whole resolved config entry, secrets included, so rotating an os.environ secret re-minted the agent_id on restart and orphaned every object_permission.agents grant while grant-less keys kept access (LIT-5144). The id now hashes only agent_name, and the old full-entry hash is kept as a legacy alias: permission checks, GET /v1/agents filtering, spend and key attachment, and public_agent_groups all normalize legacy ids so pre-upgrade grants keep working * fix(proxy): persist stable agent ids into stored grants at startup The runtime alias only translates a legacy grant while the current config still hashes to it, so a secret rotation after upgrading would orphan the grant, and an orphaned grant intersecting a stable team grant collapses to an empty list that downstream reads as allow-all. Rewriting the stored ids once at boot removes both. This cannot be a SQL migration because only the running proxy can recompute the legacy hash from resolved config secrets * fix(proxy): make the grant id migration a compare-and-swap A grant edited between the migration's read and write kept the stale snapshot. The update now predicates on the agents array read at scan time via update_many, so a concurrently modified row is skipped and the runtime alias covers it until the next boot retries * fix(proxy): retry the grant id migration and stay within the LIT002 ceiling The one-shot startup task now retries up to three times with a short delay so a transient DB error at boot cannot leave a legacy grant unmigrated until an operator's next restart is the rotation itself. The new list constructions in the migration and the alias-expanded agent id lookups are tuples now, keeping the branch under the mutable-collection budget * fix(proxy): count compare-and-swap misses in the grant id migration migrate_legacy_grant_ids now returns rewritten and missed counts from the update_many results instead of reporting scanned rows as migrated, and the startup task retries while any rows remain unmigrated, not just on errors * fix(lint): clear basedpyright budget breaches in agent id aliasing	2026-08-07 19:19:49 +00:00
Yassin Kortam	c9292d3af2	fix(proxy): stop alerting on health probes that lose the planned engine-restart race (#36141)	2026-08-07 09:57:30 -07:00
ryan-crabbe-berri	527dc0a8bb	feat(proxy): add apply_user_budget_to_team_keys opt-in (#36102) ... * feat(proxy): add apply_user_budget_to_team_keys opt-in PR #32005 made a user's personal max_budget apply to their team-scoped keys too, and PR #35271 reverted the whole thing (behavior plus the skip_user_budget_on_team_key opt-out) because that flipped the default for everyone. This brings the behavior back the other way round: default is unchanged, and general_settings.apply_user_budget_to_team_keys opts a deployment into charging the key owner's personal budget on team keys. The flag reaches all three personal-budget gates so an opted-in deployment enforces consistently: the read-time check in common_checks, the optimistic reservation counter in _get_budget_counters, and the _PROXY_MaxBudgetLimiter pre-call hook. It is also in the /config/list allowed args and, unlike the reverted flag, in the _update_general_settings propagation allowlist, so the Admin UI General Settings toggle actually takes effect at runtime; an explicit YAML value still wins over the DB value on reload. get_config_list's allowed_args moves to a module-level frozen mapping of field name to type string, dropping 18 LIT002 violations and rebuilding one less dict per request. * style(proxy): drop explanatory comments from the budget flag paths	2026-08-07 15:40:13 +00:00
ryan-crabbe-berri	83ab6e08da	fix(proxy): invalidate cached project object on project update and delete (#36028) ... * fix(proxy): invalidate cached project object on /project/update and /project/delete The auth path reads projects cache-first via get_project_object with a 60s TTL and no freshness check, but no project write endpoint ever evicted the project_id:{id} cache entry. A project cached before /project/update added a model allowlist kept an empty models list in cache, so _run_project_checks skipped can_project_access_model and project-bound keys could call team models outside the project allowlist until the TTL expired. The same staleness applied to blocked status and budget fields, and /project/delete left the deleted project enforceable from cache. Evict the cache entry after the DB write in update_project and delete_project via a shared delete_cached_project_object helper, with the cache key derivation shared with get_project_object. * fix(proxy): broadcast project cache invalidation to all workers and make eviction best-effort Single-worker eviction leaves every other worker serving its in-memory copy of the mutated project until the 60s TTL expires, so a project allowlist change was still bypassable on multi-worker deployments. Add a coordination Redis pub/sub channel (litellm_proxy.auth_cache_invalidation): project eviction publishes the cache key and a per-worker subscriber deletes the local in-memory entry, with the next auth read refetching from the DB. Subscriber starts on any deployment with a coordination Redis and falls back to the TTL when none is configured. Also wrap the eviction in a best-effort catch: the DB write has already committed when eviction runs, so a cache backend error must not turn a successful update into a 500 or abort the remaining ids in /project/delete. * fix(lint): sort auth cache invalidation import and suppress best-effort shutdown catch The strict-budget gate flagged the new import block as un-sorted (I001) and the broad except in stop_auth_cache_invalidation_subscriber (BLE001); the catch is intentional since a failing stop must not break proxy shutdown, so it carries a named suppression instead of counting against the budget.	2026-08-07 15:19:00 +00:00
Aayush Gid	d332accabc	fix(proxy): improve Headroom 404 compression error diagnostics (#35952)	2026-08-07 08:16:41 -07:00