mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
3592 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Tin Chi Lo
|
52ad2538df | feat: pre-adoption shadow eval for the auto-router (blind pairwise judge, per-tier win rates) | ||
|
ryan-crabbe-berri
|
be71a8fdbf
|
fix(alerting): dedupe scheduled Slack spend reports across pods (#36489)
* fix(alerting): dedupe scheduled Slack spend reports across pods Every pod ran its own weekly/monthly spend report jobs, prometheus fallback stats cron, and daily report loop, so deployments with multiple replicas or uvicorn workers received one copy per pod. Gate each scheduled send behind the shared PodLockManager redis lock. The lock is never released: its TTL (the full reporting window for the weekly interval job, whose per-pod anchors drift by boot time and jitter) doubles as a sent-this-window marker. acquire_lock returning None (no redis wired) proceeds, preserving single-pod behavior. Also generalize the pod lock could-not-acquire log line, which claimed to be about spend tracking for every consumer. Fixes #14809 * fix(alerting): harden spend report locks after adversarial review Weekly lock TTL gets an hour haircut: with ttl equal to the interval, the winner re-fires just before its own key expires, reacquires without a TTL refresh, and the key then lapses in time for a trailing pod to re-send. Job/lock ids move to litellm/constants.py per convention, and spend_report_frequency now rejects non-positive day counts, which previously coerced to an every-second schedule and would now compute a negative lock TTL that silently never sends. Adds the missing test coverage the review flagged: startup_event's pod_lock_manager wiring (identity-asserted), the prometheus closure's positive path, and the ungated immediate prometheus send pinned to exactly one await. * test(alerting): consolidate spend_report_frequency validator coverage Drops a duplicate non-positive-days test and parametrizes the survivor over the suffix half of the validator too * fix(alerting): route the startup prometheus fallback send through the pod lock Greptile caught that the boot-time send still ran once per pod when PROMETHEUS_URL is set, the same duplication class this PR removes * fix(alerting): make report lock acquisition non-reentrant Greptile caught that a pod booting within an hour of the fallback stats cron sent twice: the startup send takes the lock, then the cron fire hits acquire_lock's reacquire branch, which returns True for the holder. Window-marker gates now pass allow_reentrant=False so a live lock blocks everyone including its holder; leader-election consumers keep the reentrant default * test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer _initialize_slack_alerting_jobs now reads it for the pod lock manager, and spec=ProxyLogging blocks instance-only attributes |
||
|
ryan-crabbe-berri
|
cbf85a015f
|
feat(proxy): per-key prompt caching toggle via enable_prompt_caching (#36466)
* feat(proxy): per-key prompt caching auto-injection via enable_prompt_caching Adds a key-level enable_prompt_caching toggle that auto-injects Anthropic cache_control breakpoints on requests made with that key, without requiring the gateway-wide enable_anthropic_prompt_caching flag. The flag lives in key metadata, is stamped onto the request root by add_key_level_controls, rides kwargs into both the /chat/completions seeding path and the native /v1/messages path, and reuses every existing gate (anthropic/bedrock only, supports_prompt_caching, client markers win). Client-supplied body values are stripped as an untrusted root control field. Includes the Admin UI switch on key create and key edit plus a read-only settings row, and dedupes the key edit view's drifted initial-values objects. * fix(proxy): drop section comment and suppress LIT011 on key-level prompt caching stamp |
||
|
Deepanshu Lulla
|
84d6666a59
|
feat(router): add required-AND (&) tag prefix and allow_fail_open flag (#36193)
* feat(router): add required-AND (&) tag prefix and allow_fail_open flag
Tag routing supported inclusion-OR and independent "!" negation, but had no way
to express a hard "must match all of these" constraint per request, and no way
for a model group to opt into degrading gracefully instead of raising when a
constraint eliminates every deployment.
Adds a "&tag" prefix for required-AND inclusion, composing with existing plain
(OR) and "!" (negate) tags: negation still applies first, then required tags
narrow the survivors, then plain tags apply today's OR/AND preference logic
unchanged. Adds model_info.allow_fail_open (default false) so a chain can opt
into falling back to the default-tagged pool instead of raising
no_deployments_with_tag_routing when "!" or "&" empties the candidate set;
existing chains without the flag keep today's fail-closed behavior exactly.
* fix(router): gate mixed negation on allow_fail_open and stop diluting required-only requests
Two gaps in the initial required-AND/allow_fail_open change: a "!" exclusion
combined with a plain positive tag that emptied the candidate set raised
unconditionally, bypassing allow_fail_open entirely, since the fail-open check
only looked at required-AND exhaustion. And a request using only "&" tags
could get narrowed down to just the deployment matching an incidental
tag_regex/User-Agent preference, silently dropping other deployments that
satisfied the required tags but had no tag_regex at all.
Fixes both: the fail-open check now fires whenever either "!" or "&" leaves
the candidate set empty, not just "&". And regex/header preference no longer
counts as a positive filter when a required-AND ask is present, so a
required-only request returns every deployment satisfying the required tags
regardless of regex/header matching.
Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
model_info.allow_fail_open field, and removes source comments explaining
the router logic per repository convention.
* fix(router): let allow_fail_open cover a non-empty !/& survivor set that fails the plain-tag preference
The unconditional raise inside the has_positive_filter loop was the one
remaining path a chain could hit despite setting allow_fail_open: when "!"
or "&" leaves a non-empty candidate set but none of the survivors match the
request's plain preference tag or carry "default", the request still raised
instead of degrading. Routes that raise through the same allow_fail_open
check used everywhere else, so it now falls back to the default-tagged pool
for opted-in chains and keeps raising unconditionally for everyone else.
This also let the now-redundant pre-loop empty-candidates shortcut be
removed, since the loop reaches the same outcome on its own.
* fix(router): deny allow_fail_open when an unrecognized required tag is masking a satisfiable answer
A caller could add a single "&" tag no deployment in the group has ever
carried to force an empty required-AND set on demand. On a chain with
allow_fail_open, that emptied set fell back to the default-tagged pool
unconditionally, discarding every other constraint merged into the same
request, including ones inherited from key/team policy, even when the rest
of those constraints were still individually satisfiable.
Before falling back, drop any required tag not carried by any deployment in
the group and recompute: if a specific, non-empty answer exists using only
the recognized tags, the unrecognized tag was the actual cause of the
exhaustion, and fail-open must not paper over it. If every required tag is
already recognized, or none are, there's nothing hidden behind an invented
tag, and fail-open proceeds exactly as before; this keeps a single opted-in
deployment's legitimate catch-all behavior working when a caller's tag
simply doesn't exist anywhere in that group.
Ratchets ANN401 and LIT001 budgets down to reflect fixes already earned in
this branch.
* test(router): cover required-AND, allow_fail_open, and unknown-tag denial across fallback chains and model groups
Extends coverage beyond single-hop scenarios: & exhausting a primary group
falls through to a fallback group exactly like ! already does; !, &, and
allow_fail_open composed together across three chained model groups each
raise or fall back independently per-hop; and the unknown-tag denial from
the previous commit is evaluated fresh per hop rather than leaking state
across groups in a fallback chain.
* feat(router): add model_info.enable_tag_filtering per-model-group override
enable_tag_filtering was router-wide only: an operator turning it on for one
model group that needs tag-driven routing exposed every other model group on
the same proxy to the same tag evaluation, even ones that never use tags.
Adds model_info.enable_tag_filtering, checked against any deployment sharing
a model_name, so a chain can flip the router-wide default in either
direction for itself alone: opt a specific group into filtering while the
rest of the proxy stays off, or opt a group out (e.g. an incident-response
catch-all) while the rest of the proxy enforces it.
Precedence, low to high: router-wide default, then the chain override if
set, then the existing request-level escalation (from key/team settings),
which still only ever turns filtering on, never off, over whatever the
router and chain already decided.
Also regenerates ui/litellm-dashboard/src/lib/http/schema.d.ts for the new
field.
* fix(router): gate plain-tag exhaustion on allow_fail_open when the tag is known to the group
A model group where every deployment is tagged "default" (a legitimate
cross-cutting safety-net pattern) never has an empty default_deployments
list, so the existing exhaustion check (len(new)==0 and len(default)==0)
never fired for a plain positive tag that matched nothing among the
currently healthy candidates. The request silently fell through to whatever
"default"-tagged deployment happened to survive, even when allow_fail_open
was never set and the caller's intent (e.g. quality:high) was never honored.
Adds a check for whether the requested tag is part of the group's real
vocabulary at all: if some deployment configured under this model_name
(regardless of current health) genuinely carries the tag, and nothing
healthy currently matches it, the request now raises by default or falls
back per allow_fail_open, through the same _resolve_or_fail_open gate every
other exhaustion path already uses. A tag that's foreign to the group
entirely (e.g. one meant for an unrelated mechanism sharing the same
request-tags list) keeps falling back to the default pool unconditionally,
unchanged, since there's nothing this group's own routing intent could be
violating.
* fix(router): preserve inherited tag constraints when allow_fail_open discards a caller-caused exhaustion
Adds metadata.caller_tags in litellm_pre_call_utils.py, populated only from
what the request itself supplied (header, body tags, body metadata.tags),
never from key/team metadata merged into the same metadata.tags list.
get_deployments_for_tag now uses it to compute a trusted-only pool before
falling open: a required/excluded tag attributable to the caller can be
discarded on fail-open, one inherited from key/team policy cannot. If the
trusted-only pool is itself empty, allow_fail_open raises instead of
silently routing around an unsatisfiable inherited constraint. When
caller_tags carries no information at all (direct SDK Router usage,
bypassing the proxy layer), behavior is unchanged: unconditional fall-open
to the default pool, exactly as before this fix.
* feat(router): add opt-in tag_routing_prefix for collision-proof tag disambiguation
router_settings.tag_routing_prefix lets a caller explicitly mark which
x-litellm-tags/metadata.tags values are routing directives, exempting
them from the known-tag-vocabulary heuristic used to guard fail-open
against caller-invented "&"/"!" tags. Unprefixed tags keep going
through today's existing handling unchanged (hybrid, no migration
required); default "" is a full no-op.
Fixes a bug caught during live-proxy verification: the prefix-stripped
"confirmed" set kept the "&"/"!" marker character, so it never matched
required_set/excluded_set (which _split_tags always strips bare) -- the
entire trusted-required/excluded-tag mechanism silently no-opped for
its primary use case. Adds regression tests for the bare-value mismatch
and updates existing _chain_allows_fail_open/_tag_known_to_group/
_caller_constraint_sets call sites for the new routing_confirmed/
routing_prefix parameters.
* fix(router): resolve model_info.enable_tag_filtering override from the full model group, not just healthy deployments
Cooldown filtering runs before get_deployments_for_tag, so
_chain_tag_filtering_override only saw the survivors of that filter.
A model group whose only enable_tag_filtering-carrying deployment goes
into cooldown lost the override entirely, silently falling back to
the router-wide default and letting any !/&/tag constraint on that
chain be bypassed by driving the one overriding deployment into
cooldown. Resolve the override from every deployment configured for
the model instead, mirroring _tag_known_to_group's existing pattern.
Verified live: with a bad-key deployment carrying the override forced
into real cooldown via allowed_fails=1, an explicit "!provider:openai"
ban on the remaining deployment reproducibly returned 200 via OpenAI
before this fix and 401 (tag filtering still enforced) after it.
* fix(router): avoid Final-reassignment lint error and a MagicMock router fixture gap from tag_routing_prefix
_chain_tag_filtering_override's try/except reassigned a Final-annotated
name across branches, which basedpyright flags as illegal; extracted
the lookup-with-fallback into its own helper so the binding is assigned
once. Also sets tag_routing_prefix on the bare MagicMock router used by
test_router_tag_regex_routing.py's fixture, which otherwise returns an
auto-generated MagicMock (truthy, non-string) for the new attribute and
crashes _strip_routing_prefix's removeprefix() call.
* fix(router): key inherited-tag protection off provenance, not value subtraction
allow_fail_open's trusted-only pool computed "not caller-attributable"
as required_set - caller_required_set. A caller who resubmits the
exact value of an inherited "&"/"!" tag (e.g. an inherited "®ion:eu"
alongside a caller-supplied "®ion:eu" plus a conflicting
"!region:eu") collapses both origins to the same set value, so the
subtraction zeroes out the inherited requirement's protection too,
letting fail-open route outside a key/team-enforced constraint.
Adds metadata.inherited_tags in litellm_pre_call_utils.py: a snapshot
of "tags" taken after key/team/project policy is merged in but before
this request's own caller-supplied tags are merged on top. A required
or excluded tag is now protected from fail-open discard if it has ANY
inherited backing (set intersection with inherited_tags), regardless
of whether the caller also happens to submit the identical value --
this is what set membership alone could never tell apart under the
old subtraction-based approach. caller_tags is kept (documented as the
complementary record) but no longer consulted for this decision.
Verified live: a virtual key with metadata.tags=["®ion:eu"] hit
with header x-litellm-tags: ®ion:eu,!region:eu (the exact
value-collision attack) reproducibly routed to the OpenAI/us
deployment before this fix and stayed on the Anthropic/eu deployment
after it.
* fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging
Regenerated ruff-strict-budget.json and type-discipline-budget.json
via make lint-ruff-budget-update / lint-type-discipline-budget-update
against the post-rebase merge-base.
* fix(proxy): compute inherited_tags from key/team/project sources directly, not a tags-list snapshot
apply_client_tag_policy_pre_auth (run from user_api_key_auth, for
_tag_max_budget_check) merges the caller's x-litellm-tags header into
the same metadata.tags list before add_litellm_data_to_request ever
runs. The previous inherited_tags snapshot ("whatever's in tags before
this function's own caller-tag merge") therefore misattributed that
caller-controlled value as policy-backed whenever a request arrived
with the header set -- Greptile flagged this as a P1 security finding.
inherited_tags is now built directly from key_metadata/team_metadata/
project_metadata's own "tags" fields, independent of the shared,
pipeline-position-dependent "tags" list's mutation history. Verified
with a direct reproduction mirroring the real pipeline (calling
apply_client_tag_policy_pre_auth on the same data dict before
add_litellm_data_to_request, as user_api_key_auth actually does): the
caller's header tag no longer appears in inherited_tags. Added a
regression test exercising that same call order; confirmed it fails
against the pre-fix snapshot approach and passes against this fix.
* fix(router): make tag_routing_prefix configurable through update_settings/get_settings and UpdateRouterConfig
router_settings.tag_routing_prefix was only ever applied via the
Router() constructor. Router.update_settings's _allowed_settings
(used directly by proxy_server.py's _add_router_settings_from_db_config
for the DB-backed router_settings path) and get_settings's
vars_to_include both omitted it, so an operator relying on that path
had the value silently ignored -- flagged by veria-ai. Also adds it to
UpdateRouterConfig (the pydantic schema behind POST /config/update),
the same bug shape LIT-3152 previously fixed for retry_policy: a field
missing from that schema gets silently dropped by
model_dump(exclude_none=True) before update_settings is ever called.
* chore(ui): regenerate schema.d.ts for UpdateRouterConfig.tag_routing_prefix
Adding tag_routing_prefix to UpdateRouterConfig changed the proxy's
OpenAPI spec; regenerate the dashboard's generated API types to match.
* fix(lint): re-ratchet budget ceilings after rebasing onto litellm_internal_staging
Regenerated ruff-strict-budget.json and type-discipline-budget.json
against the post-rebase merge-base. LIT002/LIT011 ceilings reflect
this branch's true current counts (confirmed unchanged across the
rebase by diffing against the pre-rebase commit); the base's own
counts moved independently.
* fix(lint): replace mutable-collection fallbacks with immutable ones in inherited_tags computation
key_metadata/team_metadata/project_metadata's "tags" fallbacks used
`or {}` / `or []` literals, each a LIT002 mutable-collection-construction
violation that pushed the branch 4 over its ratchet ceiling relative to
a moved base. Swapped to MappingProxyType({}) / () to match the
immutable idiom the rest of tag_based_routing.py already uses; no
behavior change, since both are falsy and only ever read via .get()/
unpacking. Tightens type-discipline-budget.json's LIT002 ceiling back
down to match, fully closing that gap (LIT011 keeps a genuine 1-count
gap from pre-existing, untouched lines in this file, non-gating).
* fix(lint): suppress LIT011 on the two new data[...] mutation sites
Both new lines follow this file's established data[...] mutation
idiom for add_litellm_data_to_request, matching the existing
suppression already on the inherited_tags line.
* test(router): lock in fallback + tag-filtering interaction
Cover the router-level fallbacks mechanism composing with tag-based
routing: a plain negation exhausting a group correctly advances to
the fallback group, the same exclusion tag exhausting every hop
correctly raises, and allow_fail_open resolving locally must not
spuriously trigger an unrelated external fallback.
* chore: retrigger CI now that litellm-docs#814 is merged
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
|
||
|
ryan-crabbe-berri
|
b144b15d48
|
fix(proxy): add config_updated_at audit timestamp for virtual keys (#36488)
* fix(proxy): add config_updated_at audit timestamp for virtual keys updated_at carries Prisma's @updatedAt, so every batched spend flush rewrites it and it cannot distinguish config changes from usage. Add an additive config_updated_at column stamped only by key management writes (update, bulk update, regenerate, block, unblock) via a shared helper, expose it on key responses, and switch the key page's Last Updated to it with a created_at fallback. * test(proxy): assert config_updated_at survives key archival * refactor(proxy): rename config_updated_at to settings_updated_at |
||
|
Mateo Wang
|
02b0ee7608
|
Merge pull request #36503 from BerriAI/litellm_fix_openai_passthrough_stream_cost
fix(proxy): inject streaming usage cost on openai passthrough streams |
||
|
mateo-berri
|
e7c8cff3b7 | fix(proxy): preserve crlf line endings when injecting streamed usage cost | ||
|
mubashir1osmani
|
b0fac57fe4
|
fix(email): stop duplicate legacy invitation email and fix its onboarding link (#36455) | ||
|
mateo-berri
|
1b488f7c2f | fix(proxy): ban caller-supplied aws identity selectors in request bodies | ||
|
mateo-berri
|
938396ef90 | fix(proxy): recognize crlf sse frame boundaries in passthrough reassembly | ||
|
mateo-berri
|
46fb1cd514 | fix(proxy): reassemble fragmented SSE frames and inject logging dependency | ||
|
mateo-berri
|
426b909447 | fix(proxy): inject streaming usage cost on openai passthrough streams | ||
|
tin-berri
|
79d412efc2
|
fix: net prompt-caching savings against the cache-write premium (#36452)
* fix: net prompt-caching savings against the cache-write premium
Prompt-caching savings priced only the cache-read discount and ignored what
the provider charges to create the cache entry. Anthropic bills cache writes
at 1.25x the input rate, so a request that writes a large cache and reads
little from it is a net loss that the dashboard reported as a gain -- or, on
a pure cold write, as a flat zero.
The counterfactual the number answers is "what would this have cost with
caching off", where every token is billed at the input rate. Since
prompt_tokens partitions disjointly into text + reads + writes, that gives
savings = reads * (input - read_rate) - writes * (write_rate - input)
The write term is the premium over the input rate, not the full write cost:
the tokens would have been paid for at the input rate anyway, so only the
markup is attributable to caching.
The premium stays signed rather than clamped. Three models in the pricing map
price writes below input, and clamping would silently drop that saving.
A model with no cache_creation_input_token_cost falls open to the input cost,
yielding a zero premium -- this is why the change is a no-op for the implicit
caching providers (OpenAI, Gemini), which publish no write price, and bites
exactly on Anthropic and Bedrock.
Verified live through the proxy on a mock Anthropic rig across four cases
(cold pure-write, warm pure-read, write-heavy, read-heavy). Reported total
matched the derived net to the cent, including the negatives; the read-only
case is unchanged.
Pre-existing rows are not backfilled, so a range spanning the deploy mixes
gross and net.
* fix: read a zero cache-write price as unpublished, not free
deepseek-chat carries a literal 0.0 cache_creation_input_token_cost. The
fall-open only caught None, so the zero was taken at face value and the
premium became 0 - input_cost -- reporting a fabricated saving of
writes * input_cost on traffic that cached nothing.
No provider gives cache writes away, so a falsy price means the same thing
an absent one does.
* test: pin that the read leg keeps a literal zero price
The two zero prices mean opposite things and the asymmetry was unpinned.
A free cache write is unpublished pricing; a free cache read is real, and
15 models charge for input while serving reads for nothing. Copying the
write leg's falsy fall-open onto the read leg would zero out their savings.
* refactor: resolve caching rates through the established pricing helpers
Addresses Greptile's P1 and P2, and replaces hand-rolled pricing lookup with
the patterns this file and the cost calculator already own:
- Deployment pricing first: rates now resolve through _effective_model_info
(Router.get_deployment_model_info), the same helper the autorouter driver
uses, falling back to _model_info public rates. A deployment with negotiated
cache rates previously priced at the public map -- a 3x error on the repro.
- Individual prices read via _get_cost_per_unit, the cost calculator's
accessor, which also coerces string prices from config.yaml and resolves
service-tier suffixes; the previous raw .get() handled neither.
- Pricing tests no longer monkeypatch litellm.get_model_info; each case now
pins a real pricing-map entry with a fixture-drift assertion, and the
deployment-rate case follows the existing Router-fixture test pattern.
Behaviour on public rates is unchanged: 101 tests pass, including the exact
same live-verified formula.
* fix(cost-optimization): computeCacheLeakage divides net savings by all cached tokens, not reads alone
prompt_caching_savings_spend is net of the cache-write premium since PR #36452.
computeCacheLeakage was still dividing by cache_read_tokens alone, which:
1. Overstates the per-token rate on traffic that writes and reads cache equally:
a 1:1 read:write key shows rate = 0.002, not 0.001, if net savings is /bin/zsh.002
2. Flips the sign on write-heavy traffic: when writes cost more than reads save
(common on Anthropic and Bedrock), the aggregate net can go negative, but
dividing by reads alone would show a positive 'potential savings' for keys
that don't cache yet — recommending they start caching when it's currently
losing money overall
Fix: divide realizedCachingSavings by (cacheReadTokens + cacheCreationTokens),
matching the semantic that a key starting to cache pays those write premiums too.
When the rate is non-positive, price nothing (potentialSavings stays null, renders
as '—'), reusing the existing no-data fallback path. The card can't meaningfully
estimate savings from a losing rate.
Rename discountPerToken → netSavingsPerCachedToken to surface the semantics and
prevent this drift in future.
Update Usage tab and Cache Leakage card tooltips to describe net-of-premium cost.
Add tests for 1:1 read:write traffic and write-heavy negative-net traffic.
|
||
|
Yassin Kortam
|
be5e9000b2
|
perf(spend): write each daily spend batch in one upsert statement (#36448)
The daily spend flush emitted one INSERT ... ON CONFLICT per aggregated key, so every replica put hundreds of statements on the database each interval, all contending for the same handful of hot rows and each holding its row locks for the rest of the enclosing batch transaction. LiteLLM_DailyTagSpend felt it worst because a request writes one row per tag, and litellm adds two user-agent tags of its own by default. A batch now goes out as a single multi-row statement. Rows are folded by the conflict tuple first, and every nullable member of that tuple is normalized to '': a NULL can never match itself in a unique index, so such a row was re-inserted on every flush rather than aggregating, and a NULL model made prisma reject the whole batch. |
||
|
Deepanshu Lulla
|
363d56f917
|
feat(proxy): add per-deployment keepalive_seconds SSE heartbeat to prevent load-balancer timeout on long streams (#34423)
* feat(proxy): add per-deployment keepalive_seconds SSE heartbeat for long-running streams
Adds _iter_with_keepalive, _keepalive_from_deployment_config, and
_resolve_keepalive_seconds helpers to proxy_server.py. When enabled
(keepalive_seconds > 0 in request body or deployment litellm_params),
async_data_generator emits ': ping\n\n' SSE comment frames every N
seconds during idle upstream intervals, preventing load-balancer
idle-timeout drops on long chain-of-thought reasoning streams.
The hot path (keepalive_seconds absent or 0) is a plain async-for with
no per-chunk Task wrapping — zero overhead. Includes 8 new unit tests
covering sentinel emission, hot-path pass-through, early-close cleanup,
priority resolution, deployment-config lookup, and end-to-end heartbeat
emission through async_data_generator.
Registers keepalive_seconds in all_litellm_params (types/utils.py) so
the parameter is not stripped from request bodies. Adds the field to
LiteLLMParamsTypedDict and GenericLiteLLMParams (types/router.py) so
deployment YAML config is parsed and validated.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(proxy): narrow BaseException to CancelledError to fix BLE001 strict lint gate
* fix: use explicit None check instead of truthiness in keepalive_seconds extraction
`float(raw or 0)` would treat any falsy value (including the integer 0)
as absent and substitute 0.0 before float() saw it. Replace with
`float(raw) if raw is not None else 0.0` so a caller-supplied zero is
correctly passed through to the `value <= 0` guard that disables
keepalive, rather than being silently overwritten.
* fix(proxy): don't guess a deployment's keepalive_seconds when model_id is missing
When a streaming response lacks _hidden_params.model_id, the fallback that
looks up keepalive_seconds by model_name previously returned the first
configured deployment's value, which could apply the wrong interval (or
override an explicit disable) when multiple deployments share the same
model_name with different keepalive_seconds settings. Only resolve the
fallback when every deployment agrees; otherwise leave it unset.
* fix(proxy): also treat an unset keepalive_seconds as disagreement in the fallback
The model_name fallback for keepalive_seconds only compared configured
values, filtering out deployments that leave the field unset entirely.
That meant a deployment with no keepalive_seconds configured could still
inherit a sibling deployment's interval when model_id is unavailable.
Compare the raw per-deployment value (including None for unset) so an
unconfigured deployment never silently adopts another's heartbeat.
* fix(proxy): deployment-level keepalive_seconds: 0 is a hard disable clients can't override
Previously an authenticated client's request-level keepalive_seconds always
took precedence over the deployment default, including when a deployment
operator explicitly set keepalive_seconds: 0 to disable heartbeats. That let
any client re-enable heartbeats for a deployment the operator opted out of,
using them to keep an idle-looking stream alive past a load balancer's idle
timeout and hold a parallel-request slot open longer than intended.
Treat an explicit deployment-level 0 as authoritative: resolve the
deployment's configured value first, and short-circuit to disabled before
ever looking at the request body if the deployment hard-disabled it.
* fix(proxy): a stale (unresolvable) model_id must not fall through to model_name guessing
A populated _hidden_params.model_id names the specific deployment that
served a stream. If that ID no longer resolves (e.g. a deployment removed
by a config reload mid-stream), the resolver was falling through to the
model_name-based fallback, letting a currently-live sibling deployment's
keepalive_seconds silently apply to a stream it never served. Return None
once a populated model_id fails to resolve, rather than degrading to a
guess.
* fix(proxy): keepalive_seconds is operator-only by default; require deployment opt-in for client override
A security review flagged that a client's request-level keepalive_seconds
could unilaterally enable heartbeats for any deployment, even one that
never configured keepalive_seconds at all, letting an authenticated client
defeat load-balancer idle timeouts and hold a parallel-request slot open
for longer than the deployment operator ever intended, with no way for
the operator to prevent it short of explicitly setting keepalive_seconds: 0.
Add allow_client_keepalive_override (default False) to LiteLLMParamsTypedDict
and GenericLiteLLMParams. _resolve_keepalive_seconds now ignores the request
body's keepalive_seconds entirely unless the resolved deployment explicitly
grants override permission; only the deployment's own configured value (or
disabled, if unset) applies otherwise. An explicit deployment-level 0 still
takes priority over everything, including a grant of override permission.
* fix(proxy): register allow_client_keepalive_override in all_litellm_params
Caught during live proxy verification against the real Anthropic API:
allow_client_keepalive_override was added to LiteLLMParamsTypedDict and
GenericLiteLLMParams but never registered in all_litellm_params, so it
leaked straight through into the provider request body as an unrecognized
field. Anthropic rejected every call on a deployment that had this field
configured with a 400 ("Extra inputs are not permitted"), regardless of
its value. Register it alongside keepalive_seconds so it's stripped
before reaching the provider, matching what keepalive_seconds already
does.
* feat(proxy): support keepalive_seconds via x-litellm-keepalive-seconds header
Some clients (e.g. the Vercel AI SDK) can set custom headers more easily
than extra JSON body fields. Add x-litellm-keepalive-seconds, following
the existing x-litellm-timeout/x-litellm-stream-timeout/x-litellm-num-retries
convention in LiteLLMProxyRequestSetup: the header merges into the same
data["keepalive_seconds"] field the request body already populates, so it
goes through the exact same _resolve_keepalive_seconds precedence and the
allow_client_keepalive_override gate -- a header can't enable heartbeats
for a deployment that hasn't opted in any more than the body field can.
Verified live against the real Anthropic API: the header produces real
heartbeats on an opt-in deployment (88 pings over a genuine long-reasoning
stall) and is silently ignored on a deployment without override permission
(0 pings), matching the existing body-field behavior exactly.
* chore: rebase onto litellm_internal_staging, drop unrelated credential_migration.py reformat, fix budget-ratchet drift
Rebased onto the current litellm_internal_staging (merge-base was 5 days
stale). Dropped the now-redundant schema.d.ts-only regen commit entirely
(the new base's own schema.d.ts already supersedes it) and regenerated
schema.d.ts fresh against the new base.
Reverted litellm/proxy/management_endpoints/credential_migration.py to
exactly match litellm_internal_staging: it was a pure reformat with no
semantic change, unrelated to this PR, flagged by review as unnecessary
noise in an encryption-migration file.
Fixed two lint-budget-ratchet failures caused by the base's ceilings
tightening since this branch last synced (other merged work lowered
ANN401/LIT001 budgets; this code was previously under budget and didn't
change):
- _iter_with_keepalive's aiter param: Any -> AsyncIterator[Any], a real
narrowing (it's always the result of .__aiter__()).
- _keepalive_from_deployment_config/_resolve_keepalive_seconds's
request_data param: dict[str, Any] -> Mapping[str, Any], matching the
existing read-only-dict convention already used elsewhere in this file
(_apply_ssrf_general_settings, _build_redis_usage_cache, etc.) for
params that are only ever read, never mutated.
- response/raw params: dropped the explicit `Any` annotation to match
async_data_generator's own (deliberately unannotated) `response` param,
its actual caller.
- litellm_pre_call_utils.py's new headers param: dict -> Mapping[str, str],
same read-only-dict rationale.
* fix(proxy): freeze the transient collections in the keepalive helpers
_iter_with_keepalive and _keepalive_from_deployment_config built a set
literal for asyncio.wait, a set comprehension for the per-deployment
config-agreement check, and two dict-literal fallbacks, all flagged by
the LIT002 mutable-collection-construction gate. Switched to a tuple
for asyncio.wait, a frozenset-wrapped generator plus next(iter(...))
for the config check, and a shared MappingProxyType({}) empty mapping
for the fallbacks.
* fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback
Greptile P1: when a streaming request falls back from model group A to
group B and the response's _hidden_params carries no model_id,
_keepalive_from_deployment_config fell straight through to guessing
via request_data["model"], which still names the pre-fallback group A
since the fallback handler mutates its own local **kwargs copy, not
this dict. request_data[metadata|litellm_metadata]["model_info"]["id"],
by contrast, is mutated on this same dict by
Router._update_kwargs_with_deployment on every attempt including
fallbacks (the same source ProxyLogging._build_litellm_call_info uses
for logging), so check it before falling through to the model-name
guess.
Added two regression tests that fail on the prior code (assert
get_model_list is never called once metadata.model_info.id resolves)
and pass with the fix.
* Revert "fix(proxy): trust metadata.model_info.id over the stale model group after a router fallback"
This reverts commit d7790678645695b20f25880315238b49c31a9143.
* fix(proxy): satisfy the new LIT010/ANN001 gates in the keepalive helpers
litellm_internal_staging picked up a LIT010 (every local/module variable
must be declared Final unless it's genuinely rebound) and tightened
ANN001 (missing parameter annotations) since this branch last synced.
Annotated every single-assignment local and module constant with
Final, suppressed pending's loop-carried reassignment with
# rebind-ok, and typed the previously-bare response/raw parameters as
object with isinstance narrowing at their use sites instead of cast
(LIT006 discourages cast; validate into a concrete type instead).
Also swapped the hand-rolled getattr(response, "_hidden_params", None)
+ isinstance(hidden, dict) check for the existing
get_hidden_params_dict() helper already used for this exact purpose
elsewhere in this file and in common_request_processing.py.
* fix(proxy): re-resolve keepalive_seconds per chunk to track mid-stream fallback
Greptile P1: the router can perform a mid-stream fallback to a
different deployment partway through a stream (MidStreamFallbackError
in router.py), and Router._apply_fallback_hidden_params_to_item merges
the fallback deployment's hidden params onto every subsequent chunk.
But _resolve_keepalive_seconds was only ever called once, before
iteration started, against the pre-fallback response wrapper, so a
stream that fell back to a deployment with a different (or disabled)
keepalive policy kept using the original deployment's interval for the
rest of the stream.
_iter_with_keepalive now takes a resolve_keepalive_seconds(item)
callback and re-resolves after every real chunk using that chunk's own
_hidden_params (which do carry the fallback deployment's identity),
rather than trusting the value picked before iteration began. Updated
the three existing timing tests to inject a constant-returning
resolver, since they pin the sentinel/cancellation mechanics rather
than re-resolution, and added two regression tests (interval lowered
and raised mid-stream) that fail against the prior static-resolve
signature and pass with the fix.
* fix(proxy): keep re-resolving keepalive even when a stream starts disabled
Greptile P1: a stream that starts on a deployment with keepalive off
(or unset) skipped _iter_with_keepalive entirely at the call site, so
a mid-stream fallback to a deployment that enables it never got a
chance to activate heartbeats for the rest of that stream, risking the
exact load-balancer idle-timeout this feature exists to prevent.
_iter_with_keepalive now has an internal fast path for
keepalive_seconds <= 0 that still re-resolves after every chunk (no
asyncio.create_task/wait overhead while inactive, same cost as a bare
async for), so activation from a disabled start works the same way
deactivation and interval changes already do. The caller now only
skips wrapping entirely when there's no router to ever fall back
through in the first place (llm_router is None), rather than whenever
the first chunk's deployment happens to start with keepalive off.
Added a regression test that starts keepalive_seconds=0, has the
resolver enable a short interval on a later chunk, and asserts
sentinels appear afterward; it fails against the prior
call-site-gated code and passes with the fix.
* perf(proxy): memoize keepalive resolution per chunk's model_id
_resolve_keepalive_seconds ran a full llm_router.get_deployment() Pydantic
rebuild after every streamed chunk, even when keepalive was unconfigured
anywhere in the deployment list, since async_data_generator wraps every
stream once a router exists. Caching the result by model_id keeps mid-stream
fallback re-resolution correct while paying the router lookup once per
deployment instead of once per token.
* fix(proxy): expire cached keepalive resolution after a bounded TTL
veria-ai flagged that caching by model_id alone lets an already-in-flight
stream keep evading a live config reload (deployment removed, keepalive
disabled, or client override revoked) for the rest of the stream. Expiring
the memo after _KEEPALIVE_CACHE_TTL_SECONDS bounds that window instead of
freezing the resolved value for the stream's full lifetime, while still
avoiding a full deployment rebuild on every chunk in the steady state.
Also fixes add_litellm_data_for_backend_llm_call's now-required request_data
kwarg in the header-merge test, picked up by rebasing onto
litellm_internal_staging.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
fancybear-dev
|
f1ed4690bb
|
fix(proxy): treat SAML as configured in UI SSO detection (#36196)
* fix(proxy): treat SAML as configured in UI SSO detection _has_user_setup_sso only checked OAuth client IDs, so SAML-only setups left /.well-known/litellm-ui-config sso_configured=false and the login button gray even when SAML IdP metadata was set. Include SAML_IDP_METADATA_URL / SAML_IDP_METADATA_XML so UI discovery matches the login redirect path. * chore: adhere to comment policy Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: apply suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
76ad1c319d
|
feat(proxy): add GET /v1/indexes to list vector store indexes (#36289)
* fix(scripts): stop type-discipline checker reading Literal strings as forward refs The checker re-parsed every string constant inside an annotation as a forward reference, so Literal["list"] was counted as the mutable list type. Skip Literal subtrees and ratchet the LIT001 ceiling down to the corrected count. * fix(proxy): keep lazy openapi snapshot fragments for transitively imported features generate_snapshot skipped register_fn for any feature module already in sys.modules, so a module pulled in transitively by an earlier feature never mounted its routes and its fragment silently vanished on regen (vector_store_management). Route collection also matched path_prefixes only, dropping suffix-matched routes from fragments. Register every feature and collect routes with feat.matches, mirroring the runtime loader. * feat(proxy): add GET /v1/indexes to list vector store indexes /v1/indexes was POST-only, so indexes created through it could never be viewed again. Add an admin-only list endpoint returning the stored index rows newest first, fix the stale index_create docstring curl, and regenerate the lazy openapi snapshot and dashboard schema types. * chore(proxy): defer lazy openapi snapshot catch-up regen to a follow-up Reverts _lazy_openapi_snapshot.json and schema.d.ts to the staging versions. The snapshot was months stale, so regenerating it here buried the actual change under ten thousand generated lines. A follow-up will land the regen together with CI enforcement that keeps the snapshot current. Until then GET /v1/indexes is served but absent from the dashboard's generated types, which the UI step needs anyway. * fix(proxy): use Annotated dependency to avoid new B008 violation |
||
|
ryan-crabbe-berri
|
c40828509b
|
fix(reset_budget_job): atomic budget cascade with chunked reset scans (#36287)
* fix(reset_budget_job): advance budget_reset_at atomically with the spend cascade A postgres timeout mid-cascade previously left LiteLLM_BudgetTable rows stamped for the next window while team member, enduser, org and tag spend stayed at cap, so every later tick skipped them until the window rolled over. All cascade writes and the budget_reset_at advance now share one prisma batch transaction; a failed run persists nothing and the rows stay due for the next ~10 minute tick. Cache and counter invalidation runs only after commit, and the catch-all enduser log line now names the cascade. * fix(reset_budget_job): elect one runner per tick and chunk the reset scans Every pod and worker previously ran the reset job every ~10 minutes, each fetching every expired row with no limit and writing one giant transaction at the same calendar-aligned boundary; that concurrency is what piled up postgres lock contention and timeouts. The job now takes the shared PodLockManager redis lock (no redis keeps the old behavior), and each phase walks its due rows in 500-row chunks, one transaction per chunk, stopping when a chunk is short, makes no forward progress, or hits the per-run cap; leftovers wait for the next tick. * chore(lint): ratchet budget ceilings down for fixed violations * fix(reset_budget_job): harden chunk loop, fail open on redis errors, heartbeat the lock Review fixes on the two prior commits. Reset scans now skip rows with no budget_duration, so permanently due rows can neither starve a phase nor have a lifetime cap zeroed every tick. Chunk progress counts rows whose new budget_reset_at actually cleared the cutoff, so a zero-length duration cannot burn the per-run chunk cap. A failed lock acquire only skips the run when another pod verifiably holds the lock; a broken redis runs unguarded instead of silently disabling resets fleet-wide. Partial row failures report real progress and fire the failure hook without killing the phase. The leader re-asserts the lock between phases and stops if another pod took over, and the budget window advance uses update_many so a tier deleted mid-chunk cannot abort the transaction. Lint budget ceilings re-ratcheted for the net-fixed violations. * fix(reset_budget_job): renew the leader lease and reject non-positive budget durations Bot review follow-ups. PodLockManager now extends the lock TTL when the holding pod re-acquires, via an atomic compare-and-expire script with a plain SET fallback, so a run longer than the TTL keeps its lease instead of silently sharing the job with another pod. The positive-duration validation that team member endpoints already had is hoisted to management common_utils and applied to key, internal user, budget, customer and team intake, so a tenant can no longer create zero-duration budgets whose permanently due rows starve other tenants' resets. Such durations now return 400 at intake; existing rows are untouched. * refactor(reset_budget_job): defer leader election to a follow-up PR * fix(reset_budget_job): satisfy strict lint gates String defaults for the two getenv calls (PLW1508) and the chunk outcome returns moved to try/else (TRY300). |
||
|
yucheng-berri
|
ade5a425e8
|
fix(proxy): isolate guardrail load failures per row (#36432)
* fix(proxy): isolate guardrail load failures per row One DB guardrail row that fails to initialize aborted the whole _init_guardrails_in_db loop, so a single typo'd guardrail type or a missing required param left the proxy running with zero DB guardrails registered and requests that should have been blocked reaching the provider. Catch per row around sync_guardrail_from_db, log the guardrail name, id and error, and continue with the remaining rows. The failing row's id is still added to db_guardrail_ids before the attempt so reconcile_db_guardrails cannot mistake a live row for a deleted one. * test(proxy): drop inline note and record reconcile via a handler double Replaces the patched bound method with an InMemoryGuardrailHandler subclass that records what reconcile_db_guardrails received, so the test injects a double instead of swapping a method on a live object. |
||
|
yuneng-jiang
|
3726bceb53
|
Merge pull request #36336 from BerriAI/litellm_/standard-lists-api-d1dc4a
test(proxy): guard management_v1 against fastapi names removed in supported releases |
||
|
Yassin Kortam
|
ade805ef0c
|
feat(rate limiting): configurable estimated output tokens per key, team and model (#36143) | ||
|
yucheng-berri
|
e014b341c8
|
feat(ptu): gate PTU flat-cost attribution behind an opt-in env var (#36138)
LITELLM_ENABLE_PTU_COST_ATTRIBUTION, read through get_secret_bool and defaulting to
false, makes the whole PTU flat-cost feature inert unless an operator opts in. The
daily rollup cron is not registered at all, so no sentinel row is ever written;
/model/new and /model/{id}/update reject a request that carries any PTU model_info
field with a 400 naming the fields and the env var rather than dropping them; the
daily activity read path reports zero flat cost; and the model add and edit forms
hide the four PTU inputs.
The read gate lives where flat cost enters SpendMetrics rather than in the aggregated
SQL select. /team/daily/activity, the endpoint the Usage page reads, is served by the
paginated find_many path and never runs that query, so forcing the select to a
constant zero would have left the reporting surface that matters still showing flat
cost.
Sentinel row filtering is deliberately not gated. An operator can enable the flag,
accrue rows under the __ptu_flat_cost__ api_key, then disable it, and those rows stay
in LiteLLM_DailyTeamSpend; gating the filter too would surface the sentinel as a bogus
api_key and mint a provider bucket for its empty provider. Response fields keep their
shape and report 0.0, so typed clients are unaffected, and the migration and the
ModelInfo field declarations are untouched.
The write gate reads the incoming request rather than the merged deployment, so a
model configured during an earlier opt-in stays editable, and the edit form drops the
PTU keys from the payload instead of sending nulls that would clear stored config.
The dashboard reads the flag from a read-only enable_ptu_cost_attribution key on
/get/ui_settings, computed from the environment on every read. It is deliberately not
an allowlisted persisted setting, and PATCH /update/ui_settings rejects it with a 400,
so an admin cannot flip an env-gated feature from the UI.
Two review findings on the gate itself. The PTU clear loop now runs only when the
feature is enabled: the write gate rejects a value but lets an explicit null through,
and a client round-tripping a model_info blob sends the PTU keys as nulls, so a
disabled proxy would have quietly erased a billing configuration set up during an
earlier opt-in. Disabling pauses PTU rather than discarding its setup. And the
dashboard flag is re-read every thirty seconds instead of the hour the other UI settings
use, since those are persisted records while this one tracks the proxy process; a
restart that flips the variable would otherwise leave the model form offering inputs
the backend now rejects. The flag is polled rather than only marked stale, since a form
that stays mounted and focused never refetches on its own.
The read gate checks the row before the flag. It runs once per metric accumulation and a
record fans out across roughly a dozen breakdowns, while the flag reads through the secret
manager uncached, so consulting it for every accumulation put thousands of lookups on a
shared endpoint that made none before. Only a row actually carrying flat cost reaches it.
|
||
|
Deepanshu Lulla
|
0580465384
|
feat(router): add per-deployment allowed_fails_policy and cooldown_time override support (#34416)
* feat(router): add per-deployment allowed_fails_policy and cooldown_time override support Three bugs fixed in the router cooldown system: (1) deployment-level allowed_fails and allowed_fails_policy in model_info now take precedence over router-level settings in _should_cooldown_deployment; (2) failed fallback deployments now get evaluated for cooldown via _trigger_cooldown_for_failed_deployment, bypassing the Logging dedup gate; (3) DualCache promotes Redis cooldown entries using default 600s TTL instead of true remaining cooldown time -- _corrected_active_cooldown now evicts expired entries and corrects stale in-memory TTLs on backfill. Adds ServiceUnavailableError, BadGatewayError, and NotFoundError fields to AllowedFailsPolicy and cooldown_time to LiteLLMParamsTypedDict. * fix(router): gate fallback cooldown trigger on has_logged_async_failure; use only litellm_metadata for deployment ID * fix(router): use X | Y union syntax to fix UP007 strict lint gate * test(router_utils): add coverage for _trigger_cooldown_for_failed_deployment and has_logged_async_failure gate * test(router_utils): cover deployment cooldown override and exception swallow paths * fix(router): add InternalServerError/ServiceUnavailableError/BadGatewayError/NotFoundError to router-level get_allowed_fails_from_policy * fix(router): format router.py and add router-level policy tests * test(router): add CI-visible coverage for per-deployment cooldown policy Tests for `_get_deployment_cooldown_policy`, `_resolve_allowed_fails_from_policy`, and `_should_cooldown_based_on_deployment_policy` (cooldown_handlers.py), the `_corrected_active_cooldown` branches in CooldownCache, and the four new exception-type branches in `Router.get_allowed_fails_from_policy` (router.py) -- all in `tests/test_litellm/` which the enterprise-routing CI job runs. * fix(router): use is not None guard for cooldown_time_override in should_cooldown_based_on_allowed_fails_policy A cooldown_time_override of 0 was previously treated as falsy and silently fell through to the router-level cooldown_time value. Switched to an explicit is not None check so that zero is honored as a valid override. Added a regression test covering the zero case. * fix(router): honor has_logged_async_failure and metadata for fallback cooldown; support both model_info and litellm_params locations Manual verification against a live proxy surfaced that the fallback-cooldown-gap trigger never actually fired: the has_logged_async_failure check read a plain attribute that Logging never sets (the real flag lives in model_call_details), and the deployment_id lookup only trusted litellm_metadata, which regular chat completions never populate (only batch/thread/file endpoints do). Router overwrites model_info on whichever key is present before every attempt, so metadata is equally authoritative there, not caller-controlled as previously assumed. Also let allowed_fails/allowed_fails_policy/cooldown_time be set under either model_info or litellm_params, each preferring its own canonical location. * fix(router): fix ContentPolicyViolationError policy shadowing and partial-policy zero-threshold Two bugs from Greptile review on PR #34416: - ContentPolicyViolationError subclasses BadRequestError, so listing BadRequestError first in _EXCEPTION_POLICY_FIELDS made the isinstance check always match BadRequestError for content-policy errors, using the wrong allowed_fails threshold. Reordered so the subclass is checked first. - A deployment with a partial allowed_fails_policy and no deployment-wide allowed_fails forced allowed_fails_override=0 for any exception type its policy didn't cover, cooling the deployment down on the first unrelated failure. Now defers to router-level behavior for uncovered exception types instead of forcing an immediate cooldown. * fix(router): only trust a metadata/litellm_metadata bucket the router itself wrote deployment info into veria-ai flagged that preferring litellm_metadata whenever present could pick up a caller-supplied litellm_metadata.model_info.id (preserved via allow_client_pricing_override) instead of the metadata bucket the router actually populated for a regular completion's fallback attempt, naming an arbitrary "victim" deployment for cooldown. Router._update_kwargs_with_deployment() always writes model_info and deployment_model_name into the same bucket together. Only trust a bucket that carries deployment_model_name alongside model_info, since that marker is only ever set by the router itself, not by request-body metadata. * test(router): add regression coverage for ContentPolicyViolationError policy shadowing The subclass-ordering fix in commit 38fe4e4490 had no regression test. Verified the new test fails on the pre-fix ordering (asserts 2, got 10) before restoring the fix, and confirmed the same behavior through the full _should_cooldown_deployment call path against a real Router instance. * fix(router): let explicit allowed_fails_policy entries override the generic 4XX cooldown exclusion _is_cooldown_required skips cooldown evaluation for any 4XX status outside {429, 401, 408, 404} by default, since a generic client error is usually not the deployment's fault. BadRequestError and ContentPolicyViolationError both carry status 400, so their AllowedFailsPolicy fields (BadRequestErrorAllowedFails, ContentPolicyViolationErrorAllowedFails, both router-level pre-existing and the new deployment-level ones) were silently unreachable: an operator could set them to any value with no effect, since _is_cooldown_required blocked cooldown evaluation before that policy was ever consulted. _should_run_cooldown_logic now also checks whether an explicit allowed_fails_policy entry (deployment-level or router-level) covers the exception's type, and if so, proceeds with cooldown evaluation regardless of the generic status-code exclusion. The exclusion remains the default for exception types with no explicit policy. Verified live against a mock-triggered ContentPolicyViolationError (config-level mock_response, azure/gpt-4.1-mini deployment) with BadRequestErrorAllowedFails=100 and ContentPolicyViolationErrorAllowedFails=0 on the same deployment: it now cools down after exactly one ContentPolicyViolationError instead of never cooling down. * fix(router): use the router-stamped failed_deployment_id for fallback cooldown targeting Greptile flagged a real gap in the metadata-bucket-based deployment lookup: for a generic-API-call fallback, the router writes the current attempt into litellm_metadata, but a stale "metadata" bucket carrying the same deployment_model_name marker (from an earlier point) would be picked first, cooling the wrong deployment. Router already has a more robust, pre-existing mechanism for this exact problem: _set_failed_deployment_id_on_exception stamps the failing deployment's id directly onto the exception at the point of failure, immune to metadata-bucket ambiguity since a caller can't influence it and it doesn't depend on which bucket the current call type happens to use. It just wasn't called from _ageneric_api_call_with_fallbacks_helper's except block, unlike _completion/_acompletion. Added the missing call there (matching the existing pattern exactly), and changed _trigger_cooldown_for_failed_deployment to prefer exception.failed_deployment_id when present, falling back to metadata-bucket inspection only for call paths that don't stamp it yet. Verified live: the standard fallback-cooldown-gap scenario (two bad-key deployments in a fallback chain) still correctly cools down both the originally-called and fallback deployment. * fix(router): address human review on per-deployment cooldown overrides Scope allowed_fails_policy override to deployment-level only (a router-level policy predates this feature and must keep its existing behavior), exempt advisor-orchestration failures from the fallback cooldown trigger, keep the single-deployment model group protection intact against a generic deployment-level allowed_fails, make cooldown_time precedence consistent across resolution paths, fix a falsy-zero swallowing bug in the router-level allowed_fails fallback, and make allowed_fails_policy resolution fall through to the next matching exception type instead of stopping at the first unset field. Also restrict allowed_fails/allowed_fails_policy/cooldown_time to model_info: litellm_params gets copied into the actual provider request, so a router-only setting placed there would leak into that request. * test(router): update test_cooldown_handlers.py for the deployment-policy signature change Surfaced by the rebase: this mirrored test file (tests/test_litellm/ mirrors litellm/) predates the router_unit_tests/ coverage added earlier in this PR and was still calling _should_cooldown_based_on_deployment_policy with its old 4-argument signature and asserting the now-removed litellm_params cooldown_time location. * test(router): update test_fallback_event_handlers.py for model_info-only cooldown_time Another mirrored test file surfaced by the rebase that still asserted the now-removed litellm_params.cooldown_time location. * fix(router): match cooldown-duration precedence in the fallback path to the primary path _trigger_cooldown_for_failed_deployment only checked deployment config before falling back to the router default, skipping the response Retry-After header step that Router.deployment_callback_on_failure applies on the primary path. * fix(router): restore litellm_params.cooldown_time as a pre-existing fallback cooldown_time already had litellm_params support on Router.deployment_callback_on_failure before this PR; the earlier model_info-only restriction (aimed at the leak concern for the genuinely new allowed_fails/allowed_fails_policy fields) incorrectly dropped that pre-existing capability too. model_info still takes priority when both are set. * fix(router): keep the fallback-cooldown trigger in sync with #35104's review fixes Applies the same two fixes landed on the split-out PR #35104 (which #34416 still duplicates until it's rebased onto the merged base): increment the deployment's per-minute failure counter before evaluating cooldown, and require the server-stamped failed_deployment_id instead of trusting a metadata bucket, since neither "metadata" nor "litellm_metadata" can be told apart from a caller-supplied one without knowing the call's function_name. * fix(router): freeze the model_info fallback mapping to satisfy the type-discipline gate * fix(router): defer f-string interpolation in fallback-cooldown debug logs * fix(router): annotate cooldown-path locals with Final to satisfy the LIT010 budget * fix(router): suppress reportPrivateUsage for cross-module cooldown helpers * fix(router): don't cool down deployments for request-scoped 404s on generic API fallbacks * fix(router): stamp the dynamic client-side-credential deployment id, not the shared static one * fix(router): keep up with upstream typing modernization and Final-annotation ratchet * fix(router): don't cool down deployments for a caller-supplied x-litellm-timeout * fix(router): stamp dynamic client-side-credential id in completion fallback paths too The generic-API-call helper already stamped the effective (dynamic-if-client-side-credential) deployment id on exceptions, but the regular _completion/_acompletion exception handlers still stamped the static shared deployment's id. A tenant using invalid forwarded credentials could generate repeated failures attributed to, and eventually cooling down, the shared deployment other tenants rely on. Extracted the stamping logic into one shared helper used by all three call sites (generic API, sync completion, async completion) so the fix and future changes to it stay in one place. * fix(proxy): recognize body-supplied timeout/request_timeout/stream_timeout as caller-controlled client_side_timeout was only set when the caller used the x-litellm-timeout header, but Router._get_timeout also resolves the effective timeout from kwargs["timeout"], kwargs["request_timeout"], and kwargs["stream_timeout"], all settable directly in the request body (and x-litellm-stream-timeout wasn't marked either). A caller could set any of those to a near-zero value, force a 408 on every deployment in a fallback chain, and cool down deployments other tenants rely on without the guard in _trigger_cooldown_for_failed_deployment recognizing it as caller-controlled. Also strip any client-forged client_side_timeout from the request body so the marker is always server-computed. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
yucheng-berri
|
457be8f00a
|
feat(ptu): surface PTU flat cost on the daily activity read path (#35391)
Aggregate the ptu_flat_cost written by the rollup into SpendMetrics.flat_cost and DailySpendMetadata.total_flat_cost, so /team/daily/activity returns flat cost alongside per-request spend. The aggregated SQL path selects ptu_flat_cost only for LiteLLM_DailyTeamSpend and a constant zero for the other daily tables, keeping the response shape uniform. Rows written under the PTU sentinel api_key add their flat cost to every parent bucket (per-model, per-day, per-team totals) but never appear as an api_key row in any breakdown, and are excluded from the per-request provider breakdown; the sentinel string is not a real key alias. Both flat_cost and total_flat_cost default to zero, so a read of any entity without PTU config is unchanged. The sentinel row now keys on the deployment id, so the per-model breakdown keys it on model_group instead. That breakdown key is rendered directly as a label by the Usage page and the daily_with_models export, and a deployment id there would read as a UUID. Two deployments sharing a public name merge under it, which is the collapse the write path used to do by summing them into one row. Request rows are untouched and still key on model, since their model_group is a routing concept rather than a display name. |
||
|
Deepanshu Lulla
|
9ce96c2d34
|
feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418)
* feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars
Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and
two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both
setters after assigning litellm_trace_id so every JSON log record emitted within the
async request context carries trace_id and, when provided, session_id — enabling log
correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without
any changes to individual log call sites.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields
* fix(logging): always reset session_id_var to empty string when no session_id provided
* feat: gate request correlation IDs in logs behind request_correlation_in_logs flag
* refactor: move correlation ID injection into CorrelationContextFilter
* feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload
Plaintext log lines (json_logs off) now get the same trace_id/session_id
suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a
visible effect regardless of log format.
StandardLoggingPayload gets a new independent session_id field, populated
from litellm_session_id. trace_id's existing session_id-first fallback is
preserved when request_correlation_in_logs is off; with the flag on, an
explicit litellm_trace_id now takes priority over litellm_session_id so
the two fields carry genuinely independent values.
* fix(logging): restore correlation context after nested calls; sanitize correlation ids
Addresses two review findings on this PR.
CorrelationContextFilter's trace_id/session_id contextvars were set on every
Logging.__init__ but never reset, so a nested LiteLLM call sharing the same
asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call,
an MCP sampling call) would leave the outer request's subsequent log lines
stamped with the nested call's ids instead of its own. set_trace_id/
set_session_id now return their contextvars.Token, and Logging stores them
and resets both once its own success/failure handler actually completes,
via a new idempotent _restore_correlation_context() called from all four
terminal handlers.
set_trace_id/set_session_id also now strip control characters and bound
length before storing a caller-controlled trace_id/session_id, since these
values can originate from request input (litellm_session_id, x-litellm-
trace-id) and get interpolated into plain-text log lines - without this, a
caller could embed \r/\n or escape sequences to forge fake log entries.
* fix(logging): restore correlation context after nested calls, not before
The previous commit called _restore_correlation_context() as the first
line of each terminal handler, before that handler's own callback
dispatch loop runs. That's backwards: a nested LiteLLM call triggered
from within a callback (e.g. a guardrail's own LLM-as-judge call) would
then capture the *already-reset* value as its own pre-call baseline,
and its own reset would restore to that instead of the true outer
value - verified live to still leak.
success_handler/async_success_handler/failure_handler/async_failure_handler
are now thin wrappers: the original bodies move to
_success_handler_body/etc, called inside a try/finally that restores
context only once the full body - including any nested calls its own
callback dispatch triggers - has actually finished, mirroring proper
stack-scoped nesting semantics.
* test(logging): cover async_failure_handler's correlation-context restore
Codecov flagged the new async_failure_handler wrapper (try/finally around
_async_failure_handler_body) as uncovered - the method had no direct test
at all before this PR's refactor split it into a wrapper. Adds a test that
awaits it directly and asserts both that async_log_failure_event still
fires and that _restore_correlation_context() puts the pre-call
trace_id/session_id back.
* fix(logging): restore correlation context by value, not by contextvars.Token
veria-ai correctly flagged that contextvars.Token.reset() only works in the
exact Context it was created in, and litellm's async success path (and
streaming failure path) dispatch async_success_handler/async_failure_handler
via asyncio.create_task and the global logging worker - a different Context
than Logging.__init__ ran in. reset_trace_id/reset_session_id silently
swallowed the resulting ValueError, so the restore was a no-op for exactly
those paths. Verified independently: reproduced the raw contextvars
behavior, then confirmed litellm's async success dispatch really does go
through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py).
Logging now captures the pre-call *value* (not a Token) and restores via a
plain set_trace_id()/set_session_id() call, which works regardless of which
Task/Context calls it. reset_trace_id/reset_session_id are removed as
dead/unreliable code. Added a regression test that spawns __init__ and the
restore in different asyncio Tasks - confirmed it fails against the prior
Token-based commit and passes here.
* fix(logging): restore correlation context in the originating task too
Greptile's re-review correctly identified a remaining gap: for a
successful acompletion(), async_success_handler is dispatched via
asyncio.create_task + the global logging worker into a *different* Task
than the one wrapper_async/Logging.__init__ ran in. The prior fix (43c164a)
only restored the handler's own (detached, throwaway) Task - it never
touched the originating request Task, which keeps this call's trace_id/
session_id set for the rest of its own execution (e.g. nested calls made
via the same Task).
wrapper()/wrapper_async() in litellm/utils.py now restore the originating
Task's correlation context in a finally block once the whole call is done,
regardless of what detached logging tasks it spawned along the way. Since
the wrapped body rebinds its own `kwargs` local via function_setup(),
sharing the dict object doesn't work here; a small mutable holder carries
the constructed Logging instance back out to the outer wrapper instead.
_restore_correlation_context() is no longer guarded against repeat calls:
with value-based (not Token-based) restoration, each distinct Task that
calls it needs its own restore to take effect in that Task's own view of
the contextvars, so multiple calls (once per Task involved in an attempt)
are required, not just tolerated.
Added a regression test using mock_response to exercise the real success
dispatch path (asyncio.create_task + GLOBAL_LOGGING_WORKER) without a live
provider call, asserting the *test's own* (originating) task context is
restored after the call - this is exactly the case Greptile flagged and
the prior commit didn't cover.
* fix(logging): restore correlation context when function_setup itself fails
Greptile's 4th finding: if function_setup() constructs Logging() (whose
__init__ already mutates trace_id_var/session_id_var) and then raises
before returning - e.g. update_environment_variables() throws - the
caller's wrapper()/wrapper_async() never receives a logging_obj reference,
so its own restore-on-finally never fires. The correlation ids leak into
every subsequent log line on that thread/task with no way to clear them.
function_setup()'s own except block now restores the context itself in
that case, using whatever logging_obj it managed to construct before
failing (locals().get(), safe against the earlier failure modes where
logging_obj was never assigned at all).
Added a regression test that monkeypatches Logging.update_environment_variables
to raise after construction, confirmed it fails without this fix (the
leaked ids show up directly in the raised exception's own log line) and
passes with it. Broader sweep (test_utils.py, test_router.py,
test_main_module_header.py, streaming handler tests, plus all
logging-specific tests): 722 passed.
* fix(logging): don't assume every litellm_logging_obj is a real Logging instance
CI caught a real regression from the last commit: tests/test_litellm/llms/xai/test_xai_key_fallback.py
injects a minimal FakeLogging stand-in (only implementing
update_from_kwargs) as litellm_logging_obj for a narrow realtime-config
unit test, bypassing the real Logging class entirely. wrapper()/
wrapper_async()'s finally block and function_setup()'s except block both
unconditionally called _restore_correlation_context() on whatever ended up
in the holder, which doesn't exist on that stand-in.
_restore_correlation_context is new plumbing specific to this PR's
feature, not part of any pre-existing stand-in's expected interface, so
callers of it can't assume every object playing the litellm_logging_obj
role implements it. Added _restore_correlation_context_if_supported(),
a small getattr-guarded helper, and used it at all three call sites.
* fix(logging): don't restore context too early on setup failure or streaming
Two more findings from Greptile's 5th review round.
1. function_setup()'s except block restored correlation context *after*
logging the "Error in function_setup" exception, so that diagnostic log
line itself was stamped with the doomed call's ids instead of the outer
ids - misleading, since the failed call never produces anything else to
attribute those ids to. Restore now happens before the log call.
2. wrapper()/wrapper_async() restored the originating task's context as
soon as a streaming call returned, before the caller ever starts
iterating the CustomStreamWrapper it just got back. Any log lines
emitted while iterating (in the same thread/task) incorrectly showed
the pre-call ids instead of this call's own ones. The wrapper finally
block now skips the restore when the return value is a stream wrapper,
deferring to the terminal handler that already fires once the stream is
actually assembled/exhausted.
Both verified with tests that fail against the prior commit and pass
against this one. Broader sweep unchanged at 829 passing.
* fix(logging): best-effort correlation cleanup on abandoned streams
Greptile's 7th finding: if a caller returns a streaming response and never
fully consumes it - stops iterating early, drops the reference, cancels
it - the terminal handler that normally restores the originating task's
trace_id/session_id never fires, since it only runs once the stream is
actually assembled/exhausted. The ids leak into every subsequent log line
in that thread/task with no bound.
There's no reliable Python hook for "this was abandoned without being
closed" - CustomStreamWrapper has no close()/__aexit__/context-manager
convention today, and the only automatic option is __del__, whose timing
is inherently unpredictable (delayed by cyclic GC, not guaranteed at
interpreter shutdown, can run on a different thread). This is a best-effort
safety net, not a guarantee, and is documented as such in the docstring.
Testing this via real garbage collection proved unreliable in practice:
per-chunk logging submits work to a thread pool executor whose worker
thread transiently holds its own bound-method reference to the wrapper
until that task completes, so refcount doesn't hit zero on a
deterministic schedule even with polling. Tests call __del__ directly
instead - a plain method, safe to invoke early - which exercises exactly
the restore logic real garbage collection would eventually trigger,
plus a case confirming a broken logging_obj can never make __del__ raise.
* fix(logging): restore consumer's context at every real stream exit point
Two more findings from this round.
Veria AI: even a *fully consumed* stream never restored the actual
consuming thread/task's correlation context. The terminal success dispatch
(dispatch_success_handlers via asyncio.create_task for async, or
success_handler via the shared executor for sync) only restores whatever
detached context it runs in - never the caller's own thread/task that's
running the for/async for loop. Same root cause as the wrapper-level fix
two rounds ago, just missed for the streaming-completion path.
Greptile: explicit aclose() (client disconnect, router fallback aborting
a partial stream) closed the underlying stream without restoring
correlation context either, since request wrappers intentionally skip
restoration for returned streams and no terminal handler runs on this
path.
Added CustomStreamWrapper._restore_consumer_correlation_context(), called
from every point control genuinely returns to the consumer: the final
raise StopIteration/StopAsyncIteration on natural exhaustion (both sync
branches, both async branches), _handle_stream_fallback_error (the shared
choke point for all three failure-raising call sites), and aclose(). __del__
now delegates to the same helper instead of duplicating it.
Verified with tests extending the existing streaming-exhaustion cases to
assert the consuming context is restored after the loop completes (fails
against the prior commit, passes now), plus a dedicated aclose() test.
Broader sweep: 832 passing.
* fix(logging): don't let a delayed __del__ finalizer clobber a newer active call
If an abandoned stream's __del__ fires late (after cyclic GC delay), a
different call may have already taken over the correlation contextvars in
the same Task/thread. Restoring unconditionally would stomp that active
call's trace_id/session_id with the abandoned stream's stale pre-call
snapshot. __del__ now only restores when the contextvars still hold the
ids this call itself set.
* fix(logging): compare sanitized ids in the __del__ ownership guard
set_trace_id()/set_session_id() sanitize (strip control chars, bound length)
before storing, so the contextvar's value can differ from the raw
litellm_trace_id/litellm_session_id. The __del__ ownership guard was
comparing against the raw values, so a caller-supplied id containing control
characters or exceeding 256 chars would never match, permanently skipping
cleanup. Capture what set_trace_id()/set_session_id() actually stored and
compare against that instead.
* fix(logging): restore consumer context on the synthesized finish_reason chunk
Both __next__ and _finalize_completed_stream() have a branch that fires when
the underlying stream ends without ever emitting an explicit finish_reason
chunk: they synthesize one via finish_reason_handler() and return it. A
consumer that stops as soon as it sees finish_reason - a common pattern -
never calls __next__()/__anext__() again, so the existing restore in the
sent_last_chunk-is-True StopIteration branch never runs for them. The
underlying stream is already exhausted at this point regardless of whether
the caller keeps iterating, so restoring here is safe.
* fix(logging): don't restore correlation context before the caller receives the final chunk
The previous fix (5147c69186) restored context immediately before returning
the synthesized finish_reason chunk from __next__/_finalize_completed_stream,
reasoning that completion_stream was already exhausted. But that chunk is
still this call's own data, and the caller's own application-level log
statements processing it run in the same synchronous frame right after the
return - restoring first made those lines carry the wrong (outer) ids,
exactly what wrapper()/wrapper_async() deliberately avoid by not restoring
while a stream is being iterated.
Revert to not restoring there. A caller that keeps iterating still gets a
correct, deterministic restore on its very next __next__()/__anext__() call
(completion_stream is exhausted, so that immediately re-raises
StopIteration/StopAsyncIteration through the already-restoring branch). A
caller that stops right after finish_reason relies on aclose() or the
best-effort __del__ guard, same as any other stream the caller doesn't fully
exhaust.
* refactor(logging): hoist a safely-hoistable function-body import to module top
CorrelationContextFilter.filter()'s `import litellm` was a function-body
import; verified it can move to module top without a circular-import failure
(litellm/__init__.py already imports from litellm._logging before setting
request_correlation_in_logs, but a bare `import litellm` only binds the
already-in-sys.modules module object - the attribute itself isn't read until
filter() actually runs, by which point litellm is fully initialized).
* test(logging): move correlation tests into their conventionally-mapped files
tests/test_litellm/ mirrors litellm/ in a parallel path. Correlation tests
for the Logging class (litellm_logging.py), function_setup/wrapper_async
(utils.py), and CustomStreamWrapper (streaming_handler.py) had all landed in
test_logging.py, which only maps to litellm/_logging.py itself. Moving each
group to its correctly-mapped file: test_litellm_logging.py (Logging class
init/restore), test_utils.py (function_setup, wrapper_async), and
test_streaming_handler.py (CustomStreamWrapper) in the next commit.
test_logging.py keeps only what actually exercises _logging.py's own
contextvars/filters/formatters/sanitization. No behavior change - same
assertions, same coverage, just relocated.
* fix(logging): restore correlation context unconditionally in wrapper()'s sync path
Blocking finding from review: a caller-visible correlation feature was
silently misattributing one request's logs to a different, unrelated one on
the sync/threaded path. wrapper()/wrapper_async() both left trace_id/session_id
"open" across a stream's entire iteration so the caller's own log lines while
consuming it would carry the right ids. That's safe for wrapper_async(): each
async call gets its own asyncio Task with its own copy of the contextvars,
and Tasks are never recycled across requests, so a leftover value can only
ever affect that one already-abandoned Task.
It is not safe for wrapper() (sync): a plain OS thread has no such per-call
isolation, and a thread pool's worker threads *are* recycled across unrelated
requests. If a sync stream was abandoned (client disconnect, early break, an
uncaught exception) without ever being exhausted or closed, nothing restored
its contextvars, and a pool could later hand that same thread to a completely
different call, which would inherit the abandoned request's ids as its own
"pre-call" baseline and then restore back to that poison when it finished -
permanently misattributing every subsequent log line on that thread,
including its own, to the abandoned request. Strengthening the __del__
finalizer already added for this can't fix it: finalizer timing is exactly
what a permanently-reused thread can't rely on.
wrapper() now restores unconditionally in its own finally, before a sync
stream is ever handed back to the caller. The trade-off: a sync stream
consumer's own application-level log statements while iterating no longer
automatically carry this call's ids (litellm's own internal per-chunk
logging is unaffected, since it's dispatched separately). That's an
acceptable cost for eliminating a silent cross-request misattribution bug.
wrapper_async() keeps the existing conditional (skip-if-streaming) behavior,
justified by the Task-isolation argument above; CustomStreamWrapper's
__del__/aclose()/next-iteration restore machinery remains meaningful and
necessary there.
This also simplifies wrapper()/wrapper_async() back toward their original
shape: both previously used a mutable-dict-holder split into a separate
_body function to smuggle logging_obj/result out to an outer finally,
working around function_setup() rebinding its own local `kwargs`. That
restructuring is no longer needed - `logging_obj` (and, for wrapper_async(),
`result`) were already function-level locals in scope for a plain
try/finally; three of wrapper_async()'s retry-return statements now assign
through `result` first so it accurately reflects what's actually returned
even on a retry path.
Regression test: test_abandoned_sync_stream_does_not_contaminate_a_later_call_on_the_same_thread
in test_streaming_handler.py reproduces the exact reported scenario with a
real single-worker ThreadPoolExecutor - confirmed it fails with the prior
(skip-restore-on-stream) wrapper() and passes with this fix.
* refactor(logging): use Mapping instead of bare dict for read-only params
_get_standard_logging_payload_trace_id/_session_id only read litellm_params
(.get() calls, no mutation) - annotate it as Mapping[str, Any] rather than a
bare mutable dict, per the repo's no-mutable-collection-in-annotation rule.
* fix(logging): scope request_correlation_in_logs to the async/proxy path only
Blocking review finding: wrapper() (the sync entry point) used the same
skip-restore-on-stream design as wrapper_async(), but a plain OS thread has
no per-call context isolation the way an asyncio Task does, and a thread
pool's worker threads are recycled across unrelated requests - an abandoned
sync stream could leave its ids stuck on a thread a pool later hands to a
completely different request, misattributing that request's logs. A fix
existed and was tested (restore unconditionally in wrapper()'s own finally),
but it doesn't benefit this feature's primary consumer - the proxy only ever
calls the async entry point - and carries sync-specific complexity this PR
doesn't need.
Scope the feature to async only instead: Logging.__init__() takes a new
supports_correlation_logging parameter (default True), threaded down from a
new function_setup(..., is_async_call: bool = True) parameter. wrapper() is
the one caller that passes is_async_call=False; every other function_setup()
call site (wrapper_async(), the router, and proxy/MCP-internal call sites)
is already async and keeps the default. With
supports_correlation_logging=False, Logging.__init__() never calls
set_trace_id()/set_session_id() at all, so a sync call has nothing to leak
in the first place. wrapper() reverts to its pre-review shape with no
correlation-specific code at all.
StandardLoggingPayload's own trace_id/session_id fields are unaffected
either way - they're a deterministic per-call read of
self.litellm_trace_id/self.litellm_session_id, not ambient contextvar state,
so they were never exposed to the cross-request bug.
Full sync/direct-SDK support (stamping + its own safe-restore mechanism) is
deferred to a follow-up PR; the fix and its regression test already exist in
this branch's history at commit 9f3a20f4b2 and can be resurrected there.
Tests: replaced the two wrapper()-level tests with ones proving the new
invariant (sync calls, streaming and non-streaming, never touch
trace_id_var/session_id_var even when the caller explicitly passes
litellm_trace_id/litellm_session_id), and added a direct unit test for the
supports_correlation_logging=False gate on Logging.__init__ itself. Verified
live: a real proxy (Postgres-backed, real OpenAI calls) shows clean
trace_id/session_id isolation across two concurrent sessions with no
cross-contamination; a standalone script confirms real sync SDK calls
against a real model never touch the correlation contextvars.
* feat(logging): fall back to W3C traceparent/baggage for trace_id/session_id
request_correlation_in_logs previously only resolved trace_id/session_id from
litellm-specific sources: x-litellm-trace-id/x-litellm-session-id headers, a
generic x-<vendor>-session-id header, or Anthropic-style metadata.user_id. If
none were present, trace_id fell back to an auto-generated UUID unrelated to
anything else, and session_id stayed empty - even when the caller already had
real distributed-tracing instrumentation sending the actual industry-standard
headers for this.
Add a fallback to the W3C Trace Context traceparent header (trace-id
component) and W3C Baggage header (session.id entry), so a request already
carrying real OpenTelemetry trace context correlates litellm's own logs with
the same trace in the caller's observability backend (Datadog, Honeycomb,
Tempo, etc.) instead of getting an unrelated generated id. Precedence is
unchanged for existing sources: explicit litellm headers and the Anthropic
metadata path both still win over this new fallback, which only fires when
neither found anything. trace_id and session_id are resolved independently
here (unlike the existing chain_id mechanism, which uses one shared value for
both), since traceparent and baggage are semantically distinct W3C concepts.
New helpers _trace_id_from_traceparent/_session_id_from_baggage in
litellm_pre_call_utils.py parse the header formats directly (no new
dependency - both are simple fixed-width/delimited strings), wired into
LiteLLMProxyRequestSetup.add_litellm_metadata_from_request_headers() only
when the corresponding litellm_trace_id/litellm_session_id key isn't already
set by the existing paths.
Verified live against a real proxy: a bare traceparent header produces a log
trace_id exactly matching its trace-id component; a traceparent alongside an
explicit x-litellm-trace-id header (different value) produces a log showing
the explicit header's value, proving precedence.
* fix(logging): reserve trace_id/session_id in JsonFormatter against message-content spoofing
JsonFormatter merges keys parsed from the message body before applying extra
record attributes, and the extra-attributes loop skips a key that's already
present. A caller-controlled log message that happens to parse as JSON/dict
with a "trace_id"/"session_id" key (e.g. the proxy logging a raw request-header
dict) could therefore make the JSON record carry the attacker-supplied value
instead of the real correlation context set via CorrelationContextFilter.
trace_id/session_id are now applied from the LogRecord's own attributes after
message-content parsing, unconditionally overwriting anything the message body
claimed for those two keys.
* style(logging): fix import order (ruff I001) in _logging.py and litellm_logging.py
- _logging.py: import litellm belongs after the stdlib from-imports, grouped
with the other litellm.* imports, not before them.
- litellm_logging.py: the refactor to Mapping introduced a second, separate
`from collections.abc import Mapping` instead of merging it into the
existing `from collections.abc import Callable` import.
Caught by the strict-rule budget gate (ruff-strict-budget.json caps I001 at
0 new violations); both auto-fixed with `ruff check --fix --select I001`.
* style(logging): freeze mutable-collection constructions flagged by LIT002
Five sites in this PR's diff built a mutable list/dict literal instead of a
frozen value: a plain list of optional strings in CorrelationPlainFormatter,
a `kwargs or {}` fallback, a `metadata or {}` fallback, two `[...]` candidate
orderings, and a `dict(headers)` copy feeding a dict comprehension. Each is
build-once/read-only, so this rewrites them as tuples, MappingProxyType, or a
plain conditional `.get()` instead of seeding then reading a fresh mutable
collection - no behavior change, confirmed by the existing test suite.
Caught by the type-discipline budget gate (LIT002 capped at 0 new
violations).
* fix(logging): reserve trace_id/session_id even when no correlation context is active
Live-proxy verification surfaced a gap in the earlier message-content-spoofing
fix (7f390a57fc): that fix only overwrites trace_id/session_id from the
LogRecord's own attribute, so it does nothing for a log line emitted before
CorrelationContextFilter has stamped anything on this record (e.g. the
"Request Headers" debug line, which fires before Logging.__init__() runs for
the request). On such a record, a caller-supplied header literally named
trace_id/session_id still got promoted into the JSON output via the embedded
JSON/dict-repr parser, since there was no genuine value to protect.
Fixed at the source: trace_id/session_id are now excluded unconditionally from
the message-content-parsing promotion step, not just superseded afterward.
Verified live against a real proxy - the exact adversarial request (headers
literally named trace_id/session_id) no longer leaks into any JSON log record.
Added a regression test for this no-active-context variant specifically,
confirmed it fails against the prior commit and passes now.
Also fixes an unrelated basedpyright regression from an earlier rebase's
conflict resolution: litellm/utils.py's `logging_obj` was incorrectly
re-annotated `Final` at its second assignment in function_setup() (it's first
declared `None` a few lines earlier), which basedpyright correctly rejects.
* fix(proxy): stop logging the raw W3C baggage session_id value
_session_id_from_baggage() extracts the caller-controlled session.id entry
verbatim - it isn't sanitized until set_session_id() runs later in
Logging.__init__(). The debug log line for this extraction interpolated the
raw value directly, so a caller could embed terminal control characters or
ANSI escape sequences that forge/alter plaintext log output for anyone
tailing the proxy's logs.
Verified live: a baggage header with an embedded ANSI escape reached the
terminal as a real, unescaped control sequence before this fix. Drops the
value from the log line entirely (the extraction succeeding is enough signal
on its own) rather than sanitizing-then-logging, matching veria-ai's
suggestion. Added a regression test using caplog that fails against the prior
commit and passes now.
* fix(logging): restore consumer context only after stream-failure exception mapping
_map_anthropic_exception/_map_aleph_alpha_exception synchronously log a debug
diagnostic (the raw status code) as part of exception_type()'s mapping.
_handle_stream_fallback_error restored the consumer's outer correlation
context before calling exception_type(), so that diagnostic log line carried
the outer (or empty) trace_id/session_id instead of the failing stream's own -
flagged by Greptile.
Moved the restore to run after mapping completes, matching the same
restore-after-not-before pattern already applied elsewhere in this file for
success/finish_reason handling. Added a regression test that captures the
correlation context live during a mocked exception_type() call; fails against
the prior commit, passes now.
* fix(logging): restore consumer context only after aclose()'s stream close completes
aclose() restored the consumer's outer correlation context as its first
statement, before awaiting the underlying provider stream's own aclose()/
close(). If that close attempt raises, the except branch's debug diagnostic
ran under the already-restored outer context instead of the closing stream's
own trace_id/session_id - flagged by Greptile, same restore-too-early pattern
as the stream-failure fix in f1cf9589d6.
Moved the restore to the end of aclose(), after the close attempt (and its
diagnostic logging) completes. Added a regression test with a fake stream
whose aclose() raises, capturing the correlation context live during the
diagnostic log call; fails against the prior commit, passes now.
* style(logging): satisfy new strict-lint budgets introduced upstream (Final, ANN401, S110, TRY300, kwargs typing)
Rebasing onto litellm_internal_staging pulled in 116 upstream commits that
introduced/tightened several lint gates this PR's own code now trips:
- LIT010 (every local/module-level variable must be Final): added Final
annotations across _logging.py, litellm_logging.py, streaming_handler.py,
litellm_pre_call_utils.py, and utils.py. Where a name is genuinely
reassigned (logging_obj: starts None, later set to the real object) or
branch-assigned, either restructured into a single ternary expression
(ordered_candidates) or suppressed with `# rebind-ok: <reason>` matching
this repo's documented escape hatch.
- LIT011 (parameter mutation): suppressed the two new `data[key] = value`
writes in litellm_pre_call_utils.py with `# rebind-ok`, matching the
unsuppressed precedent already used for every other `data[...]` write in
the same function - `data` is an intentional out-param there.
- ANN001/ANN003/ANN202 (missing parameter/return type annotations): fully
typed success_handler/_success_handler_body, their async twins, and
failure_handler/_failure_handler_body/async variants in litellm_logging.py,
plus function_setup in utils.py (added Rules to its existing TYPE_CHECKING
block for the rules_obj: Rules annotation).
- ANN401 (explicit Any disallowed): suppressed with `# noqa: ANN401` on the
handful of genuinely-heterogeneous result/*args/**kwargs parameters, since
ordinary suppression is this repo's documented path.
- S110 (try/except/pass): added to the existing BLE001 noqa on the one
best-effort correlation-cleanup try/except this PR added.
- TRY300 (return inside try): moved two `return result` statements into
`else:` blocks in the retry-fallback paths this PR's own diff touched.
- reportPrivateUsage (basedpyright): renamed the two new
StandardLoggingPayloadSetup static methods (get_standard_logging_payload_
trace_id/session_id) to drop their leading underscore, since they're
genuinely called from a sibling module-level function in the same file.
No behavior change - confirmed by the full existing test suite (819 passed)
plus all four lint gates (ruff format, ruff-strict, type-discipline,
basedpyright) passing clean.
* fix(lint): stop RUF100 flagging noqa suppressions the strict gate needs
CI's plain "ruff check" job uses the default ruff.toml, a narrower config
than ruff-strict.toml (used only by the strict-rule budget gate). ANN401 and
S110 aren't enabled in the default config, so RUF100 (unused-noqa) flagged
the `# noqa: ANN401`/`# noqa: ...,S110` suppressions this PR added as pointless
under that config, even though they're genuinely needed under ruff-strict.toml.
- ANN401: added to ruff.toml's existing `lint.external` list (same mechanism
already used for C901/TID251, enforced by the strict gate but not by this
config) - these Any usages are genuinely dynamic/forwarded, so the
suppression itself is correct and just needed registering.
- S110: fixed the underlying code instead of registering another external
code - the try/except/pass in
CustomStreamWrapper._restore_consumer_correlation_context now logs at
debug level on failure (matching the existing best-effort-cleanup pattern
in _record_partial_usage_for_failure elsewhere in this file), which
satisfies S110's own suggestion directly and needs no suppression at all.
Verified against both ruff.toml and ruff-strict.toml directly, plus all
three other gates (ruff format, type-discipline, basedpyright) and the full
test suite (821 passed).
* fix(lint): scope the ANN401 exemption to file level instead of a repo-wide noqa
Ruff has no per-line-scoped way to register a noqa code across configs (that
requires the default ruff.toml's lint.external list, which is repo-wide in
scope even though the noqa itself is per-line). Since ruff does support
file-level exemptions via per-file-ignores, and ANN401 only needed exempting
in exactly two files, moved the exemption there instead:
- ruff-strict.toml: added [lint.per-file-ignores] disabling ANN401 for
litellm_logging.py and utils.py specifically, with a comment explaining
why (heterogeneous response/forwarded-args parameters with no fitting
concrete type - already verified by trying CostResponseTypes and hitting
a real basedpyright mismatch).
- ruff.toml: reverted the ANN401 entry from lint.external - no longer
needed, since there's no `# noqa: ANN401` left anywhere for RUF100 to
second-guess.
- Removed the now-redundant `# noqa: ANN401` from the 10 affected
parameters in both files, keeping the existing kwargs-ok reasons and
adding a short inline comment on the `result`/`*args` lines pointing at
the ruff-strict.toml exemption for context.
Verified against both configs directly (ANN401 clean under ruff-strict.toml
for these files, RUF100 clean under the default config), all four gates
(ruff format, ruff-strict, type-discipline, basedpyright), and the full
test suite (821 passed).
* fix(logging): redact credential-shaped trace_id/session_id before stamping log records
CorrelationContextFilter stamps trace_id/session_id onto a LogRecord after
SecretRedactionFilter has already run, so a caller-controlled value (e.g. via
x-litellm-trace-id or a W3C baggage header) that happens to look like a real
credential reached JSON and plaintext logs unredacted. Apply the same
credential redaction already used elsewhere in this module at
_sanitize_correlation_id(), the single choke point both set_trace_id() and
set_session_id() route through, so every caller-facing entry point is covered
without depending on filter ordering.
* fix(logging): restore correlation context when a stream's max-duration timeout fires
CustomStreamWrapper.__anext__() called _check_max_streaming_duration() before
entering its try block, so the litellm.Timeout it raises bypassed the except
Exception -> _handle_stream_fallback_error path entirely, leaking the timed-out
stream's own trace_id/session_id into whatever the consumer's task logs next.
Move the check inside the try so it flows through the same restoration path
every other stream failure already uses.
* test(streaming): make dispatch_failure_handlers mock awaitable for the async max-duration test
Moving _check_max_streaming_duration() inside __anext__()'s try block (prior
commit) means a max-duration Timeout now dispatches failure handlers through
the same path every other stream failure already uses, instead of bypassing
it entirely. dispatch_failure_handlers is async on the real Logging class;
the test's plain MagicMock logging_obj made asyncio.create_task() choke on a
non-coroutine return value once that path actually got exercised.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
yucheng-berri
|
61218f5f9f
|
feat(ptu): daily rollup writes per-model PTU flat cost by active hour (#35343)
Add the daily rollup that reads PTU config off model deployments and writes flat cost to LiteLLM_DailyTeamSpend. For each UTC day a deployment carrying ptu_count and cost_per_ptu_per_hour accrues ptu_count * cost_per_ptu_per_hour * active_hours, where active_hours is the overlap between the day and the optional [ptu_effective_from, ptu_effective_to) window clamped to 24; a window opening at 23:00 charges one hour that day. Rows use a sentinel api_key so they stay distinguishable from per-request rows and share the existing unique constraint, and the write is idempotent so re-runs never double count. The cron is registered at proxy startup and runs at 00:15 UTC. Pricing one day per fire leaves two ways for a day to end up unpriced and stay that way: a window backdated at configuration time, which no fire ever revisits, and a fire that is missed or lands late, which the next one does not replay because the billed day comes from the wall clock rather than the scheduled time. Both are silent, since the failure alert only fires for a charge that was attempted. Each scheduled run therefore follows the day's reconcile with a catch-up pass that prices the (team, model, date) charges inside every declared window that carry no row yet, bounded at the earliest ptu_effective_from and floored at PTU_ROLLUP_MAX_BACKFILL_DAYS. It writes only what is missing: a day already priced keeps the amount it was billed whatever the config says now, and it runs no prune, so deciding a row is stale stays the single-day path's job. Zero-cost days write nothing, which leaves an out-of-window day reconsidered each run rather than recorded as done. A catch-up pass that fails cannot take the day's own result with it, and an explicit target_date still means reconcile exactly that day. The sentinel row keys on the deployment id, with the operator-facing name alongside it in model_group, which sits outside the table's unique key. The name is what a usage view displays, but a deployment can be renamed, and two runs holding config views from either side of a rename then wrote the same day under two different keys, so nothing collided and both charges survived. A multi-pod rig reproduced that as a permanent double charge that no later run repaired. Keyed on the id both writes land on one key and the upsert collapses them; when the rate changed too, last writer wins on the amount rather than adding a row. Deployments sharing a public name inside a team therefore no longer need collapsing into a single charge: each keys its own row, and the read path merges them back under the shared name. The read path that surfaces the amount lands in a follow-up PR. The prune is the one destructive step, so it only runs when the pod took the cross-pod lock. The upserts stay unguarded, since they are idempotent and no lock problem may cost a day, but the delete compares a cutoff and an updated_at stamped on different hosts, and a live rig showed a pod whose clock ran ten minutes ahead sweeping the charge a concurrent pod had just written, leaving the day at zero. Its cutoff also allows PTU_PRUNE_SKEW_GRACE_SECONDS of slack, which separates the two populations without requiring clocks to agree: a stale row is hours old and a concurrently written one is seconds old. The catch-up deletes nothing. Removing a deployment or narrowing its window stops it accruing new charges and leaves the days it was already billed for standing, since those days were incurred and a usage view has to keep reporting them. A deployment carrying no ptu_effective_from is skipped rather than treated as open ended. The endpoints require a start, and substituting the cap floor for a missing one meant a windowless deployment accrued the whole ninety day window on its first run, billing days it did not exist while the result still reported a single row written. |
||
|
yucheng-berri
|
e5386c10a7
|
feat(ptu): configure provisioned-throughput flat cost on a model deployment (#35341)
Add ptu_count, cost_per_ptu_per_hour, ptu_effective_from and ptu_effective_to to
ModelInfo so a model deployment can carry the inputs for provisioned-throughput
flat-cost attribution. ModelInfo validates per-field bounds (positive count,
non-negative rate, effective_to after effective_from); model/new and
model/{id}/update enforce the cross-field invariant (count and rate set together,
team_id required) on the effective model_info so partial updates validate the
merged result, and v1/model/info returns the fields.
LiteLLM_DailyTeamSpend gains ptu_flat_cost and ptu_source_model_id columns plus a
sentinel api_key constant; the daily rollup that writes them lands in a follow-up
PR. Adding the optional model_info fields is backward compatible; models without
them are unaffected.
ptu_effective_from is required alongside the count and rate rather than optional. Flat
cost accrues from that instant, so an absent start has to be inferred, and inferring it
let a deployment configured today be billed for days it did not exist. Both PTU validators
also run over the merged view before any write on the update path, beside the premium check the create path
already runs there: the team ACL update below autocommits, so a validator raising further
down left the team mutated and the deployment row never written.
The update path validates the model_info a patch would store rather than the patch
alone. An invariant holds over the deployment as it will exist, not over whichever
subset of fields a caller sent, and validating the patch rejected raising the rate on
an already configured model because that patch carries no start of its own.
|
||
|
Mateo Wang
|
f6b9518ddb
|
Merge pull request #36092 from BerriAI/devin_ai_fix_openai_passthrough_files_route_36086
Some checks are pending
Unit Tests: LLM Provider Transformations / All Other Providers (push) Waiting to run
Unit Tests: MCP, Secrets, Containers & Misc / misc (push) Waiting to run
Unit Tests: Proxy Auth & Key Management / proxy-auth (push) Waiting to run
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / assert-shard-coverage (push) Waiting to run
Unit Tests: Proxy DB Operations / auth-checks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / budgets (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / custom-logging (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / db-and-spend (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / endpoints-and-responses (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / guardrails-hooks (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests: Proxy API Endpoints / proxy-endpoints (push) Waiting to run
Unit Tests: Proxy API Endpoints / proxy-server (push) Waiting to run
Unit Tests: Proxy Infrastructure / proxy-infra (push) Waiting to run
Unit Tests: Proxy Legacy Tests / auth-and-jwt (push) Waiting to run
Unit Tests: Proxy Legacy Tests / key-generation (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-config (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-response-and-misc (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-server-extras (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-token-counter (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-user-auth-and-spend (push) Waiting to run
Unit Tests: Proxy Legacy Tests / proxy-utils (push) Waiting to run
Unit Tests: Responses, Caching & Types / responses-caching-types (push) Waiting to run
GitHub Actions Security Analysis / zizmor (push) Waiting to run
fix(proxy): stop /{provider}/v1/files from capturing /openai_passthrough
|
||
|
mateo-berri
|
bd719c21dc | test(passthrough): annotate route match scope as Final | ||
|
Yuneng Jiang
|
00600c1af7
|
test(proxy): guard management_v1 against fastapi names removed in supported releases | ||
|
yuneng-jiang
|
ecba48dd7c
|
Merge pull request #35773 from HuanQian571/litellm_fix_management_v1_get_flat_params
fix(proxy): restore management_v1 query-param validation under fastapi>=0.140.7 |
||
|
mateo-berri
|
85c1b5d04a | Merge remote-tracking branch 'origin/litellm_internal_staging' into devin_ai_fix_openai_passthrough_files_route_36086 | ||
|
Mateo Wang
|
6eaeab8eae
|
Merge pull request #36273 from BerriAI/litellm_dbless_hook_registration
fix(proxy): skip prisma-dependent hooks when no database is attached |
||
|
yucheng-berri
|
efc4e6f28c
|
fix(batches): keep batch state in sync on a poll without claiming attribution (#34456)
A poll of a Vertex passthrough batch wrote nothing to the managed-object row, so status and file_object stayed frozen at the create-time snapshot and GET /v1/batches served a stale status and an empty output file id for the life of the batch. Only the create may claim a batch, but every observation of one may refresh its state. store_unified_object_id takes create_if_missing, which the poll clears: it refreshes status and file_object through update_many, and leaves a row that is absent absent rather than creating one owned by the observer, since created_by and team_id are written by whoever reaches the create branch. The update payload is now shared with the upsert so it cannot drift into writing api_key, request_tags, created_by or team_id. The passthrough identity re-assertion that was previously part of this PR ships separately in #36121, so this PR keeps only the batch attribution work. The creating key owns user_api_key_alias only when it actually has one. Guarding the overwrite on the presence of a key rather than on a resolved alias nulled the field out for every key generated without key_alias, and for any key rotated or deleted before its batch finished, losing the creating user's alias that the spend row previously carried. The guard now matches the team-alias line below it. |
||
|
tin-berri
|
e35ee4e5fa
|
feat(router): independent, default-on deployment affinity for the auto-router (#36146) | ||
|
devin-ai-integration[bot]
|
12aeb53aec
|
fix(otel): mark v2 server spans as failed for pre-call errors (#34546)
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): cover malformed-body rejection when auth error is recovered Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(auth): skip authorization for a request whose body never parsed Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
8c0556abf6 | fix(proxy): authenticate managed ids before routing | ||
|
mateo-berri
|
855c49d0ef | fix(proxy): skip prisma-dependent hooks when no database is attached | ||
|
mateo-berri
|
5f7a663005 |
fix(proxy): enforce require_managed_files on every raw provider id route
require_managed_files was only checked on upload, so raw provider ids still reached the batch, fine-tuning and vector store file routes. Ownership rows exist only for managed ids, so those requests were forwarded under shared credentials with no tenant check: knowing another tenant's id was enough to read, run against, cancel or delete their object. Generalise the file-id guard to validate_managed_id_requirement(resource_id, resource_kind) and call it on batch create/retrieve/cancel, fine-tuning create/retrieve/cancel (training_file and validation_file both) and the shared vector store file id resolver. Behaviour is unchanged when the setting is off. |
||
|
mateo-berri
|
10209a8f91 | Merge branch 'litellm_internal_staging' into devin_ai_require_managed_files_read_paths_35530 | ||
|
Mateo Wang
|
f32d89f828
|
Merge pull request #36244 from BerriAI/litellm_fix_stale_team_budget_assertion | ||
|
yucheng-berri
|
d4dc2c39e7
|
fix(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing (#36119)
* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for
|
||
|
mateo
|
0791dd941b |
test(proxy): assert the copy _add_team_member_budget_table returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
1bafdb3c93
|
Merge pull request #36049 from BerriAI/litellm_list_batches_resolves_unified_ids
fix(managed_files): return unified output file ids from GET /batches |
||
|
ryan-crabbe-berri
|
2a9aac7004
|
fix(ui): let access groups be a team's only model source, with hover provenance (#36234)
* feat(proxy): return per-group model provenance on /team/info /team/info now carries access_group_details, one entry per resolved access group with its id, name, and model list, so the UI can attribute each inherited model to the group granting it. The batch resolver returns the access group rows keyed by id instead of a stringly dict of lists, and the team member budget helper returns a copy instead of mutating its parameter. Type discipline and basedpyright budgets ratchet down accordingly. * feat(ui): allow group-only teams and show model provenance on hover Team create and edit no longer require a model selection: an empty selection is saved as the no-default-models sentinel, never as a bare empty list, since an empty team model list means unrestricted access. The team info Models card now renders every badge with a hover tooltip naming how the team got that model: directly, via named access groups, or both, and group-granted badges stay visible when the direct list is empty or a sentinel. * refactor(proxy): dedupe access group ids and return copies instead of mutating Duplicate access_group_ids no longer amplify the /team/info response: ids collapse order-preserving before provenance is built, pinned by a regression test. The resolver returns a model_copy rather than mutating its parameter, and the team create call sends a new object instead of reassigning formValues.models. Budgets ratchet down further with the mutation removal. |
||
|
devin-ai-integration[bot]
|
1a45bf9afe
|
fix(proxy): resolve entity access groups in the model listing endpoints (#36230)
* fix(proxy): resolve entity access groups in the model listing endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse the fetched team object when listing models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover key-level access group resolution in model listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
3238ce8406
|
feat(auto-router): track turns per complexity tier (LIT-5302) (#36209)
* feat(auto-router): track turns per complexity tier (LIT-5302)
Stamps complexity tier at decision time (rollup never re-derives from routed
model, since tier->model mapping is mutable config). Records per-tier turn
counts in LiteLLM_AutoRouterSession.tier_turns (jsonb), rolls up per router
in benchmarks SQL via jsonb_object_agg, returns on AutoRouterBenchmarkGroup
for dashboard turns/share metrics.
Addresses Greptile/Bugbot findings:
- Missing _SessionAggRow.tier_turns field: added with field_validator to
parse jsonb text cast and handle NULL. Would 500 every benchmarks read.
- Missing ::text cast on tier parameter: Postgres fails type inference on
parameterized CASE/IS NULL without explicit cast. Added to all usages.
- Docstring false claim (only complexity routers produce tiers): quality
router stamps numeric tier '1'/'2'/'3'. Per-type grouping in SQL prevents
cross-contamination. Rewrote docstring to clarify isolation.
- Comment convention violations: stripped per CLAUDE.md rule.
- Test gaps: 8 unit tests for extraction/validation/aggregation, 7 behavior
tests for SQL semantics against real Postgres. 12 mutations killed.
Fixed fragile complexity_router test that broke on nested function calls.
No API change; extends existing GET /auto_router/benchmarks response only.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): address review findings on tier turns tracking
- Guard router_type update so a mid-session reconfigure can't pool
foreign tier names into tier_turns
- Keep pinned turns attributed to the tier that actually serves them
- Drop stray -- AlterTable comment from hand-written migration
- Drop the now-unnecessary ::text/json.loads round-trip; prisma
already returns tier_turns as a parsed dict
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(auto-router): satisfy type-discipline lint gate
- tier_turns fields: dict[str, int] -> Mapping[str, int] (LIT001,
mutable collection in annotation); these are read-only after
construction
- _summed_agg_row: {} -> MappingProxyType({}) (LIT002, mutable dict
literal)
- default-fallback branch: replace the reassigned-without-Final
fallback_tier with a Final default_model_first flag and a single
ternary assignment (LIT010)
Verified locally: type_discipline_gate.py, ruff_strict_gate.py, and
type_check_gate.py all pass against the litellm_internal_staging
merge-base; full test_complexity_router.py (374), auto_router
management-endpoint tests (26), db-layer rollup tests (31), and the
live-Postgres proxy_behavior rollup suite (17) all pass.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
mateo
|
860e37597f |
fix(a2a): align agent list annotation and test with the tuple return type
PR #36020 changed AgentRegistry.get_agent_list to return tuple[AgentResponse, ...], and PR #35163 added a test asserting the result equals []. Both were green on their own branches and only collided once they were both on litellm_internal_staging, so proxy-server has been failing on every PR since with 'assert () == []'. Nothing user-facing was wrong: get_agents only iterates the result and rebuilds it with comprehensions, and FastAPI serializes a tuple to the same JSON array. The test expectation was simply stale, so it now compares against (). The get_agents local was still annotated list[AgentResponse] while two branches assign the registry tuple straight into it, so it widens to Sequence[AgentResponse]. That covers both the tuple and the list branches without pretending the value is mutable. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
78addb230b
|
fix(proxy): deny agent access when key and team grants resolve to nothing (#36221)
* fix(proxy): deny when agent grants resolve to nothing `get_allowed_agents` returned a plain list where the empty value meant both "this caller was never restricted" and "this caller's grants resolved to nothing". Downstream read either as allow-all, so a key restricted to one agent inside a team restricted to another reached every agent on the proxy, and an access group that resolved to no agents did the same. Replace it with `resolve_agent_access`, returning a tagged UnrestrictedAgentAccess | RestrictedAgentAccess. Only a caller with no grant anywhere is unrestricted; an empty restricted set denies. Access group lookup failures now propagate to the key/team resolvers so a DB error still fails open exactly as before, while a group that genuinely resolves to nothing denies. * style(proxy): drop redundant comments from the agent access match |
||
|
ryan-crabbe-berri
|
eb3c8c168f
|
fix(proxy): derive config agent ids from agent_name so grants survive secret rotation (#36020)
* fix(proxy): derive config agent ids from agent_name so grants survive secret rotation Config-defined A2A agents were identified by a sha256 of the whole resolved config entry, secrets included, so rotating an os.environ secret re-minted the agent_id on restart and orphaned every object_permission.agents grant while grant-less keys kept access (LIT-5144). The id now hashes only agent_name, and the old full-entry hash is kept as a legacy alias: permission checks, GET /v1/agents filtering, spend and key attachment, and public_agent_groups all normalize legacy ids so pre-upgrade grants keep working * fix(proxy): persist stable agent ids into stored grants at startup The runtime alias only translates a legacy grant while the current config still hashes to it, so a secret rotation after upgrading would orphan the grant, and an orphaned grant intersecting a stable team grant collapses to an empty list that downstream reads as allow-all. Rewriting the stored ids once at boot removes both. This cannot be a SQL migration because only the running proxy can recompute the legacy hash from resolved config secrets * fix(proxy): make the grant id migration a compare-and-swap A grant edited between the migration's read and write kept the stale snapshot. The update now predicates on the agents array read at scan time via update_many, so a concurrently modified row is skipped and the runtime alias covers it until the next boot retries * fix(proxy): retry the grant id migration and stay within the LIT002 ceiling The one-shot startup task now retries up to three times with a short delay so a transient DB error at boot cannot leave a legacy grant unmigrated until an operator's next restart is the rotation itself. The new list constructions in the migration and the alias-expanded agent id lookups are tuples now, keeping the branch under the mutable-collection budget * fix(proxy): count compare-and-swap misses in the grant id migration migrate_legacy_grant_ids now returns rewritten and missed counts from the update_many results instead of reporting scanned rows as migrated, and the startup task retries while any rows remain unmigrated, not just on errors * fix(lint): clear basedpyright budget breaches in agent id aliasing |