The previous fix (refund through the raising index inclusive) was
itself a regression: these are shared, chain-wide buckets with no
per-request ownership tracking, so decrementing a key whose own
increment might or might not have committed is just as likely to erase
a different, legitimately-admitted concurrent request's charge on the
same key as it is to undo our own. An attacker could repeatedly cancel
requests to deliberately erase other callers' charges and exceed the
configured limit -- worse than the alternative this reverts to: a
committed-but-unrefunded key self-heals via its own TTL. Only strictly
earlier admissions in the same batch (this request's own
confirmed-successful increments, never ambiguous) are refunded now.
The earlier exception-refund fix only rolled back indices before the
one that raised, on the assumption a raise meant nothing committed for
that key. That's not guaranteed: Redis can commit the INCRBY and still
have the call raise if the response back to us is lost (a timeout, a
dropped connection), which the caller can't tell apart from a call that
never reached Redis. Now refunds through the raising index inclusive;
a clean rejection (no exception) still only refunds the earlier ones,
since TAG_RL_CHECK_AND_INCR_SCRIPT guarantees that path never committed.
Same class of issue as the earlier LIT001 fix: ruff format wrapped a
setdefault(...)-then-append pattern across lines, separating the
mutable-list construction from its justification comment. Shortened
variable names so both fit on one line stably under both ruff format
and the gate.
tag_value has no length or content bound before this hook embeds it
directly into an in-memory dict key (bypassing max_in_memory_cache_size,
which caps item count, not key bytes) and an uncapped Redis key. Hashing
to a fixed-length digest bounds this hook's own contribution to key size
regardless of the caller's input, while preserving distinctness.
Existing tests that hand-wrote the raw tag value into an expected key
string now build it through the real key-construction helper instead of
hardcoding the (now-hashed) internal format.
_atomic_check_and_increment's refund loop only ran on a normal
rejection return, not when a later key's own admission raised (a
transient Redis error) or the coroutine was cancelled mid-call.
Everything admitted earlier in that batch stayed permanently charged --
for concurrency, a leaked reservation the caller never releases,
incorrectly throttling that tag for up to the 1-hour safety TTL. A
try/finally now refunds every earlier admission before the exception
propagates, covering both raised exceptions and cancellation uniformly.
The base branch's type-discipline ceiling tightened from unrelated
merged PRs, newly flagging an annotated dict declaration ruff format
had to wrap across lines. Added a _PartitionOperations type alias so
the declaration fits on one line the gate can associate its
justification comment with.
A deployment declaring the identical concurrency_limits entry twice
appended its own id twice, inflating len(declaring_ids) past
total_deployments. That made is_chain_wide false for an entry every
deployment actually agreed on, and a non-chain-wide concurrency entry
is silently dropped entirely rather than degraded -- disabling
enforcement instead of just scoping it.
self.keys: T = value annotations inside a method body are never
evaluated at runtime, so the "_PartitionKey" forward-reference quoting
was unneeded despite _PartitionKey being defined later in the file.
A prior `ruff format` run wrapped several annotated mutable-dict/list
declarations across multiple lines, moving their `# mutable-ok`
justification off the line the type-discipline gate checks. Shortened
the annotations (a new _DedupSignature type alias, trimmed comments) so
they fit on one line under both ruff format and the gate.
Releasing a reservation whose key had already expired made INCRBY
recreate it, and the floor-to-zero SET left that recreated key
permanently in Redis (SET clears any TTL). Verified against a real
Redis instance: DEL removes the key outright instead, which reads back
identically to 0 everywhere this key is read.
A key_ttl_seconds override below period_seconds expired the bucket key
before its window rolled over, resetting the counter to zero mid-window
and letting tagged traffic exceed the configured limit. Only
period_seconds and above is now accepted.
Confirms _partition_key treats scope_by_key_hash as distinguishing (two
entries identical otherwise but differing only on this flag must not
share a partition), and that per-key bucket independence still holds
when key_ttl_seconds and max_in_memory_cache_size are also set, going
through the real _build_limits_index path rather than a hand-built
_ConfiguredLimit.
Adds TagRateLimitEntry.max_in_memory_cache_size so a single
high-cardinality entry can get its own dedicated in-memory cache
partition instead of sharing the hook's single default one, keeping the
knob alongside key_ttl_seconds and the rest of that entry's config
rather than only as a proxy-wide setting. Partitions are keyed by each
entry's full signature, not the override value alone, so two unrelated
entries that happen to pick the same size don't get merged.
tokens/dollars accounting and concurrency-slot release are both
partition-aware too: each partition owns its own v3 handler, and a
concurrency reservation is released against the exact partition it was
incremented on so it can't leak onto the default partition instead.
Also fixes a bug this surfaced: _configured_limit_for_signature
rebuilt each entry from a 5-field dedup signature that excluded
key_ttl_seconds and max_in_memory_cache_size, silently resetting both
to their defaults for every entry reachable through the real indexing
path used by async_filter_deployments and async_log_success_event.
An unresolved os.environ/ substitution or a config typo could set
tag_rate_limiter_max_in_memory_cache_size to a negative number or a
string; InMemoryCache raises comparing its size against that value, and
DualCache.async_set_cache swallows the exception, silently disabling
every counter write for this hook without Redis. Only positive integers
are now accepted; anything else falls back to the safe default with a
warning.
Also adds TagRateLimitEntry.key_ttl_seconds so a high-cardinality tag_id
can shed its Redis (or in-memory fallback) keys sooner without
shortening period_seconds itself. Concurrency's safety-floor TTL is
still never lowered by this override.
The isolated in-memory cache added for this hook still defaults to 200
entries. A deployment rate-limiting on a high-cardinality tag_id without
Redis can churn past that cap, evicting an active counter before its
period elapses. litellm_settings.tag_rate_limiter_max_in_memory_cache_size
lets that ceiling be raised; 0 is rejected in favor of the safe default
since it would disable the in-memory cache outright.
internal_usage_cache is the same DualCache instance the proxy's
key/team parallel-request limiter uses for its own authentication-
bound counters, and its default InMemoryCache evicts at 200 items.
Without isolation, a caller flooding this hook's own caller-controlled
tag buckets past that ceiling could evict an unrelated, authentication-
bound counter and let some other caller exceed a limit nothing here
configured. Give this hook a dedicated in-memory layer while still
sharing the real Redis connection when one is configured, so cross-
instance correctness is unaffected.
Router deliberately keeps a callable routing-group name (and, per its
own design, a direct deployment-id address) distinct from every member
deployment's own model_name (Router._get_routing_group_deployments).
Since _LimitsIndex only keys by model_name and team alias, a
group-addressed call previously matched neither table and the limiter
silently no-opped for both admission (async_filter_deployments) and
success accounting (async_log_success_event), even though the member
deployments carried real tag_rate_limits under their own model_name.
Add _LimitsIndex.resolve_any(), which falls back to resolving via each
candidate deployment's own model_name when the caller-visible name
matches neither table, stamping each result with the model_name it
came from (_ConfiguredLimit.resolved_group) so hashing stays
namespaced per underlying model_name -- otherwise two different
model_names sharing one routing group with an identically-named,
identically-configured limit would collide onto one Redis counter.
Direct deployment-id addressing has a related but separate, more severe
gap: Router.async_get_healthy_deployments returns a single dict (not a
list) for that path and short-circuits before Router.async_callback_
filter_deployments is ever called, so every CustomLogger.async_filter_
deployments-based hook is skipped, not just this one. That is a
Router-level structural issue affecting many hooks and is out of scope
for this PR.
async_log_success_event's kwargs is Logging.model_call_details, not the
router's flat request kwargs admission sees: metadata/litellm_metadata
are never top-level there, only nested under kwargs["litellm_params"].
Resolving the field name against kwargs itself always defaulted to
"metadata", so on LITELLM_METADATA_ROUTES (/v1/messages, /responses,
batches, bedrock, files) this read the caller's native, tag-less
metadata instead of the real, server-computed litellm_metadata.tags
admission already used -- silently skipping token/dollar accounting
for every successful call on those routes. Resolve against
kwargs["litellm_params"] instead, mirroring what admission already
does one level up.
Rebasing onto a much-advanced litellm_internal_staging tipped the
basedpyright reportPrivateUsage budget: importing _get_parent_otel_span_from_kwargs,
_PROXY_MaxParallelRequestsHandler_v3, _get_tags_from_request_kwargs, and
_PROXY_TagRateLimiter across module boundaries is the same pattern the
sibling dynamic_rate_limiter_v3 hook already uses (its own imports
just predate this budget check), so suppress with a reason rather than
renaming widely-referenced symbols.
Pre-existing mutable list[TagRateLimitEntry] field newly tripped the
type-discipline LIT001 ratchet after the rebase lowered its ceiling.
No other code relies on list mutation here (the one reader already
wraps it in tuple()), so switch to tuple[TagRateLimitEntry, ...].
- _ConfiguredLimit now carries team_scope: two teams can publish the
identical team_public_model_name alias, and the limits index already
scopes lookup by (team_id, alias) correctly -- but the Redis bucket
key itself never included team_id, so identically-named,
identically-configured limits from two different teams collided on
the same counter. Fold team_scope into the hash tag for both
admission and concurrency keys.
- _extract_key_hash and _extract_team_id no longer OR across metadata
and litellm_metadata. litellm_pre_call_utils.py writes the real,
server-authenticated value into only whichever one field is
authoritative for a given route, leaving the other exactly as the
caller sent it -- so an OR-fallback let a caller-forged
metadata.user_api_key (on a route where litellm_metadata is
authoritative) win over the real hash and bypass every
scope_by_key_hash=True limit by sending a fresh forged value per
request. Both extractors now read only the field
get_metadata_variable_name_from_kwargs names as authoritative,
matching the pattern this file already uses correctly for tags.
- Rewrite tag_rate_limiter.py's dict/list usage to immutable equivalents
(Mapping/Sequence params, tuple/frozenset/MappingProxyType returns and
locals, Final everywhere) to clear the ruff-strict type-discipline
budget (LIT001/LIT002/LIT010), keeping the pending-concurrency-keys
holder mutable by design with a documented # mutable-ok.
- Rebuild _build_limits_index's grouping via a stable sort + groupby
instead of a setdefault accumulator; caught and fixed a real bug in
that rewrite where a per-unit loop re-consumed groupby's already-
exhausted sub-iterator, silently returning empty limits for every
group after the first unit.
- Fix a basedpyright reportIncompatibleMethodOverride: match
async_filter_deployments's signature to CustomLogger's base exactly
(list/dict, not Mapping/Sequence) since it's an override.
- Fix a second reportGeneralTypeIssues: two Final-annotated locals
named `key` in sibling branches of the same function tripped
"previously declared as Final" despite being on mutually exclusive
paths; renamed them apart.
- Re-fix an eager-built f-string log message (%-style args instead)
that a prior rewrite pass had inadvertently reintroduced.
- Add the missing __init__ return annotation (ANN204) and drop the
now-unused typing.Any import in favor of a concrete object fallback
for the optional-otel-Span type alias (TID251), both newly over the
ruff-strict budget.
- Regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts: the prior
comment-trimming pass left it out of sync with the trimmed
TagRateLimitEntry/TagRateLimits docstrings.
- TagRateLimitEntry.period_seconds must be a positive integer: a
configured 0 previously crashed bucket admission with a
ZeroDivisionError instead of failing config validation up front.
- Replace the pending-concurrency-keys ContextVar's immutable-tuple
rebind with a mutable holder shared by reference across every task
forked off the admitting context. asyncio.create_task only copies
which object a ContextVar is bound to, not that object's contents, so
a .set(()) performed inside a detached failure-logging task (e.g.
after a sibling pre-call-check/filter callback rejects an already-
admitted hop) was invisible to the parent task that goes on to a
fallback hop, leaving a stale key that got double-released once the
fallback's own completion event fired in the parent's context.
Release now pops an exact snapshot from the shared holder instead of
rebinding or blanket-clearing it, so a sibling hop's own
concurrently-appended reservation is never swept up either.
- Trim non-essential comments/docstrings added by this feature to
match repo convention, keeping only the ones documenting a genuinely
non-obvious invariant.
Port of the tag-based rate limiter from feature/tag-based-rate-limiting
(PR #36459), squashed to the final state of the 18 rate-limiting-specific
commits and rebased onto litellm_internal_staging.
Kimi K3 accepts exactly low, high and max, defaults to max, and always thinks.
The map could not say that: medium and high have no supports_*_reasoning_effort
flag because every other reasoning model takes them, so the ten kimi-k3 entries
carried supports_reasoning alone and resolved to unknown. The dashboard then fell
back to a capability-blind level list that deliberately omits max, which is why a
kimi-k3 tier cannot be set to max thinking today.
Add reasoning_effort_levels, an array key in the shape the map already uses for
supported_endpoints and supported_modalities. Where present it is read first and
wins whole; every other entry keeps answering through the per-level flags,
unchanged. It is deliberately a different name from the computed
ModelGroupInfo.supported_reasoning_efforts, which stays derived from a group's
deployments and is never seeded from one deployment's model_info.
The levels are per entry rather than per model, because the deployments differ:
Moonshot, Together, Fireworks and Azure Foundry all forward the level unchanged
and get the model's own low/high/max, while Perplexity documents a six-value
enum it maps down internally and gets that. The /v1/messages degradation chain
consults the same declaration, so the level the map advertises is the level that
path forwards.
An MCP server behind an API gateway needs two credentials on one request: the
gateway's own token on a private header, and a separate bearer on Authorization
for the server behind it. Every arm that minted or held a token hardcoded
Authorization, and the conflict rule then dropped the operator's static
Authorization to make room, so the second credential never arrived.
ApiKeyConfig already modelled this as header_name plus value_prefix behind a
header() method. Extend that carrier to the four minted-token configs, have each
resolver arm ask its config which header to use instead of naming one, and drop
only the header the resolved credential is about to occupy.
Operators set it per server via upstream_token_header, plumbed through
config.yaml, the credentials blob, the management API and the admin form, on the
M2M, token-exchange, authorization-code and ID-JAG arms. It is non-secret so it
stays plaintext and round-trips on admin reads. Unset keeps today's behaviour.
Moving a credential off Authorization means it stops inheriting what Authorization
gets for free, so the slot now carries those protections itself. httpx drops
Authorization when a redirect crosses origin and keeps every other header, so a
custom slot is dropped by the client on the same condition, mirroring httpx's own
scheme/host/port rule with an agreement test that fails if the two ever diverge.
The v1 path also mirrors the v2 conflict rule, so an injected header cannot shadow
the credential the gateway resolved for that slot.
Which header a credential occupies, and what counts as being that header, was
answered independently in nine places by four hand-rolled comparisons. same_header,
has_header and without_header in litellm/types/mcp.py are now the one owner, shared
by both MCP stacks, and the client derives its slot once instead of three times.
The header name reaches egress verbatim, so the RFC 7230 grammar lives in one
place and is checked where servers are built: a bad value fails the config load
and the management API returns 400, rather than raising while a spec is built
and emptying the aggregate tool list for every other server. A blank means unset,
matching what the endpoint already accepts.
The two vision tests pointed at a Wikipedia-hosted cat photo, so every run
depended on upload.wikimedia.org staying up and unthrottled. It throttled,
and the 429 surfaced as a bedrock APIConnectionError, which reads as a
gateway failure rather than what it was.
The image is now a fixture in the repo, passed as a data URL. That also puts
the two providers on the same bytes: litellm downloads the image itself for
bedrock, while openai is handed the link and fetches it from its own servers,
so the hosted URL quietly meant the two tests were not testing the same thing.
The image was generated for this repo rather than borrowed, so nothing here
carries a third-party license. Also drops a stale comment about openai prompt
caching that sat above the vision helper; no caching test uses it.
