`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
- scale batch output-token reservations by the row's n / best_of candidate count
- parse client-supplied output caps defensively instead of 500ing on unparseable values
- exclude project IO descriptors from the first should_rate_limit pass when TPM
reservation is disabled so their buckets are not double-charged
Image, file, video, and previous_response_id requests reserved the whole
project ITPM limit up front, so any window with existing usage rejected
them and one in-flight multimodal request blocked the entire project.
Reserve the token_counter estimate instead, like every other request;
post-call reconciliation already charges actual usage.
Resolves conflicts from the upstream merge and addresses the Veria-AI
review comment on this PR: batch rows could bypass a project's
per-model ITPM/OTPM quota when the batch's file-bound/routing model
had no quota configured. Charges each row's own model against its own
project quota instead of only the routing model's, and fixes rate
limit error messages to attribute the correct model via a new
descriptor_value field on RateLimitStatus/AtomicCounterMeta. Also
re-syncs the ruff-strict, type-discipline, and basedpyright budgets
against the correct (non-stale) merge base.
Co-authored-by: Cursor <cursoragent@cursor.com>
Add model_itpm_limit and model_otpm_limit to project create and update requests, storing both quota maps in project metadata without a database migration
Reserve input and output tokens independently before provider dispatch, expose separate project rate-limit headers, and reconcile counters across successful calls, failures, retries, fallbacks, streaming, caching, and cancellation
Harden token estimation for pre-tokenized embeddings, multimodal inputs, Responses API requests, native Gemini requests, multiple candidates, and conflicting output-cap aliases
Reject negative output caps, preserve conservative reservations when usage is missing or zero, bind reconciliation and refunds to the reservation window, prevent double refunds or negative counters, update generated API types, and add regression coverage
The v3 parallel-request limiter stashed its per-request bookkeeping (TPM
reservation, descriptors, parallel slot, rate-limit response snapshot,
released flag) in the request body's metadata channels. On routes where
metadata is a provider request parameter (Responses API and the other
LITELLM_METADATA_ROUTES) that leaked internal keys upstream and produced
HTTP 400s, and it required denylist stripping plus dual-channel writes to
contain.
The stash now lives on an asyncio ContextVar holding a single typed
RequestRateLimiterStash per request. The pre-call hook writes it, and the
success/failure callbacks, disconnect release, and post-call hooks read
and clear the same shared instance, which keeps the refund and slot
release idempotent across sibling callbacks. The request body is never
touched, so the stash-key stripping, the metadata mirror writes, and the
all_litellm_params denylist entries are removed
* fix(rate-limit): stop v3 limiter from leaking internal stash to provider body
PR #27001 (atomic TPM rate limit) introduced a reservation flow that
writes four LiteLLM-internal keys onto the request data dict:
_litellm_rate_limit_descriptors
_litellm_tpm_reserved_tokens
_litellm_tpm_reserved_model
_litellm_tpm_reserved_scopes
_litellm_tpm_reservation_released
These keys are forwarded as request body params to the upstream provider,
which rejects them as unknown fields:
OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors'
(mapped by litellm to RateLimitError / 429, hiding the bug
behind a misleading 'throttling_error' code)
Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are
not permitted'
Net effect: every chat completion against any real provider fails the
moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced
key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check
itself still runs (raises 429 on over-limit), but the success path
poisons the upstream body.
Reproduced on litellm_internal_staging HEAD (410ce761dc) against
gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request
fails with the provider's unknown-field error.
Fix: the stash is metadata only.
- Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS
registry so we have a single source of truth for stash keys.
- New helper _stash_value_in_metadata_channels writes to
data['metadata'] / data['litellm_metadata'] without touching the
top level.
- _stash_reservation_in_data and the descriptor stash now route
through that helper. _mark_reservation_released stops writing
top-level.
- _lookup_stashed_value also checks kwargs['metadata'] /
kwargs['litellm_metadata'] (raw request_data shape) in addition to
kwargs['litellm_params']['metadata'] (completion kwargs shape).
- async_post_call_failure_hook now reads descriptors via the unified
metadata lookup instead of request_data.get(top-level).
- Defense in depth: async_pre_call_hook strips any stash key that
somehow surfaced at the top level (stale cache, future refactor,
test fixture) before returning.
Tests:
- New regression test asserts no _litellm_* stash key is present at
the top level of data after async_pre_call_hook, and that the
metadata channel still carries the reservation + descriptors so
success / failure reconciliation works.
- Existing test_tpm_concurrent.py tests that asserted top-level
presence are updated to read from data['metadata'] — the location
is an implementation detail; the spec is that post-call callbacks
can resolve the stash.
Verified end-to-end against OpenAI gpt-4o-mini and Anthropic
claude-haiku-4-5 via /v1/chat/completions on a low-rpm key:
- With limits not exceeded: HTTP 200, valid completion response,
no leaked fields in body.
- With RPM exceeded: HTTP 429 from v3 enforcement
('Rate limit exceeded ... Limit type: requests').
- With TPM exceeded: HTTP 429 from v3 enforcement
('Rate limit exceeded ... Limit type: tokens').
Full v3 hook test suite passes (171 tests).
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments
Address greptile P2: test fixture now uses the imported constant.
Drop comments that re-explain what well-named identifiers already convey.
* fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse
Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at
the start of async_pre_call_hook. Without this, an authenticated caller can
inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in
body metadata, trigger a proxy-side rejection, and cause
async_post_call_failure_hook to refund TPM counters against attacker-named
scopes (e.g. another tenant's api_key).
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
