dist/litellm-1.79.1.tar.gz (a 64-byte build artifact) was committed by
mistake. Release CI wipes dist/ before building, so it never affected
published artifacts, but it doesn't belong in version control. Add dist/
to .gitignore to prevent a repeat.
OpenAI GPT-5.6 Claude Code cells burn minutes on CLI timeouts under the
full stage suite; gate them behind COMPAT_OPENAI_GPT_CELLS=1 like Mantle.
Multi-window budget e2e used max_tokens=1 which gpt-5.5 rejects mid-message
PR #33736 made the shielded streaming cleanup await
proxy_logging_obj._arelease_max_parallel_requests_on_disconnect on the
client-disconnect path. The four streaming cancel and disconnect tests in
test_budget_reservation.py drive the generator with a bare MagicMock as
proxy_logging_obj, so the cleanup crashed with TypeError: object MagicMock
can't be used in 'await' expression, breaking proxy-infra CI on every PR
Give the mocks an AsyncMock for the release method and assert it is awaited
exactly once on each disconnect path, pinning the single-owner slot release
contract that PR #33736 introduced without test coverage
Remove the python_version < '3.14' environment markers from redisvl,
pypdf, and openapi-core now that all three install and import cleanly
on 3.14. The relock is marker-only: no package version changed for any
Python branch, and the locked versions (redisvl 0.4.1, pypdf 6.13.3,
openapi-core 0.22.0) now serve 3.14 as well. semantic-router and
aurelio-sdk stay gated because every published release caps
python_requires below 3.14
Editing an existing LLM credential and changing only the api_base also
overwrote the stored api_key with its masked display value (e.g. sk****IA).
The edit form pre-fills fields from the credential the backend returns, whose
secrets come back masked, and the update handler sent every field straight
back; the endpoint then encrypted and stored the asterisks over the real key.
Run credential_values through stripMaskedSecrets before the PATCH so masked
placeholders are never sent, mirroring the guard the model edit form already
uses. The isMaskedSecret / stripMaskedSecrets helpers move out of
model_info_view into a shared utils module so both call sites share one
implementation.
Add a Playwright e2e that seeds a credential, edits only the api base in the
LLM Credentials tab, and asserts the outgoing PATCH no longer carries the
masked api_key while the new base persists.
team_pattern_routers retained deleted/replaced deployments, so team users could
keep resolving stale credentials; now set_model_list resets the registry and
deployment removal prunes it. Also consult the team wildcard router before the
global pattern_router in get_deployment_credentials_with_provider so a global
pattern like "openai/*" no longer shadows the team's own entry
Co-authored-by: Cursor <cursoragent@cursor.com>
glm-5p2 (and its fireworks_ai/glm-5p2 alias) carried cache_read_input_token_cost
of 2.6e-07, the GLM 5.1 rate; the entry was seeded from the wrong row. Fireworks'
standard serverless rate for GLM 5.2 is $0.14/1M = 1.4e-07, so every prompt-cache
hit was billed at nearly double the real rate.
Corrects the value in both the canonical map and the bundled backup. The existing
fireworks cost-calculator test now reads the cached rate from the map instead of
hardcoding it, so it tracks the shipped value.
pyo3 0.23.5 hard-caps the interpreter at Python 3.13, so building the
native bridge against a 3.14 interpreter aborts inside pyo3-ffi's build
script before anything links. This raises pyo3 and pyo3-async-runtimes
to 0.29 (currently the newest line, and the range starting at 0.26 that
supports 3.14) and migrates the three call sites whose APIs were renamed
across that range: Python::with_gil is now Python::attach and
Python::allow_threads is now Python::detach. On a GIL-enabled interpreter
those are pure renames with identical semantics, so behavior on 3.10
through 3.13 is unchanged
Verified by compiling the native module for cp313 and cp314 and driving
it directly on both interpreters: gil_stats reports exactly one GIL
release per sync OCR call and the async path completes, matching the
0.23.5 baseline. cargo fmt, clippy, and the workspace tests pass on both
3.13 and 3.14 with the lockfile locked, and the lock churn is confined to
the pyo3 crates
Part of #26343; addresses the pyo3 build failure reported in #33116
The tag delete action moved into a Base UI dropdown menu when the tags
table was migrated onto the shared DataTable. That menu is modal by
default and holds a pointer-events lock on the page while it opens and
closes, which left the hand-rolled inline confirmation modal unclickable,
so deleting a tag stopped working
Replace the inline modal with the shared DeleteResourceModal, which
renders through an antd Modal portal that manages its own pointer-events
and z-index, matching every other table's delete flow. Add a deleting
loading state so the confirm button reflects progress and cannot be
double-clicked
Cover the wiring with a regression test that drives the delete flow
through the shared modal and asserts tagDeleteCall runs with the tag name
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes#33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in c698b88686 to match upstream litellm_internal_staging.
Regenerates basedpyright and ruff-strict budget baselines via make lint-budget-update.
* fix: scan system messages as indirect prompt injection in Singulr guardrail
* chore: restore lint budget files to upstream baseline
* fix: resolve ruff UP006 and I001 violations in singulr guardrail
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* resolve review comments on Singulr guardrail
* fix: scan tool call results as indirect prompt injection in Singulr guardrail
* Apply suggestion from @greptile-apps[bot]
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* minor
* formating fix
* refactor: shift extraction logic to singulr side
* refactor:keep precall hook only
* fix:formatting
* fix:linting
* improve config description
* Trigger CI
* fix
* fix:field description
* fix:errors due to change in field names
* style: apply ruff line-wrap formatting to singulr guardrail
* fix:exception
* fix:formatting
* fix playground
* improved
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* fix
* fix ci issues
* remove uv.lock from pr
* fix
* fix:resolved comments
* chore: trigger CI
* remove uv.lock
* fix
* fix linting
* fix linting
* fix linting
* remove doc strings
* remove test fixes
* chore: retrigger CI
* change in singulr api contract
* remove some ut
* send litellm call_id to singulr
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
* Fix non-conformant UUIDv7 generation in native Opik integration (#31294)
create_uuid7() encoded the timestamp in units of 16 seconds instead of
milliseconds, so the top 48 bits came out ~4096x the real unix-ms. Opik's
backend validates the embedded UUIDv7 timestamp on ingestion (OPIK-7067);
the bad encoding decoded to ~year 2201 and every trace/span batch was
rejected with HTTP 400.
Rewrite create_uuid7() to be RFC 9562 conformant (top 48 bits = unix-ms),
using the standard library only so no new dependency is added. Add unit
tests covering UUIDv7 validity and millisecond timestamp encoding.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(proxy): expose uvicorn concurrency limit (#33077)
Expose uvicorn's limit_concurrency as a --limit_concurrency CLI flag and
LIMIT_CONCURRENCY environment variable. Uvicorn counts both active tasks and
accepted connections and returns HTTP 503 once the configured limit is reached.
Reject non-positive limits at CLI parse time and only add the setting to the
uvicorn startup arguments. Because idle connections also consume capacity,
deployments should use upstream connection/header timeouts and per-client
connection limits.
* test: reorder test_utils tail to keep the daily merge conflict-free (#33788)
The daily OSS branch and litellm_internal_staging each appended an
independent test block at the very end of tests/test_litellm/test_utils.py,
so merging the two collides on that shared end-of-file position even though
the additions are unrelated (this branch adds the vertex embedding
encoding-format tests; staging adds the per-model prompt-cache-minimum
tests). Moving this branch's new TestVertexEmbeddingEncodingFormat class
above test_gemini_image_models_do_not_support_reasoning, which both branches
share, gives the two additions different anchors, so git applies both
without a conflict and without pulling staging into this branch. Pure
reorder; no test bodies change
---------
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: madan-singulr <150280287+madan-singulr@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: aniket-kardile <aniket.kardile@singulr.ai>
Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>
Co-authored-by: Aliaksandr Kuzmik <98702584+alexkuzmik@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Salva Madrid <50212436+salvamadrid@users.noreply.github.com>
Add an e2e suite at tests/e2e/mcp/ that proves MCP authorization over the
api_key auth family. An admin registers an upstream MCP server through the
management API (POST /v1/mcp/server, persisted in the DB and picked up without
a restart) and queues its deletion. Two keys are created against that one
server: one granted access through object_permission.mcp_servers and one with
no MCP grant. The permitted key is a live control proving the upstream is
reachable and the tool is callable, so a denial on the ungranted key is an
authorization decision rather than a dead server. The denied key then sees
none of the server's tools on tools/list and is refused a tools/call with a
403 access_denied.
A deterministic self-hosted FastMCP upstream (add/multiply over
streamable-http) is added to the e2e compose stack so the suite runs offline
with a known tool set. KeyGenerateBody gains an optional typed
object_permission so the shared gateway can create a key with an MCP grant.
* refactor(e2e): replace bespoke result reporter with standard JUnit report
tests/e2e/e2e_result_reporter.py hand-rolled a per-test logfmt emitter that
reimplemented outcome mapping, logfmt escaping, and node-id parsing to print one
E2E_RESULT line per finished test. Outcome, duration, and node id are all things
a standard pytest reporter already produces, so the only genuinely custom data is
the covers marker ids and the normalized package label
Delete the module and emit a standard pytest JUnit XML report (--junitxml)
instead, carrying the two custom signals as user_properties (JUnit <property>
entries) attached at collection time in pytest_collection_modifyitems, so they
land on every test on every outcome including skips and setup errors. The small
package/covers extraction lives in junit_properties.py and is unit tested plus
checked end to end against a real JUnit artifact in test_junit_properties.py
Shipping the JUnit report to Loki is a thin infra-side transform, documented in
grafana/status_history_panels.md
* chore(e2e): remove grafana status history panels doc and junit properties e2e test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Outcome keys in the tools/list _meta, the spend-log outcome and count maps, and the REST error
messages now all use get_server_prefix (alias, or the short prefix when that mode is enabled), the
same naming the caller already sees on tool names. Keying them by canonical server_name let an
authenticated caller enumerate internal server names and their health or auth state that the alias
and short-prefix schemes deliberately hide (Veria finding). One helper decides the key for every
surface; exception messages reaching the multi-server REST error list are mapped to their fault tag
with the display prefix instead of relaying exception text carrying canonical names. Server-side
logs keep the real names
The e2e docs claimed `e2e`-marked tests skip when no proxy answers the
liveness probe, but the harness has always hard-failed: conftest.py's
pytest_runtest_setup calls pytest.fail, its module docstring states
"hard failures only ... never skip", and logging/conftest.py forbids
skipping outright. Align the docs to the code so the single most
important contract reads the same everywhere; a dead proxy turns a run
red instead of being silently skipped and mistaken for a pass. The
per-suite conftest docstrings that described the shared hook as a
"proxy liveness skip" are corrected to "liveness gate" for the same
reason.
Also scope the no-unit-tests hard rule to what it means: never
substitute a unit test for e2e feature coverage, while explicitly
allowing tests that cover the harness itself (e.g.
coverage_registry/test_collector.py), which carry no e2e marker and
run whether or not a proxy is up.
No product code and no harness logic changed.
Resolves LIT-4554
tests/e2e/grafana/status_history_panels.md was prose describing Loki/Grafana
status-history panels and LogQL queries. Nothing in the tree imports, reads, or
links to it; the e2e suite only emits the E2E_RESULT lines those panels consume
(tests/e2e/conftest.py, tests/e2e/e2e_result_reporter.py) and never depends on
this file. Dashboards drift when versioned as prose in the repo, so remove it;
if we want them versioned it should be dashboard-as-code in the observability
repo, not markdown here.
* fix(proxy): bill partial streamed spend when the client disconnects mid-stream
* fix(router): guard FallbackStreamWrapper chunks alias for non-CSW streams
* fix(proxy): await disconnect billing dispatch instead of unrooted create_task
* fix(proxy): make disconnect slot release single-owner to avoid double release
* fix(proxy): use union syntax for disconnect cleanup params (UP045 budget)
The toggle and ttl descriptions were a wall of text, with a panel intro that
mostly repeated the toggle description. Drop the intro and cut both descriptions
to one or two lines, keeping a one-clause note that the cache is shared across
callers on the same upstream credentials.
Conflict in _list_mcp_tools: staging (#33612) moved toolset-grant expansion into the shared
permission primitives and removed the _merge_toolset_permissions call; resolution applies that
removal to this branch's AggregateToolListing structure
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
* test(e2e): assert an org budget block is a 429 naming the organization
Rather than mixing the flag and its ttl into the generic General settings table
(which also surfaced the confusing Not Set / In Config / In DB provenance badges),
give prompt caching a dedicated tab with a purpose-built toggle and ttl dropdown.
Each registry field gains an optional tab, surfaced as ConfigList.field_tab, so
the General tab renders the ungrouped fields and the caching fields render on
their own tab. The update, persist and reset endpoints are unchanged.
_should_auto_execute_tools returned True as soon as any MCP reference set
require_approval="never", so a request that mixed a "never" reference with an
"always" or "manual" one auto-executed every tool call the model produced,
including the approval-gated ones. A prompt could name the approval-required
tool and have it run with no approval.
Make the gate fail closed: auto-execute only when every reference opts in with
"never". A single approval-required reference (including the object form or an
unset value) returns the model's tool calls to the caller instead of running
them, so an approval-gated tool can never be auto-invoked. This is the shared
decision behind /chat/completions, /responses, the streaming iterator and the
new /v1/messages path, so all four fail closed from one change. The common case,
every reference "never", is unchanged.
The alternative, executing the "never" calls and returning only the
approval-required ones, needs partial execution that the Anthropic tool loop
cannot express without fabricating tool_result blocks for the calls it withheld,
so the whole-request fail-closed gate is the safe minimum. A future change can
add per-call partial execution if a caller needs it.
Test covers the mixed and manual cases; reverting to "any never" fails it.
* test(e2e): assert bare-key budget refusal is 429 and /key/info spend reaches the cap
* test(e2e): keep the bare-key budget assertion to the 429 refusal shape
* test(e2e): assert a team's max_budget blocks every key on the team
* test(e2e): focus the team budget case on the 429 blocking behavior
The provider caches a prefix against the credentials that sent it, not per end user, so
turning the flag on makes every caller's prompts cacheable on that shared account. Surface
that where the toggle is, since it is the operator's call to make.
The value cell was a ternary chain over field_type; adding Select made it a fourth
level and tripped no-nested-ternary. Early returns read better than a deeper chain
and let the suppression baseline ratchet down.
Register enable_anthropic_prompt_caching and anthropic_prompt_caching_ttl on the
General Settings table so caching can be turned on without hand-writing config.
The registry could not express either field: validation was hardcoded to a float in
(0, 1], reset set every field to None (not a bool for a boolean flag), and the listing
reported any non-None value as 'In Config', which a False default would always trip.
Validation now dispatches on the declared type and reset restores each field's own
default. ConfigList carries field_options so the table can render a Select for enums
instead of no editor at all.
The gpt-realtime family (OpenAI and Azure) only serves /v1/realtime and is rejected by /v1/chat/completions with "This is not a chat model", but the cost map tagged them mode=chat. Retag them mode=realtime (a value already used by gemini-live and handled by the health-check realtime handler) and add realtime to the ModelInfoBase mode literal.
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Two review findings, both a chat-vs-messages divergence.
transform_mcp_tool_to_anthropic_tool sent the MCP inputSchema to Anthropic almost
as-is, while the chat path (_map_tool_helper) coerces the type to object, inlines
legacy definitions with unpack_legacy_defs, and allow-lists keys to
AnthropicInputSchema. So a tool whose schema carried $schema, legacy definitions
or oneOf worked on /chat/completions and 400d on /v1/messages; a clean-schema
server hid it. Both paths now run the same sanitize_input_schema_for_anthropic,
extracted next to unpack_legacy_defs so they cannot drift again, and the chat
path is refactored onto it rather than keeping its own copy.
buildMcpToolBlocks percent-encoded the server and toolset names inside
litellm_proxy/mcp/... urls, but the gateway resolves the name with a raw
server_url.split("/")[-1] and never url-decodes, so a name with a space failed
lookup. The already-working chat path does not encode; the shared builder now
matches it.
Tests pin both: reverting the transform to the unfiltered schema fails, and
re-adding encodeURIComponent fails the builder test.
* feat(complexity-router): user-triggered escalation keywords
Add an escalation_keywords config option to the complexity router so a user
can force a bump to the next-higher complexity tier by including a phrase in
their message (a stronger model, but not one they get to choose). Defaults to
['LITELLM ESCALATE'] when unset, case-sensitive so it only fires on the
deliberate shouted form; admins can override the list or set [] to disable.
Escalation applies across every routing path: heuristic/LLM classification,
literal and semantic keyword_tier_rules overrides, adaptive routing, and
session affinity (where it bumps relative to the pinned model and persists the
higher tier for the rest of the session). Capped at the highest configured
tier and skips unconfigured intermediate tiers.
Expose it in the Auto-Router v2 UI as an Escalation Keywords field wired into
the complexity_router_config payload.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity-router): validate escalation keywords and pin at tier ceiling
Strip blank/whitespace escalation keywords so an empty phrase can't match every message and escalate all traffic. Keep the exact pinned model when a session escalates at the highest configured tier instead of randomly hopping to a peer in a multi-model pool.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Append-append conflict at the end of test_mcp_server.py between this branch's aggregate-outcome
tests and the mode-aware preemptive-401 tests from staging; both kept
* fix(proxy): stop treating upstream model body field as a LiteLLM model on auth-enforced pass-through routes
An auth: true user-defined pass-through endpoint runs full virtual-key auth, and get_model_from_request unconditionally extracted the request body model field, so key/team/user/project model allowlist checks rejected requests whose model only exists upstream (key_model_access_denied), even when the key was explicitly granted the route via allowed_passthrough_routes.
The pass-through route registry moves to a leaf module (route_registry.py) that the auth layer can import without re-entering the pass_through_endpoints -> user_api_key_auth -> auth_utils import cycle. get_model_from_request now returns None for routes registered as user-defined pass-through endpoints (exact and subpath), which skips model allowlist and per-model budget enforcement on those routes while key auth, allowed_passthrough_routes, and spend/budget checks stay intact. Built-in provider passthrough routes (/vertex_ai, /gemini, ...) keep model enforcement.
Resolves LIT-4299
* fix(proxy): key pass-through model-access skip on the dispatched endpoint, not the request path
Addresses a model-authorization bypass: the first version decided whether to skip
model-allowlist extraction by matching the request path against the pass-through
route registry. That ignored the HTTP method and, more importantly, whether the
request was actually dispatched to a pass-through handler. A custom pass-through
whose path collides with a built-in route (e.g. /v1/chat/completions, or an
include_subpath prefix of one) still writes a registry entry even though FastAPI
serves the built-in handler, so a normal request to that route had its model checks
skipped and could reach a model outside the key/team/user/project allowlist.
The skip is now keyed off the FastAPI-resolved endpoint. create_pass_through_route
tags its handler with LITELLM_PASS_THROUGH_ENDPOINT_MARKER, and get_model_from_request
returns None only when request.scope["endpoint"] carries that marker. Because routing
runs before auth dependencies, this reflects the handler that actually serves the
request: on a collision the built-in handler is dispatched and carries no marker, so
model enforcement stays on. This also removes the need for the separate route_registry
module, so that extraction is reverted.
Regression tests cover a pass-through-dispatched request (model suppressed), a
built-in-dispatched request on the same path (model still enforced), and the no-request
budget path.
Resolves LIT-4299
* fix(proxy): enforce max_parallel_requests as a per-slot concurrency gauge
The v3 rate limiter tracked max_parallel_requests with the same
sliding-window machinery as RPM/TPM. A concurrency gauge cannot live on a
windowed counter: every window roll reset the counter to 1 while requests
were still in flight, the completion decrements for those forgotten
requests then drove the counter negative, and rejected requests left
stranded increments that nothing released. Under sustained load a key with
max_parallel_requests=5 let backend concurrency climb to the full client
concurrency (observed 60 on a live proxy) while the proxy kept returning
429s for everyone else
Replace the windowed counter with a per-slot registry (Redis sorted set of
slot ids scored by acquire time, with an asyncio-locked in-memory fallback):
admission atomically prunes expired slots and registers a new slot id only
when in_flight + 1 <= limit, so rejected requests never occupy a slot;
success, failure, and client-disconnect paths release exactly the slot id
this request acquired (stashed in the request metadata channels), so a
release without a matching acquire or a double-fired callback can never
free another request's slot; and a slot leaked by a crashed worker is
pruned individually after its TTL even under continuous traffic
Resolves LIT-4259
Fixes#16011
* fix(proxy): release every acquired gauge and respect mirrored counts in the in-memory fallback
Address review findings on the slot-registry gauge: the acquisition stash
now carries the gauge counter keys alongside the slot id, so the release
paths free the slot from every gauge it was registered under instead of
hardcoding the api_key scope, and the disconnect release keys off the
stashed acquisition instead of the key object's current
max_parallel_requests configuration (which can change mid-request). The
in-memory fallback now treats a cached integer (the count mirrored from
the last successful Redis script call) as real occupancy, carrying it
forward as a floored counter during a Redis outage instead of restarting
from an empty registry
* fix(proxy): release the parallel slot on proxy-level rejections
async_post_call_failure_hook is the only callback that fires when a
downstream hook (guardrail, budget check) rejects a request after the rate
limiter's pre-call hook acquired a slot; async_log_failure_event is a
completion-level callback and never runs for proxy-side rejections.
Release the stashed acquisition at the top of the hook, before the TPM
reservation guard, so those slots do not linger for the full slot TTL and
wedge the key at its limit under moderate rejection rates. Clearing the
acquisition marker keeps the release idempotent when a later failure
callback runs in the same flow
* test(proxy): cover success release, read-only count, Redis release mirror, and TPM rejection release
Four behaviors of the slot-registry gauge had no direct test: a successful
completion releasing exactly its acquired slot, read_only callers counting
in-flight slots through the count script (and degrading to the local
mirror when the script fails) without acquiring, the Redis release script
mirroring returned counts into the local cache, and the TPM reservation
rejection releasing the already-acquired slot before raising
* style(proxy): use builtin generics and union syntax in new rate limiter annotations
The slot-gauge code added Tuple/List/Dict and Optional[...] annotations, pushing
the UP006 and UP045 strict-rule totals past their ceilings in ruff-strict-budget.json.
Convert only the annotations this branch introduces to builtin generics and PEP 604
unions, leaving the rest of the module untouched.
* fix(router): tag-aware pre-routing strategy selection for shared model_name
Complexity/auto/adaptive/quality router registries were keyed by model_name
alone, so a second deployment sharing a model_name but carrying different tags
was rejected and every request used the first config. This made tag-based
routing to distinct provider configs behind one alias impossible, surfacing as
401 'Not allowed to access model due to tags configuration' for the second tag.
Each registry now holds a list of tag-scoped strategies and async_pre_routing_hook
selects the entry whose tags match the request before classification, falling
back to a default-tagged then first-registered entry. A repeat of the same
(model_name, tags) pair is still rejected.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover tag-scoped pre-routing strategy registry helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: re-trigger CI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
