Confirmed real. _canonical_target_is_allowed returned True when
user_api_key_dict was None, on the reasoning that no key context meant
non-proxy Router use. That reasoning was wrong: of the 12 route_request call
sites, only one (common_request_processing, the /chat/completions path)
forwards user_api_key_dict. Image generation, rerank, moderation, speech,
transcription, realtime, and the Responses WebSocket path are all
authenticated endpoints that call route_request without it, so the rewrite ran
with no target authorization at all on exactly those paths -- a key whose
allowlist held only the stale requested spelling could reach a canonical
target it was never granted.
Now returns False. Declining costs those endpoints only the convenience
rewrite: they behave exactly as they do today, resolution simply never
engages, and no request that works now breaks. The AND-on-target guarantee
becomes unconditional rather than depending on whether a given caller happens
to thread the key through.
Verified the primary path is unaffected: with key context present and
authorized, resolution still resolves -- that is the Claude Code 403 case this
PR exists to fix. Threading user_api_key_dict through the other call sites is
the follow-up that re-enables resolution for them; doing it here would touch
11 unrelated endpoints in a PR that should not be changing their signatures.
Updated test_no_auth_context_is_allowed -> test_absent_auth_context_fails_closed
with the reasoning recorded.
Gates: pytest 531 passed - ruff format/check clean - type-discipline all LIT
rules at base parity - basedpyright new module 0 errors.
Co-Authored-By: Claude <noreply@anthropic.com>
All three confirmed real and reproduced before fixing.
1. [High] Team allowlist bypassed AND-on-target. The rewrite re-auth called
can_key_call_model, which checks only the key's own allowlist, so team,
team-member, and project allowlists were never re-checked against the
resolved target. An unrestricted key on a team whose allowlist held only a
stale unserved name could ride the rewrite onto a deployment that team was
never granted. Now calls can_key_call_resolved_model -- the helper every
other post-resolution auth site already uses (model_group_alias rewrites,
realtime endpoints, auto-router), which runs the full key/team/member/project
chain.
2. [Medium] Index ignored the deployment's custom_llm_provider, canonicalizing
only litellm_params.model. Reproduced: a deployment with
model='claude-haiku-4-5' + custom_llm_provider='openrouter' was indexed under
('anthropic', ...), so an Anthropic-form request resolved onto OpenRouter
credentials, quota, and billing -- precisely the cross-provider hop rule 2
of the module docstring forbids. canonicalize() now takes the override and
prefers it over the provider inferred from the model string.
3. [Medium] Operators could not select model_name_resolution: strict on the
proxy. Both proxy Router construction sites passed a hardcoded
RouterGeneralSettings(async_only_mode=True) as an explicit keyword alongside
**router_params. Since router_general_settings IS an accepted config key,
setting it raised TypeError: got multiple values for keyword argument at
startup -- so the documented opt-out was unreachable either way. Added
_proxy_router_general_settings(), which forces async_only_mode (a proxy
runtime requirement) while preserving every other operator-set field, and
copies rather than mutating the caller's object.
Added 10 regression tests. Verified the two re-auth tests actually fail against
the pre-fix code (2 failed / 1 passed) rather than passing vacuously.
Gates: pytest 531 passed - ruff format clean - ruff check clean - strict
BLE001/PERF401 at base parity (2957/22) - type-discipline all LIT rules at base
parity - basedpyright new module 0 errors - router_code_coverage 0.0% untested
- proxy_server imports cleanly.
Co-Authored-By: Claude <noreply@anthropic.com>
The lint job has three independent gates; this is the third (type-discipline
budget, delta vs base). Rather than keep discovering them one CI round at a
time, ran scripts/check_type_discipline.py locally over the whole tree and
brought every LIT rule back to exact base parity.
Conformed rather than blanket-suppressed where the rule was right:
- build_canonical_index takes a read-only Sequence[Mapping[str, object]] and
returns a MappingProxyType, so the index cannot be mutated by callers after
construction. This also removed the cast() at the router call site (LIT006)
and the now-stale reportUnnecessaryIsInstance suppressions -- with object
values the runtime validation is genuinely necessary, not redundant.
- spellings is built functionally (two generator-fed tuples) instead of
seed-then-append.
- The AND-on-target auth re-check moved into a small _canonical_target_is_allowed
helper, which removes a rebound flag and makes the AND-on-target contract
documented in one place.
- Genuine cases carry the house-style justification comments: mutable-ok for
the grow-only log-dedup ledger and the local accumulators (frozen on return),
rebind-ok for the in-place rewrites route_request already performs on
every other route, and for per-iteration loop variables (where Final is
illegal).
Verified locally against every gate this time, not just the ones CI had
surfaced: pytest 522 passed · ruff format clean · ruff check clean ·
ruff-strict BLE001/PERF401 at base parity (2957/22) · type-discipline all LIT
rules at base parity · basedpyright new module 0 errors · router_code_coverage
0.0% untested.
Co-Authored-By: Claude <noreply@anthropic.com>
The lint job's 'Check ruff format' step (separate from the strict-rule budget
step that failed earlier) flagged two files. Both diffs are confined to lines
added by this PR -- the multi-line isinstance/and condition in
_can_object_call_model, and the build_canonical_index call in
_get_canonical_model_index -- so no unrelated formatting is swept in.
Cosmetic only, no semantic change.
ruff format --check: clean. ruff check: clean. basedpyright (new module): 0
errors. strict-rule budgets BLE001/PERF401: unchanged at base parity. Tests:
32 passed.
Co-Authored-By: Claude <noreply@anthropic.com>
Three issues flagged by automated PR review (greptile-apps, veria-ai), all confirmed real and reproduced before fixing:
1. [High/Security] Team-owned deployments entered the global canonical index
unconditionally. A no-team (unrestricted) key could request an unclaimed
spelling of a model (e.g. the dated Anthropic ID) whose only server was a
team's private deployment, resolve onto it, and use that team's
credentials/quota -- target-authorization passes for unrestricted keys and
doesn't itself re-derive team ownership. Fixed by excluding any deployment
with model_info.team_id set from the index entirely: a team boundary is an
access/billing boundary exactly like the cross-provider boundary this
module already respects, and team-scoped models remain reachable exactly as
before, via team_public_model_name through the existing team-route
machinery this module never touches.
2. [P1] Sentinel collision defeated the ambiguity guard: index.get(key,
'__absent__') treated a model group literally named '__absent__' as a
missing entry, so a second group with the same identity would silently
overwrite it instead of triggering the ambiguity decline. Fixed with a
proper 'in' check.
3. [P2] Log deduplication was keyed on target alone, so a second distinct
requested spelling resolving to an already-logged target never got its own
log line -- undercounting the (requested, target) cardinality that's the
whole point of the observability story (sizing follow-up-resolution
demand). Now keyed on the (requested, target) pair.
Added 5 regression tests reproducing each bug pre-fix and asserting the fixed
behavior. Full affected suite: 522 passed. router_code_coverage: 0.0% untested.
ruff-strict BLE001/PERF401: unchanged at base parity. basedpyright: new module
0 errors.
Co-Authored-By: Claude <noreply@anthropic.com>
Three CI gates failed on the initial push; all three were real:
1. lint (strict-rule budget): BLE001 +5, PERF401 +1 over base. The defensive
'except Exception' catches are intentional (an unplaceable name or a
malformed deployment must never fail a request or brick a router), so they
now carry '# noqa: BLE001' with justifications, matching the convention
already used in router.py and common_request_processing.py. The candidate
-spelling loop becomes a generator passed to list.extend (PERF401).
Both rules are now back at base parity.
2. code-quality (router_code_coverage): the checker only scans test files whose
filename contains 'router', so tests/.../test_canonical_model_resolution.py
was invisible to it and both new Router methods read as untested. Renamed to
test_router_canonical_model_resolution.py and added direct coverage for
_get_canonical_model_index (memoization + fenced build failure). Checker now
reports untested_perc: 0.0.
3. proxy-infra (test_route_non_a2a_model_raises_error_if_not_in_router): a real
regression. The test drives route_request with a Mock() router, so
resolve_canonical_model_name returned a truthy Mock and the hook treated it
as a resolved target instead of raising. Both hooks (routing and auth) now
require an actual non-empty str before acting on a resolution -- correct
hardening independent of the test: a stub or partially-initialised router
must never be read as a grant.
Full affected suite: 517 passed. basedpyright: new module 0 errors.
Co-Authored-By: Claude <noreply@anthropic.com>
basedpyright: new module at 0 errors; modified files back to baseline
(+1 structural 'Return type is Any' on the new route return, matching the
file's existing pattern on every route return). ruff clean.
Co-Authored-By: Claude <noreply@anthropic.com>
Implement Feature A: when a request names a model using one spelling (e.g.
dated 'claude-haiku-4-5-20251001') that the gateway serves under a different
spelling (e.g. 'anthropic/claude-haiku-4-5'), route the request to the serving
deployment instead of failing with a 403/400. Resolution only fires after all
existing routes (exact name, alias, wildcard, default_deployment, team routes)
have declined, so a request that succeeds today is never re-pointed (I1).
Identity is enforced strictly: same provider (never cross-provider like Bedrock
to Vertex), same model per cost-map attestation (never family/version hops like
claude-sonnet-4-5 to claude-sonnet-5) (I2).
Auth is AND-on-target: the caller must be allowed to call the resolved target
group; the requested spelling's presence in an allowlist alone grants nothing,
preventing privilege escalation via stale spellings (I3).
Observability: requested name preserved in request metadata, resolution logged
at INFO per (requested, target) pair, Prometheus counter for cardinality.
Changes:
- litellm/router_utils/canonical_model_resolution.py: new module with
canonicalize, build index, lookup functions.
- litellm/router.py: Router.resolve_canonical_model_name(), index caching +
invalidation, config flag model_name_resolution.
- litellm/types/router.py: RouterGeneralSettings.model_name_resolution field.
- litellm/proxy/route_llm_request.py: resolve hook before 400, re-auth on
target, metadata stamping.
- litellm/proxy/auth/auth_checks.py: canonical lookup in _can_object_call_model
with AND-on-target semantics.
- tests/test_litellm/router_utils/test_canonical_model_resolution.py: 23
property tests covering I1–I4, cross-provider block, ambiguity fail-closed,
auth semantics.
Defaults to 'canonical' (on); users can set model_name_resolution: strict in
router_settings to opt out entirely.
Fixes the Claude Code case: dated Haiku requests now resolve across harness
versions without client-side env vars.
Follow-up: once operator data shows a single provider deployment of all
Anthropic models, Feature B (cross-deployment resolution when only 1 provider
exists) becomes safe to implement under similar guards.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(proxy): requeue spend logs when the DB write fails with a transport error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hardcode the spend log queue cap and drop the stale re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): keep the spend log requeue within the type discipline budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): apply the spend log queue cap to producer appends too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): lower the spend log queue cap to 1k and make it env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound the spend log queue by bytes instead of row count
A row cap cannot bound memory: a row carries the whole prompt under store_prompts_in_spend_logs, so a cap that rides out an outage of counter-only rows is an OOM once prompts are stored. Every enqueue and dequeue now goes through one pair that tracks what the queue costs and drops the oldest rows past a 64 MB budget.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte budget env configurable
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): use a string default for the spend log queue byte budget env read
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): make the spend log queue byte total a public attribute
The queue it accounts for is already public, and a private name only bought reportPrivateUsage errors at every call site.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: shivam <shivam@berri.ai>
Dropping it falls back to the public cost map's tier table, whose rates outrank the
zeros written beside them, so a PTU deployment on a tiered model keeps billing its
traffic per token. Stored empty, the tiers no longer apply and the zeros win
tiered_pricing is a list, so the 0.0 the flat-rate zeroing stores does not
even validate. Supplying tiers alongside PTU config gets the same 400 as a
flat rate; tiers already stored are dropped from both blobs
parse_prompt_tokens_details and parse_completion_tokens_details are imported
by four modules, so the leading underscore made every import a
reportPrivateUsage violation
The previous assertion read data-chips, which is derived from the anchor prop
being truthy, so it stayed true even when the ref never reached the DOM and the
popup was still anchored to the inner input.
Stub distinct widths on the chips container and the input, then read the width
the positioner resolved. Reverting the anchor wiring now reports the input's
width instead of the field's, which is the actual bug.
Base UI positions a combobox popup against the Combobox.Input by default. In
chips mode the visible field is the ComboboxChips wrapper and the input is a
smaller box nested inside it, so every chips-combobox in the dashboard opened
its popup 11px right of the field and 17px past its right edge.
shadcn ships the wiring for this and their combobox-multiple example uses it:
useComboboxAnchor on the chips container, passed to ComboboxContent as anchor.
The anchor prop also drives data-chips, which cancels the extra min-width an
ordinary combobox wants. Every chips site in the dashboard omitted it.
The anchor is attached through Base UI's render prop rather than a plain ref,
because React 18 drops refs on function components and ComboboxChips is one.
Adds MultiSelect's first test, covering the anchor wiring plus selection,
chip rendering and custom values.
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
The tab strip carried overflow-x-auto directly on the TabsList. CSS forces
overflow-y from visible to auto once overflow-x is not visible, and the line
variant's active-tab underline is an absolutely positioned ::after that hangs
5px below its trigger, so the strip picked up a pixel of vertical scroll on top
of the horizontal scroll it actually wants.
The scroll container now lives on a wrapper whose bottom padding leaves room for
the underline, offset by a matching negative margin so the row keeps its exact
geometry.
* fix(mcp): drop caller host and configured upstream headers from logged metadata
The synthetic request that carries MCP client headers into
add_litellm_data_to_request forwarded the caller's Host header, and
Request.url is built from it, so a caller chose the proxy_server_request
url and the metadata endpoint that every logging callback records.
_upstream_credential_headers also only knew the configured client side
auth header and the x-mcp- prefix family, so a header name declared in
mcp_servers.<name>.extra_headers reached logging metadata in cleartext.
Those names are admin chosen, so no prefix rule can recognize them; read
them off the server registry instead. The header is still forwarded
upstream, which is what extra_headers is for. authorization is left out
because clean_headers already strips it and claiming it here would move
authenticated_with_header on the oauth passthrough config.
The Responses bridge tests stub the server manager, so their fakes gain
the registry accessor the sanitizer now reads.
* fix(mcp): drop caller host from the sanitized header mapping too
The synthetic request stopped forwarding host, but the parallel sanitizer
did not, so a forged hostname still reached the guardrail payload and the
list_tools spend row. Drop it there as well.
Exempt the configured identity headers from the upstream credential set.
get_user_from_headers resolves end user attribution off the same request
this module reconstructs, and it only fills end_user_id when auth left it
unset, so claiming user_header_name or a user_header_mappings name would
lose attribution on the MCP paths that authenticate upstream.
Drop the isinstance guard on extra_headers entries: the field is typed
list[str], so the check is dead and basedpyright scores it.
* fix(mcp): accept a bare user_header_mappings entry when exempting identity headers
get_internal_user_header_from_mapping and get_customer_user_header_from_mapping
both normalize a single mapping to a one element list, and config_settings.md
documents the key as a dict. Iterating the bare form yields its keys instead,
so the exemption silently matched nothing and an identity header also named in
an MCP server's extra_headers was dropped after all.
