* fix(proxy): log disable_budget_reservation notice once at config load
The disabled-budget-reservation reminder fired as a WARNING inside request
authentication, so every authenticated request on a proxy that deliberately
set the flag produced one warning line. The notice now runs once per worker
when general_settings loads, at INFO, and the request path only skips the
reservation. Reservation skipping and read-time budget checks are unchanged
* fix(proxy): keep budget notice sentinel with constants
* fix(proxy): expose shared budget notice state
* fix(bedrock): keep x-amzn-RequestId on chat error responses
Bedrock chat error paths built BedrockError from only a status code and a
message, so the provider response headers were gone before exception mapping
ran and the proxy had nothing to forward. AWS support needs x-amzn-RequestId
to investigate a server-side error.
- converse and invoke chat handlers pass the real headers and response when
they turn an httpx.HTTPStatusError into a BedrockError, and read the body
through error_response_text so a streamed body nobody read does not throw
- every bedrock chat get_error_class honors the headers it is already handed:
invoke, moonshot, bedrock-hosted openai, agentcore and the invoke agent
- BedrockError carries those headers into the response it synthesizes when a
caller has headers but no response, skipping values httpx cannot carry
- the bedrock 500 mapping forwards the provider response like its 4xx and 503
siblings instead of fabricating a blank one
The proxy now returns llm_provider-x-amzn-requestid on Bedrock chat errors.
* fix(bedrock): keep request-id on text-classified errors
The context-window and image branches of _map_bedrock_exception built their
litellm exception without the provider response, so a Bedrock 400 classified
by its body text lost x-amzn-RequestId while the sibling branches kept it.
Also narrows the new BedrockError types and trims its docstrings.
* chore(bedrock): drop the docstrings on the new error helpers
* fix(bedrock): keep request-id on every error path that has one
The ticket's root cause is that every BedrockError raise site under
litellm/llms/bedrock/ was built from status and message alone. The first
commits covered the chat and invoke handlers; this covers the rest.
Embeddings, rerank, image generation, image edit, count tokens, search and
the transformation layers now hand on the provider response or its headers,
and both bedrock_mantle configs return a BedrockError instead of the
OpenAI error that drops them.
Two blockers surfaced while verifying the streaming path. The trailing
`except Exception` in make_call and make_sync_call swallowed the BedrockError
raised a few lines above, relabelling a provider status as a 500, and the
non-200 branch read an unread streamed body, which throws.
The raise sites left alone have no provider response to carry: timeouts,
credential and config errors, and mid-stream event frames.
* fix(bedrock): forward provider headers from the count tokens route
The count tokens route converts BedrockError into an HTTPException, and dropped
the headers the handler had just kept, so that route still lost the request id.
get_response_headers now takes a Mapping so an httpx.Headers can be handed to it
without a copy.
* fix(bedrock): classify every bedrock surface through BedrockError
Eleven bedrock configs still inherited a provider-agnostic get_error_class
that builds a blank response, so the request id was gone before the proxy
read it. Claude platform, bedrock anthropic-messages, both image edit
configs, passthrough, realtime, vector stores and agentcore search now
return BedrockError, and a parametrized audit drives all 36 configs.
* fix(proxy): keep provider headers on the httpx status error branch
_handle_llm_api_exception forwards safe_headers on every branch except the
httpx.HTTPStatusError one, which the bedrock passthrough route reaches, so
the request id was dropped before the client saw the response.
* fix(bedrock): keep the request id on the timeout mappings
Timeout takes no response argument, so the three bedrock timeout branches
dropped the provider headers even when the upstream answered 408 or 504
with an x-amzn-RequestId. They now ride on the exception, already
llm_provider-prefixed, which is the form the proxy emits.
* fix(bedrock): keep the provider response on mapped timeouts
The previous round attached llm_provider-prefixed headers directly to the
Timeout. That shadowed the raw upstream headers for _get_response_headers,
so router cooldown and fallback cooldown stopped honouring retry-after on
bedrock 408/504 replies.
Give Timeout an optional response instead, the way every other mapped
bedrock exception already carries one. Retry logic reads the raw
retry-after off the response, and the proxy prefixes those headers on the
way out, so clients still see llm_provider-x-amzn-requestid.
* chore(bedrock): drop the explanatory comment on Timeout.response
Take staging's test_bedrock_knowledgebase_hook.py, which drops the duplicate
embedding_executor parameter that turned the lint check red, and make the two
cross-module helpers this branch added public (raise_denied_scoped_mcp_access
and routes_through_gateway) so the private-usage budget stays at its base count
Encrypt secret maps at the shared persistence boundary, preserve plaintext API/runtime views, and extend rotation and migration scanning to legacy rows.
Co-authored-by: Claude Code <noreply@anthropic.com>
Move the only guidance worth keeping into the parity assertion message so a
failing run tells the contributor to add a catalog entry instead of growing
the frozen unlisted set.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
The Add Model provider dropdown is driven entirely by provider_create_fields.json,
and chatgpt had no entry there, so the documented ChatGPT subscription setup was
unreachable from the Admin UI. Add the entry plus the dashboard enum, slug, logo and
placeholder mappings so the provider can be selected and its cost-map models listed.
The entry carries no credential fields on purpose: the chatgpt backend ignores
api_key and api_base and signs in through the device-code auth file on the proxy
host, so any field here would be inert.
Add a parity test that every LlmProviders value is either listed for Add Model or
frozen in an explicit unlisted set, so a new backend provider cannot silently miss
the dropdown again.
Claude-Session: https://claude.ai/code/session_011Tn3657NkV6ojLqewL64Kb
* fix(guardrails): accept on_violation block and alert for mcp_security
The MCP Security policy template sends on_violation: "block", but the shared
LitellmParams model only allowed the /v1/realtime values "warn" and
"end_session", so POST /guardrails returned 422 before the MCP guardrail was
initialized. Widen the literal to include the MCP actions, map every non-alert
value to MCP's default "block" at init, and regenerate the lazy OpenAPI
snapshot and dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): restrict on_violation block/alert to mcp_security and keep legacy MCP mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): return 422 when PATCH sets an mcp_security-only on_violation on another guardrail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep provider response headers on streaming chat completions
The Router re-wraps a deployment's CustomStreamWrapper in FallbackStreamWrapper
(and its sync twin) so a mid-stream failure can fail over. Neither wrapper
forwarded `_response_headers`, so every streaming chat completion handed the
proxy's callbacks and its response-header builder a wrapper with no provider
headers, and a successful mid-stream fallback still published the failed
deployment's identity, `x-request-id` and rate limit counters.
Forward `_response_headers` into both wrappers, repoint the wrapper at the
deployment that served the stream once a fallback takes over, and rebuild the
proxy's response headers from that deployment while `create_response` still has
the first chunk buffered.
* fix(router): follow a nested fallback to the deployment that served the stream
A fallback the router picks is itself a fallback-aware wrapper, and it only
repoints at its own fallback once it yields, so reading its hidden params at
selection time named a deployment that produced no output. Re-read them when
the first fallback item arrives, which is still before the proxy commits
response headers.
Also addresses review feedback: the streaming header builder reads self.data
instead of taking a coarse request_data parameter, and the new test recorder
local is Final.
* test(router): cover the fallback header adoption helper directly
The router_code_coverage gate wants every router.py function named in a
router test, and this also pins the weak-reference behavior: a wrapper
collected mid-stream must not break the generator still draining it.
* refactor(proxy): take a read-only mapping for the model-id lookup
_get_model_id_from_response only reads its request payload, so a Mapping
says what it needs and the two metadata hops are narrowed instead of
assumed to be dicts.
* test: drop mutable recorder locals and routine comments from the new tests
An AsyncMock await_count and an asyncio.Event say the same thing as a
list and a dict that the test mutates.
* chore(router): justify the two rebinds in the fallback loops
Both are the one-shot re-read that follows a nested fallback, so they get
the repo's rebind-ok note like the rest of the file.
* feat(skills): semantic search over the LiteLLM-hosted skill registry
Adds GET /v1/skills?query= (custom_llm_provider=litellm_proxy) and a
skill_search MCP virtual tool, ranking the caller's accessible skills by
semantic similarity, mirroring the A2A agent registry search (LIT-6309).
Also fixes a pre-existing bug where create_skill() dropped description and
instructions for the litellm_proxy provider, which left every LiteLLM-hosted
skill with no searchable text.
* fix(mcp): coerce skill_search top_k instead of raising 500 on malformed input
The MCP-REST skill_search dispatch validated raw tool arguments through a
pydantic model directly, so a non-numeric top_k raised a ValidationError
that the endpoint's catch-all turned into an HTTP 500. Mirrors the
agent_search branch's tolerant coerce_top_k handling instead.
* fix(skills): enforce key limits on search embeddings and bound the semantic index
Semantic search embeddings now run the same pre_call_hook the /embeddings
route runs, so key rate limits, budgets and guardrails apply before the
embedding model is called. The shared SemanticTextIndex caps cached vectors
and evicts the least recently searched entries, and each skill's embedded
text is capped so one skill cannot inflate the embedding batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): surface proxy 429s from search embeddings instead of a 503
ProxyRateLimitError is also an OpenAIError, so the search engine was folding
a key rate limit into skill_search_unavailable. Proxy HTTPExceptions now
propagate so the caller gets the same 429 the /embeddings route returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): import assert_never from typing_extensions for Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(skills): embed the request as the pre-call hooks returned it, not the original text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(skills): keep the litellm_proxy provider check for GET /v1/skills?query= inside llms/
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(skills): move the GET /v1/skills?query= endpoint tests under tests/test_litellm/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add key-scoped auto-router usage tab
GET /auto_router/benchmarks takes an optional api_key filter, applied in the
rollup aggregate on the primary key's leading column. Proxy admins get a
separate Auto-router usage tab on key detail pages with spend, baseline,
savings, tier routing, cache metrics and the existing router selector
* fix(ui): share key analytics date range
* feat(proxy): resolve root_path per request from SERVER_ROOT_PATHS
One deployment can encode exactly one client-visible URL path prefix
today: SERVER_ROOT_PATH is a scalar stamped onto the app at startup, so
a pod fronting several ingress prefixes 404s every prefix but one before
any handler runs, and MCP OAuth discovery can emit only one prefix's
URLs (RFC 9728 section 3 exact-match fails for the rest).
Add an opt-in outermost ASGI middleware that matches the request path
against a configured prefix list (SERVER_ROOT_PATHS, comma-separated) on
a segment boundary and sets scope["root_path"] for that request only.
Everything downstream is stock Starlette: route matching strips
root_path so routes stay registered root-relative, and request.base_url
re-includes it, so the discovery documents' resource and the 401
challenges' resource_metadata land under the prefix the client actually
called — with no discovery-builder changes.
LazyFeatureMiddleware now strips the scope root_path (falling back to
the cached SERVER_ROOT_PATH scalar) before feature prefix matching, so
lazily-registered routers — the MCP OAuth discovery router among them —
load under per-request prefixes.
Follow-up to the routing discussion on #35226; composes with, but does
not depend on, #35576.
* fix(proxy): import Sequence from collections.abc (ruff UP035 strict-budget gate)
* review(greptile): trim implementation commentary; fixture-own MCP registry state in tests
Addresses both P2s from the first Greptile pass:
- per_request_root_path_middleware.py (and the related _lazy_features /
proxy_server comments) cut down to the constraints the code cannot
express, per repo comment guidance
- the new discovery tests no longer clear/repopulate the shared MCP
registry inline; a fixture snapshots it, hands the test an empty
registry, and restores it afterwards so no state leaks between cases
* fix(lint): mutable-ok marker on the prefix accumulator (LIT002 type-discipline gate)
* fix(proxy): tie 401 challenges and get_custom_url to the per-request root_path
The per-request root_path middleware sets scope["root_path"] to the
prefix the client actually called, but the OAuth 401 challenges
(raise_user_oauth_challenge / raise_token_exchange_challenge) still
built their resource_metadata from SERVER_ROOT_PATH. On a pod fronting
several prefixes, the challenge advertised a discovery URL under a
different prefix than the discovery document served — the two
disagreed on where the resource metadata lives, and a strict RFC 9728
client refused the challenge. Route the challenges through a small
ContextVar the middleware populates so they read the same effective
root_path Starlette resolves the request under.
The same accessor fixes get_custom_url: when a request lives under a
SERVER_ROOT_PATHS-matched prefix, request.base_url already carries it,
so appending the SERVER_ROOT_PATH scalar on top produced e.g.
/tenant-a/legacy/sso/callback — a path that does not exist. Reading
the per-request prefix instead (and relying on join_paths's tail-dedup)
keeps SSO login/callback URLs under one prefix — the one the request
actually arrived on.
Fallback: outside a request (module-load-time UI URL builders,
background tasks) the ContextVar is unset and the accessor reads
SERVER_ROOT_PATH, matching get_server_root_path() so scalar-only
deployments are byte-identical.
* fix(mcp): challenge URL under per-request prefix must route, and mock parity
Two follow-ups to the review fix that made the 401 challenge use the
per-request root_path:
1. oauth_protected_resource_path must pick the URL structure that
actually routes for the mechanism in use:
- The scalar SERVER_ROOT_PATH deployment registers the well-known
routes with the prefix INSERTED (via well_known_root_suffix at
import time), matching RFC 8414 §3. The challenge URL must use the
same insertion or a client fetching it 404s.
- The per-request SERVER_ROOT_PATHS deployment can't register routes
per prefix; PerRequestRootPathMiddleware strips the prefix from
scope["path"] and the router matches the un-inserted route. The
URL must place the prefix BEFORE .well-known so the strip leaves a
matching path.
The previous fix used the insertion form for both, which 404'd the
discovery fetch on the per-request path — the discovery doc and the
challenge would then disagree on where the resource metadata lives,
the very failure the review flagged. End-to-end verified: the URL
the challenge advertises routes and the doc's `resource` field
equals the URL the client originally called (RFC 9728 §3).
2. get_request_root_path now delegates its fallback through
get_server_root_path() instead of reading the env directly, so every
existing `monkeypatch.setattr("litellm.proxy.utils.get_server_root_path"`
test override keeps working. This unstubbed the mock on the /v2/login
test that failed on the last CI run.
Plus the lint budget: annotate the local accumulator Final, tag the
scope["root_path"] rewrite as an intentional ASGI-contract mutation,
tag the reused `path`/`root_path` rebinds in LazyFeatureMiddleware, and
add reason strings to the two new PLC0415 lazy-import noqas.
* test(mcp): pin the reviewer's expected end-state — challenge URL routes, resource matches called URL
End-to-end regression test that mounts the discoverable router + the
per-request root_path middleware, hits an MCP endpoint that raises
raise_user_oauth_challenge, fetches the resource_metadata URL the
challenge advertises, and checks the returned document's `resource`
equals the URL the client originally called (RFC 9728 §3 exact match).
Covers /tenant-a, /tenant-b, and the unprefixed path on the same app so
a regression on any prefix — challenge URL 404s, or doc emits a
different prefix than the client called — fails at this test rather
than in a strict MCP client's discovery.
---------
Co-authored-by: gym-cmd <186399764+gym-cmd@users.noreply.github.com>
* fix(spend_logs): keep partition DDL transactions alive for their statement timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format changed files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): avoid dict-literal kwargs and keep cast-ok on the cast line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): cast at the call site instead of widening PrismaClient.tx
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend_logs): require partition tx timeout to strictly exceed statement bound
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): apply key and team guardrails to MCP tool calls
Guardrails attached to a virtual key or team were only enforced on LLM
routes. The synthetic request built for MCP tool call guardrail hooks
carried no guardrails in its metadata, so a guardrail with default_on
false never ran on tools/call even when the key explicitly listed it.
Resolve key, team, and project guardrails onto the synthetic request
with the same helper the chat path uses.
* fix(mcp): pass project metadata through without a mutable default
* fix(mcp): mark the request dict parameter mutable-ok with a reason
* test(mcp): explain the premium_user patch and tighten the helper docstring
SPEND_LOGS_URL only diverts spend logs when db_writer_client is set, and nothing in the proxy ever assigns that global, so the queued copy was only ever skipped as a duplicate by the local insert.
disable_spend_logs has to keep meaning that no request gets logged, and the row
that makes a batch chargeable exactly once is the one row it cannot drop, so with
logging off that row now carries only what tells the retrieves apart. SPEND_LOGS_URL
deployments get their copy back too: the claim writes straight to this table, so the
row is queued as well when an external writer is the one that takes the spend logs.
The takeover of a $0 row an older proxy left behind used to charge the batch when
the update could not reach the database. That leaves the row still reading $0, so
every later retrieve finds the same row and charges the batch again, which is the
repeat charging this PR exists to stop. The retrieve that does take the row over
is the one that charges, and a batch nobody retrieves again after that failure is
never charged, the same as one whose proxy died inside the write window.
A proxy without this fix wrote the batch's cost row on every poll while the batch
was still running, so that row reads $0 and the insert that claims the charge has
nowhere to land. The retrieve that charges the batch now writes its own payload
over that row under a where clause that still names spend 0.0, so exactly one
retrieve takes it over and every later one reads the charge and charges nothing
A proxy running the old code wrote <batch id>_batch_cost at $0 every time it polled a batch that was still running, so after an upgrade the claim found that row and read it as proof the batch had already been charged. Only a row that recorded a charge counts now, which leaves those $0 rows, and any row a client planted under the batch id, to be charged over
disable_spend_logs skipped the claim entirely, so under that setting every retrieve of a finished batch charged again. The claim now runs either way and writes the one row per batch that makes the charge exactly once, while the per-request logs stay off
The AzureAIStudioConfig.map_openai_params override now carries dict[str, object]
annotations instead of bare dict, and the docstrings added to the new tests go away
since the test names already say what they cover. No behavior change
Foundry deployments of gpt-6-astra reached through azure_ai used the bare OpenAI card
for the reasoning_effort none gates, so temperature and top_p were refused while the
azure_ai card says none is supported. AzureAIStudioConfig now dispatches gpt-5 series
params through AzureAIGPT5Config, which looks capabilities up under the azure_ai/
prefix the way the azure route does
Also carries the search_context_cost_per_query block azure/gpt-6-astra has, adds a
flex service tier cost test that fails at the merge base, and keeps the wildcard test
from stripping azure_ai/gpt-6-astra out of the provider set
The cost callback used to look for an existing `<batch id>_batch_cost` row before charging a
completed batch, which left a window where concurrent retrieves on any instance all charged the
key, and it would honor a row any request had written under that id. The spend update writer now
inserts the batch cost row itself with `create_many(skip_duplicates=True)` and only the retrieve
whose insert lands charges the key, team, and user. An existing row only takes the charge when it
is a successful `aretrieve_batch` row, so a client-chosen `x-litellm-call-id` on another endpoint
cannot suppress billing. Batch cost rows no longer get their own immediate flush path
`batch_cost_is_final` now treats the proxy's normalized `complete` status like `completed`, which
the enterprise batch cost poller relies on when it decides whether a completed batch is safe to
retire. Tests build that status with `model_copy` since the OpenAI `Batch` model rejects it
The `test-quality-ok` markers sit on the `patch(` lines the gate keys on, and the logging tests no
longer wrap the priced retrieve in `contextlib.suppress`
The retrieve tool was injected whenever any hash=<24hex> string appeared in the
restored conversation, including protected rows and caller-authored text, so a
git SHA in a tool result registered a bogus hash and billed a useless retrieval
round trip on every later turn. The compression service reports the hashes it
actually stored in ccr_hashes; that field is now the only source, validated to
the service's own 12 to 24 hex grammar before it reaches the retrieve URL.
Assistant rows are no longer flattened to strings before compression: the
service protects assistant text blocks but has no gate for assistant strings,
so the model's own earlier tables came back as a schema line plus CSV.
Adds ccr_retrieval (default true) so operators on a marker-free sidecar can
turn the retrieval loop off entirely.
* fix(guardrails): don't inspect embeddings in the AIM and Cato hooks
`pre_call_hook` fires for /embeddings as well as chat. An embeddings body
carries `input` — documents being indexed, not a prompt — which
`build_inspection_messages` lifts into synthetic chat messages, so both hooks
inspect it as a conversation and a policy verdict on that text breaks a request
that was never one:
- AIM, anonymize + batched `input`: `has_non_string_content` is true for any
list, so `_anonymize_request` raises 400 "...multimodal input...".
- AIM, anonymize + single-string `input`: no error — the input is rewritten to
redacted text and the caller embeds text it never sent.
- AIM and Cato, block: the embeddings request is blocked outright.
Gate both hooks on a new `NON_CONVERSATIONAL_CALL_TYPES` deny-list. This is
deliberately not `TEXT_CONTENT_CALL_TYPES`: that allow-list omits
`anthropic_messages`, `responses` and `call_mcp_tool`, so gating on it would
stop these guardrails inspecting real chat traffic. An unrecognised or newly
added call type is still inspected.
* feat(guardrails): add inspect_embeddings toggle for AIM and Cato
* fix(guardrails): redact batched embedding input on anonymize
A list of plain strings is the /embeddings batch shape. AIM rejected it as
multimodal and Cato forwarded the original strings, so anonymize never
reached the provider for batched input. Redactions are now written back
element-wise, one redacted message per non-empty element, so a fully
redacted element cannot shift the following documents into the wrong slot.
* fix(guardrails): reject partial embedding redactions
* fix(guardrails): avoid unnecessary batch type check
* style(tests): drop trailing blank line in cato guardrail tests
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject malformed batch redactions
* fix(guardrails): reject aim redactions with no text content
The anonymize path read role and content off every entry of the vendor's
redacted_chat before the shared write-back helper could refuse the payload,
so a message missing content, or a bare string in place of a message, raised
out of the hook as a 500. Validate the vendor list first and return the 400
the guardrail already uses for an unusable redaction.
* fix(guardrails): validate all aim redaction paths
Validate AIM redaction containers before request or output rewrites, reject
cardinality mismatches and empty output, and cover malformed vendor payloads
with regression tests.
* fix(guardrails): preserve aim output redaction alignment
AIM returns the inspected request messages followed by the assistant output.
Validate that full response and select the final redacted message instead of
requiring a single entry.
* test(guardrails): cover aim output anonymize alignment and malformed redactions
---------
Co-authored-by: Guy Levi <guy.levi@catonetworks.com>
A gpt-6-astra deployment on a Foundry project reached through the
azure_ai route had no cost map entry of its own, so it resolved to the
OpenAI gpt-6-astra card: missing from the azure_ai/* wildcard listing,
flex and priority prices and /v1/batch it does not sell, and no none
reasoning effort. Add azure_ai/gpt-6-astra mirroring the
azure/gpt-6-astra Standard Global sheet the way azure_ai/gpt-5.5 mirrors
azure/gpt-5.5, and extend the cost, reasoning-effort, and wildcard
listing tests to the Foundry route.
The sidebar and header were still keyed on legacy ?page= ids and mapped
back and forth through MIGRATED_PAGES, legacyPageHref and
legacyKeyForPathname. Leaves are now plain Next links to their path
route, the active item and breadcrumb come from usePathname, and the
setPage/defaultSelectedKey prop chain is gone.
The id-to-route table moves next to the dashboard root page as its only
consumer. That redirect now forwards the remaining query params instead
of dropping them, so deep links such as the proxy's MCP env-var setup
link (?page=mcp-servers&fill_env_vars=) no longer rely on the target page
reading the pre-redirect URL during its first render. The proxy builds
that link as /ui/mcp-servers?fill_env_vars= directly, and the Playground
warnings link to the real routes instead of relative ?page= URLs.
migratedHref is renamed uiHref, the /ui base-path helper it always was.
Every retrieve of a batch through the proxy shares one spend row, the batch id
plus the batch cost suffix, and spend log inserts skip duplicates. A poll that
landed while the batch was still validating or in progress wrote that row at
$0 and no later retrieve could overwrite it, and every completed retrieve after
the first added the cost to the key, team, and user counters again with no new
row to show for it.
The cost callback now writes nothing for a batch retrieve until the batch is
final, releasing the poll's budget reservation instead, and once it is final it
charges only when no spend row for that batch is queued for flush or already
stored. Batch cost rows are flushed to the database right away so a second
instance sees them, and the logger prices a batch only once it is final, which
also covers a failed batch that never produced an output file.
The regression test's recording logger overrode async_post_call_failure_hook
with untyped parameters. It now mirrors the base signature, and the
UserAPIKeyAuth import moves to module level so the annotation resolves.
A failed pass-through call logged the httpx traceback, whose message
quotes the upstream URL with the provider API key in its query string,
into the spend log's error information and into every failure callback.
The error information built for logging now redacts its traceback and
error message, and the traceback is redacted once before the failure
callbacks receive it.
The test-quality gate counts every patch of a litellm internal against a ceiling, and the three patches this test needs pushed it over. The callback imports increment_spend_counters, update_cache and proxy_logging_obj from proxy_server inside its own body, so there is no seam to inject fakes through; every other test in this file uses the same three patches for the same reason
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
The proxy cost callback zeroed response_cost whenever cache_hit was true. That rule dates from Jan 2024 when it was the only place cache hits were priced. The logging layer has priced the LLM share at 0 on a cache hit since Aug 2024, and since guardrail cost joined the standard logging payload the proxy-side zeroing has thrown away a real provider charge: a pre_call guardrail runs before the cache is consulted, so a cached response still cost whatever the guardrail billed. Drop the redundant zeroing so the payload's response_cost, which is already LLM 0 + guardrail cost, reaches spend logs, daily tables and budgets untouched
Claude-Session: https://claude.ai/code/session_01EX13mWex6RaBo9PYnkAtFW
