When a post-call hook raises after a stream's success log went out, the
failure row lifted no usage and fell back to the success's token counts, so
the daily usage tables counted the stream's tokens twice. Lift a zero usage
and zero cost while the stream success bills the current attempt.
* feat(videos): add first-class OpenRouter video request fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): remove recursive image parameter fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve EdenAI native parameter passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): deduplicate frame roles in linear time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve image selection and dropped parameter semantics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): preserve provider scalar parameters and raw extra body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(videos): retain accepted scalar inputs for Vertex and DashScope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
#40154 made the deployment post_call hook really run apply_guardrail guardrails, so the proxy
checked the same response a second time (LIT-10481). The deployment hook now records guardrail +
response hash on the request and on the response, and the proxy skips a guardrail only on a
matching record. The hash covers every field except ids, usage, model and other metadata set
between layers, so a response rewritten between the layers (OCR included) is still checked.
Guardrails with their own post_call hook keep their pre-#40154 behavior
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(usage): tolerate entity metadata database failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(usage): keep spend when metadata lookups hit database failures
Treat transport errors and the query engine crash on a closed connection as database
failures in the key, user and guardrail lookups of the aggregated, keys, users,
model_top_keys and cache_leakage_keys routes, and keep prisma out of module-level imports
so a proxy install without prisma still imports the usage endpoints.
* fix(usage): fall back only on database outages, on every daily activity route
Data-layer Prisma errors such as a rejected query or a missing table return 500 again, a key whose details failed to load reports key_exists null instead of false, and the non-aggregated daily activity routes keep spend when the key or entity metadata lookup hits an outage
* fix(proxy): skip the key metadata lookup when the daily rows carry no keys
* fix(usage): keep the guardrail breakdown fallback for any Prisma error
A missing guardrail usage table or column made the Usage call return 500 after the outage-only narrowing. The guardrail lookup now falls back on outages and on any Prisma error, as it did before this PR
* fix(usage): keep spend when the team, organization, customer, or agent name lookup drops
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
* feat(proxy): serve Codex its own release's model catalog, fetched from GitHub
When Codex asks /v1/models for its catalog, the proxy now reads models.json and prompt.md at that
release's tag on GitHub and serves those entries, falling back to the bundled 0.166.0 copy when the
fetch fails, times out after 2 seconds, does not validate, the version has no tag, or
LITELLM_OFFLINE_MODE is on. A failed fetch is remembered per version for an hour and only the eight
most recent versions are kept. The integration tests ask as the bundled release so their expectations
hold whether or not the proxy can reach GitHub
* fix(proxy): bound the GitHub fetches a caller can trigger through Codex catalog versions
Failed versions now live in their own cache so they never push out a fetched catalog, requests for a
version already being fetched share that fetch, each worker starts at most six fetches a minute and
serves the bundled catalog past that, and the version check only accepts ASCII digits
* fix(proxy): keep a failed Codex version's cooldown past the fetched catalog cap
Failures are tiny, so they get their own cap of 256 instead of sharing the eight-catalog cap, which let nine
failing versions retry each other well before their hour was up. The TypeVar takes Final like the rest of the
codebase
* refactor(typing): clear basedpyright Any errors (wave 1)
* refactor(typing): clear basedpyright Any errors (wave 2)
* refactor(typing): clear basedpyright Any errors (wave 3)
* refactor(typing): clear basedpyright Any errors (wave 4)
* refactor(typing): clear basedpyright Any errors (wave 5)
* refactor(typing): clear basedpyright Any errors (wave 6)
* fix(typing): keep the cached-plan fallback's direct query_first calls so span naming resolves
* fix(logging): keep callbacks running when litellm_params is a params model
should_run_callback and the sync success and failure handlers read litellm_params through any object with a get method again, as they did before this branch, so a logging hook that returns a GenericLiteLLMParams no longer stops sync logging for that request
Adds behavior tests for the changed lines codecov listed as not run
* fix: read the regional uplift for every data residency again and reuse one dict TypeAdapter
The typed lookup only read the eu and us multipliers, so a deployment that set regional_processing_uplift_multiplier_fedramp stopped applying it. /model/update and the model write paths now share one module-level TypeAdapter instead of building one per call.
* fix: bill regional uplift multipliers float() accepts, bytes included
* fix: raise the in operator's own error from one shared json_contains, and name the bad uplift type
* feat(router): add Switchyard Stage Router strategy
* feat(router): decision-model classifier and safer fallbacks for the Switchyard stage router
Moves to nemo-switchyard 0.4.0a1, which adds decision-model classifiers and makes
ModelCall.respond and fail async. The classifier is now a tagged union of an LLM
judge and a decision model, and decision calls go through the router's adecisions
so any decisions provider works. Algorithm misbehavior and library errors route to
the capable model instead of failing the request, and the session id plus client
session headers are always passed so capable holds apply without a classifier
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): keep the Switchyard router inside the type-check gate
The gate's environment did not install the optional switchyard extra, so every
libsy symbol counted as Unknown. nemo-switchyard joins proxy-dev like the other
optional runtime packages, and imports go through switchyard_rust.libsy, which
ships py.typed. Step handling moves into helpers so no Final is rebound in the
loop, and the dashboard API types are regenerated for the new routing fields
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): route Switchyard on every endpoint and reject unsafe router saves
The router now passes each request's call type to routing strategies, and
Switchyard converts the request to chat messages with the existing converter for
that call type before translating it. /v1/messages turns with tool or thinking
blocks used to fall back to the capable model, and now route on their signals.
The Anthropic converter can keep tool_result.is_error on an analysis copy, so
Switchyard sees tool failures there, while requests sent to models never carry it
Saves of a strategy router that lists itself now fail with a 400 instead of a
500 with a dead row, the capable and efficient models must differ, judge and
decision calls are not retried before failing open, the Switchyard routing
fields are classified for redaction, and the edit dialog returns the saved
router instead of the stale one
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): catch self-routing on patch and rename, and trim Switchyard tests
The self-routing check now reads the stored params through the decrypting
helper and runs when a patch carries no litellm_params, so a PATCH that points
a router at itself or renames it onto one of its targets returns 400 instead of
a 500 with a dead row. The Switchyard tests share one transcript builder and
router fixture, and reformatting noise in unrelated test files is reverted
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(ui): make Switchyard a strategy in the existing auto router form
Switchyard is now a card in the existing strategy picker. Picking it swaps the
tier and classifier fields for the Switchyard fields, and the shared name, team
and save button create it through one submit branch, the same flow as the other
strategies. The separate add form, the extracted picker and the hidden-form
swap are gone, and the edit dialog keeps its own Switchyard form
The judge's session_affinity and message_hash_fallback settings are removed,
since Switchyard rejects both for the stage router
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): accept partial tool semantics and show server errors on model edit
The Switchyard tool semantics field rejected mappings that list only some
categories, although the server accepts them. Model PATCH errors were
reduced to a generic message, so it now goes through the shared API client
like model create and surfaces the server text
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): show the routing strategy picker up front in the auto router form
Every strategy was hidden behind an Advanced configuration link, so the
quick setup read as the only option and never said it creates a heuristic
router. The picker now always shows with Heuristics as the labelled
default, the quick setup sits under it, and picking another strategy opens
that strategy prefilled with the chosen models
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(ui): show the server's reason when a model settings or API key update fails
Both screens replaced the rejection with a fixed message, so a 400 such as
a router routing to itself read only as a generic failure
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(router): route Switchyard on the Rust router core
main replaced the Python router core with the Rust one, which left
Switchyard routers registered nowhere. Register them with the other
pre-routing strategies, hand each strategy the endpoint the request came
through, and evict Switchyard routers on reload and delete like the rest
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): scope Switchyard sessions per API key and fix review gaps in the Switchyard form
Switchyard session state is keyed by a hash of the caller's key hash and session id, and client
session headers no longer reach Switchyard unscoped. The add form carries the quick setup models
into Switchyard, sends the team and access groups, clears a stale tool semantics error, and the
schema reads API-saved configs with nulls and structured decision instructions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(router): make the Switchyard test folder a package
* fix(router): read the Switchyard session key hash only from the proxy's own metadata
* fix(proxy): keep nested nulls as null when masking deployment params
* fix(router): send Switchyard judge calls with the caller's own metadata bucket so tag routing applies
* fix(router): put Switchyard judge metadata in the bucket the judge endpoint reads
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): mixed Codex model picker and LiteLLM routing on native codex routes
The native Codex catalog now lists the key's other chat and responses models after
ChatGPT's, within Codex's catalog size limit, and drops the ETag so the merged list
refreshes on Codex's timer. A request for a model configured by exact name on another
provider goes through /v1/responses without the ChatGPT bearer or account header,
while ChatGPT names and wildcard-only matches keep the subscription passthrough.
Reasoning items another provider produced are dropped before forwarding to ChatGPT.
* fix(proxy): keep Codex's picker from turning reasoning off for LiteLLM models
Codex fills a missing default_reasoning_level with effort "none" when a model is picked in /model, while
`codex -m <model>` sends no effort at all. A thinking-only provider model then answers 400. On the native
/backend-api/codex/responses route, drop effort "none" for a LiteLLM model whose catalog entry names no
default, so the provider applies its own default exactly as `codex -m` does
* fix(proxy): harden Codex mixed-model routing on the native ChatGPT routes
Keep the ChatGPT credentials out of hook raw headers on the inner /v1/responses
request, route to LiteLLM and merge LiteLLM catalog rows only for keys that may
call /v1/responses, drop unpersisted reasoning ids on the passthrough when store
is not true, keep any slug the caller's ChatGPT account lists on the subscription
via a shared account catalog cache, and keep X-Models-Etag for keys whose catalog
added no LiteLLM rows. The guide covers the new behavior and the /v1 wildcard
precedence over a catch-all deployment.
* fix(proxy): keep Codex's native route on the subscription when the account list is unknown
A name OpenAI lists for the caller's ChatGPT account now goes to LiteLLM only when the
account's list is known and leaves it out. A failed catalog read is remembered for 60 s and
the Codex version comes only from a User-Agent whose product matches the originator header.
The merged catalog serves a stable ETag digest of OpenAI's ETag and the key's LiteLLM rows,
and /responses replaces X-Models-Etag with the same value, memoized per key for 300 s, so
Codex stops refetching or dropping its catalog cache.
LiteLLM-routed native replies wrap their reasoning with the model id, so an OpenAI or Azure
blob is dropped before chatgpt.com and unwrapped for LiteLLM, and the chat-completions bridge
decodes wrapped thinking blocks with their signature.
The access-group recipe uses chatgpt/ model names so /v1 routing does not change, and the
native route checks the chatgpt/ form of a name the key holds only through that group. A
none reasoning effort is dropped only under drop_params. The guide covers app tools, every
routing case, and the ETag, and the catalog logs how many models the byte limit left out.
* fix(fireworks_ai): move Codex additional_tools items into tools on the Responses API
* fix(proxy): keep Codex native turns working when the key's model listing fails
A listing callback that raises no longer turns chatgpt.com's catalog or
responses answer into a 500. The catalog keeps upstream's body and ETag, and
responses keep upstream's X-Models-Etag.
Under drop_params, a `none` reasoning effort is now dropped only when a
candidate deployment does not explicitly support it, so an alias to a model
like openai/gpt-5.4 keeps it.
The account slug list and its unavailable marker are keyed on the account id
plus the caller's ChatGPT Authorization header, so a failed fetch with one
bearer cannot turn off LiteLLM routing for another caller on the same account.
The auth step checks the Codex route before importing the Codex module.
* fix(proxy): keep Codex model switches working across LiteLLM providers
On the Codex native route, a LiteLLM model now gets only the encrypted reasoning every deployment it can route to is able to decrypt, so ChatGPT or another deployment's blobs no longer make xAI answer 400
Codex additional_tools items are hoisted into tools for every provider except the OpenAI API, and namespace tools become flat tools named mcp__docs__search, the shape xAI accepts
A JWT caller's models ETag is cached on a hash of its LiteLLM credential, so ChatGPT turns stop rebuilding the catalog rows
* fix(proxy): keep Codex namespace tool calls intact across LiteLLM providers
Fireworks and Bedrock Mantle accept Codex namespace tools natively, so the
native route now hoists them without flattening, like OpenAI and Azure and
the chat completions bridge. Other native Responses providers still get
flat names, now built with the bridge's namespace_tool_name_map, and each
function_call they return gets its tool name and namespace back in the
non-streaming body and in streamed output_item.added, output_item.done,
and completed/incomplete/failed events. Every other event passes through
unchanged
* fix(proxy): bridge Codex turns for providers whose Responses API rejects Codex tools
Codex sends its freeform exec tool as a custom tool on every turn, and xAI's
Responses API answers 422 for custom, namespace and additional_tools shapes.
On the Codex native route, a model whose deployments include a native
Responses provider other than OpenAI, Azure, Fireworks or Bedrock Mantle now
asks for the chat completions bridge with use_chat_completions_api, which
already turns custom and namespace tools into functions, hoists
additional_tools, and puts the namespace back on each returned call
OpenAI and Azure still get the request unchanged, and Fireworks and Bedrock
Mantle still get it on their own Responses API with additional_tools hoisted.
The route's namespace flattening and the response-side namespace restore are
removed, since no path sends flattened names anymore
* fix(proxy): drop reasoning ids a Codex turn's LiteLLM target cannot resolve
A reasoning item another deployment produced, replayed with store false and no blob the
target can decrypt, now reaches the target as its readable summary without id or status,
and is dropped when nothing readable is left. OpenAI answered 400 on the status field and
404 on the id for a glm-5p3 item replayed whole to an openai deployment. Same-deployment
replays and stored conversations keep the item whole
* refactor(codex): reach encrypted reasoning stripping through the router registry
The Codex route read Router.optional_callbacks and imported EncryptedContentAffinityCheck from
litellm.router_utils, which the router surface fence forbids outside the router. RouterRegistry
gains decryptable_reasoning(model_group, request_input, targets), which keeps the same rule: with
encrypted content affinity on for the group, reasoning one target can decrypt stays, otherwise
every target must decrypt it
* fix(proxy): keep Codex's priority tier off LiteLLM rows in the native model catalog
* fix(proxy): drop Codex's none effort for an alias whose native entry names no default
* fix(proxy): start a native Codex alias at its pinned reasoning effort and build the catalog off the event loop
* test(proxy): map the pinned reasoning efforts field in the router registry stub
* fix(proxy): honor admin-only /v1/responses and keep Codex's default tier off native aliases
Codex native routes now send a key to LiteLLM only when admin_only_routes allows it,
a native alias with configured service tiers starts with no tier picked, and the
account slug cache declares its Redis key family.
* fix(proxy): check the key's tool allowlist on native Codex LiteLLM turns and additional_tools items
* fix(proxy): check vector store grants on file_search tools inside additional_tools items
A Responses request can define file_search inside an additional_tools input item, which the native Codex route hoists for LiteLLM-served models and OpenAI runs as is on /v1/responses. The vector store check now reads those tools in both strict and legacy modes. The input item scan uses a Mapping guard instead of a pydantic error per item, since auth now runs it on every list input.
* fix(proxy): apply the proxy-wide allowed_routes to LiteLLM models on native Codex routes
* fix(proxy): cap native Codex account catalog reads per LiteLLM caller before rate limits apply
* fix(guardrails): check additional_tools items next to top-level function tools in tool_policy
ToolPolicyGuardrail read tools from the request body only when no top-level
function tool existed, so a blocked tool defined in an additional_tools item
reached Fireworks once the Responses transform moved the item into tools after
the pre-call hooks. Check every list on each request, including namespace
members inside items.
* fix(router): serve decryptable_reasoning from the Rust core's optional checks
The merge with main removed the Python router core, so the registry's host has no
optional_callbacks and no routing context, and native Codex requests with encrypted
reasoning raised RustCoreUnsupported. The filter now lives with the optional checks,
which own both. The ported pinned_reasoning_efforts field gets a test on the Rust core
Also give the static litellm.Router a declared constructor. Since the Python core was
deleted a bare __init__ assignment reads as RustCore's, so every Router(model_list=...)
is a type error and make lint fails on main. TYPE_CHECKING only, no runtime change
* fix(proxy): check the budget of a saved alias target on native Codex LiteLLM turns
Native /backend-api/codex/responses binds {endpoint}, so auth skipped the key's or team's model_group_alias and read budgets on the requested name, while the later /v1/responses handoff served the target unchecked. Auth now resolves the alias on a turn that routes to LiteLLM, as it does on /v1, and leaves ChatGPT-only names on the subscription. Also makes the tool_policy tests build a real ToolPolicyRegistry and type their fixtures.
* fix(proxy): keep a name the ChatGPT account lists on the subscription despite a saved alias
* fix(codex): keep LiteLLM keys off ChatGPT, enforce tool policy on namespaces and pin reasoning on aliases
Auth no longer asks ChatGPT for the account's model list when the bearer is
missing or is the LiteLLM key, so a key sent as the bearer never reaches it.
tool_policy now reads the members of a top-level namespace, which is where
the native route's hoist puts them.
A global alias now reads encrypted content affinity from its target group.
A non-list top-level tools value and a non-string function name now reach the
provider, which refuses them, instead of raising a 500.
The catalog ETag no longer depends on the order a process iterates priced
service tiers in, so workers and pods agree on it.
* fix(proxy): list a team's models in the order the team set them
get_team_models built the names in a set, so each process listed a team key's models in its own hash order and the native Codex catalog sent a different row order and ETag from each worker
* fix(proxy): apply JWT route policy before native Codex routes serve LiteLLM models
A standard JWT caller now gets a LiteLLM model on /backend-api/codex/models
and /backend-api/codex/responses only when the JWT auth flow would admit a
direct /v1/responses call for that model, so team_allowed_routes,
admin_allowed_routes and RBAC route limits hold on the handoff too.
The Fireworks Responses transform also leaves additional_tools input items
in place when the tools param is not a list, instead of raising while hoisting
* fix(fireworks): hoist additional_tools when tools is a tuple
litellm.responses passes a tuple tools argument through to the Fireworks
transform unchanged, so the non-list guard skipped hoisting and left the
additional_tools item in the input. Treat tuples like lists and cover both
shapes in the drop_params merge test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(model-hub): show each deployment's full prices in the model hub
The model hub only showed one input and one output price per model, the highest across its
deployments. Each model group now lists every deployment's prices, read from the same entry
billing charges, including cached tokens, long-context tiers, flex, priority, batch and
tiered ranges. Both the logged-in and public hub detail views render them in one table per
deployment, labeled by provider only
* refactor(ui): move model hub pricing breakdown text onto i18n
Route every string the deployment pricing breakdown writes through t() under a new deploymentPricing group in en.json, plus the three hub headings that host it. The English on screen is unchanged
* fix(model-hub): show Vertex regional prices and explicit cached reads as billed
A Vertex deployment outside the global endpoint now shows its token prices with the regional
uplift billing applies at the deployment's location. The explicit cached read price shows per
1M tokens on its own row instead of a raw per-token value. The breakdown also says that
adjustments made per request, like off-peak discounts and regional or data residency
surcharges, aren't included
* fix(model-hub): show prices written as text the way billing reads them
A price written in config.yaml as 3e-7 arrives as text. Billing turns it into a number, but
the hub dropped it, so a deployment showed no price or the cost map's price instead of its own.
The hub now reads numeric text for regular prices and size-based price ranges, and skips text
that isn't a number
* style(ui): format public model hub test
* test(model-hub): name the deployment pricing fixture to stay within the inline object lint budget
* feat(bedrock): enable native structured output by default for selected Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep govcloud and bare model rows off native output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected native schemas with the JSON tool on default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep the rejection retry on the unforceable-tool native path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry rejected schemas on the Messages path for default-native models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format Messages native output retry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): allow profile-only native output metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): only disable default-native output on schema-rejection retries
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type fixture parameters in native output retry tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): retry with the JSON tool when Bedrock rejects a schema's grammar size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop strict tools after a native-output retry hits the grammar limit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format native output fallback
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): honor the deployment native output opt-out and keep wide batch schemas on the JSON tool
A deployment's supports_bedrock_native_structured_output: false now keeps the JSON tool on rows that default to native output. Bedrock batch has no retry, so a record whose schema has more than 64 properties keeps the JSON tool there too.
* fix(bedrock): retry a native structured output stream as a buffered stream when Bedrock denies the stream action
A role allowed only bedrock:InvokeModel got a 403 on schema streams once these
models went native, since native streams call converse-stream while the JSON
tool path buffered through converse. The retry now sends that request again
as a buffered JSON tool stream, the shape the base sent.
The wire test moves its neither-key and opt-in cells to Opus 4.7, its
Bedrock-key-only cells to Sonnet 5.5, and its unforceable-tool cells to the
bare Opus 5.5 id, since this PR makes Opus 5.5 and Haiku 5.5 always native.
The batch wire test now checks a thinking schema record on both sides of the
contract the batch unit test already pins, the JSON tool on opt-in Sonnet 5.5
and native output on always-native Haiku 5.5, since its Haiku 5.5 and Opus 4.8
cells assumed those models were opt-in.
* fix(bedrock): move the Claude 5 native output default to a key released proxies do not read
Released proxies load the cost map from main and have none of the native
output retries, so setting supports_native_structured_output on the Opus
5.5, Fable 5.1, Sonnet 5 and Haiku 5.5 profile rows turned working wide
or min/max schemas into 400s on v1.108.1.
Those rows keep supports_native_structured_output false and gain
bedrock_native_structured_output_by_default true, which get_model_info
reports as supports_native_structured_output, so this version behaves as
before while released versions keep the JSON tool.
* fix(bedrock): rename the Claude 5 native output default key to supports_bedrock_native_structured_output_by_default
The cost map guard runs main's model catalog on the PR's cost map, and that catalog rejects any undeclared key that does not start with supports_. Declaring the old name on main first needs its own PR, so the key takes the supports_ prefix. Released proxies still never read it
* fix(bedrock): honor registered and batch opt-outs, move Opus 5 and 4.8 to the default key
A registered supports_bedrock_native_structured_output false now beats the
native output default, and batch uploads read the deployment's model_info, so
an opted-out deployment keeps the JSON tool on every record. The Opus 5 and
Opus 4.8 profile rows move to supports_bedrock_native_structured_output_by_default,
so released proxies on main's cost map keep the JSON tool on them. The 64
property batch cutoff now applies only to rows with that key, so models that
were already native keep their batch output shape.
* fix(proxy): report a deployment's Bedrock native output opt-out on /model/info
/model/info merged the deployment's model_info over the backend row, so a
deployment that set only supports_bedrock_native_structured_output: false
still showed the backend's derived supports_native_structured_output: true.
The generic flag is now derived from the merged info unless the deployment
set it itself, through the same helper get_model_info uses.
* fix(bedrock): keep native output when a denied schema stream is resent buffered
A role without bedrock:InvokeModelWithResponseStream had its native output schema stream resent as a buffered JSON tool call. The resend now keeps the native params with fake_stream set, and falls back to the JSON tool only if Bedrock then rejects native output.
* fix(bedrock): drop rejected strict tools on the JSON tool retry after a buffered native resend
* fix(bedrock): send a buffered schema stream to /invoke and keep the by-default key on its deployment
The shared HTTP handler now builds the non-streaming URL for a fake-streamed request, so the invoke route's buffered resend after a denied stream calls /invoke instead of invoke-with-response-stream. supports_bedrock_native_structured_output_by_default is deployment scoped, so setting it on one deployment no longer reaches sibling deployments of the same model
* fix(bedrock): answer a buffered invoke stream from the non-streaming reply
* fix(bedrock): keep the JSON tool retry buffered after a denied stream's schema is rejected
* fix(bedrock): count only a true fake_stream and keep the deployment by-default key out of /model/info
A fake_stream sent as "true" or 1 picked the non-streaming URL but kept the streaming reader, so Bedrock invoke streams answered 500 after three billed calls. Only a real True counts now, as on main.
/model/info read supports_bedrock_native_structured_output_by_default from the deployment, while routing reads it only from the cost map, so it reported native output for deployments that use the JSON tool.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): do not surface server_tool_use as client tool_calls
Anthropic server-executed tools (web search, tool search, code execution)
were returned in chat completions tool_calls with srvtoolu_ ids, so
OpenAI-protocol agent clients treated them as pending work and re-invoked
the model until their recursion limit.
Move them to provider_specific_fields.server_tool_calls on both the
non-streaming and streaming paths. History replay, the /v1/responses
bridge and the /v1/messages streaming guardrail read that field, so
multi-turn reconstruction, web_search_call output items and the
guardrail's fail-closed check on server tool blocks keep working.
* fix(responses): return Claude web_fetch calls as web_search_call open_page items
* fix(responses): keep Claude server web calls without a result as hosted items
A non-streamed Claude turn that calls web_search or web_fetch next to a
client tool can return the server call before its result. Build an
in_progress web_search_call for it, so the Responses bridge no longer
returns it as a client function_call
* fix(anthropic): keep pending server calls native and hide server calls from client tool paths
Replaying a server call that has no result yet now sends it back as
server_tool_use, so Anthropic accepts the next turn. The tool permission
guardrail checks server calls again and rewrite mode removes a denied one.
The /v1/messages adapter keeps Claude's server blocks, and the Responses
bridge never returns a server call as a client function_call.
* fix(anthropic): let guardrails, tool usage, tracing, and Responses replay see Claude's server calls
Claude's server tool calls moved out of tool_calls into provider_specific_fields.server_tool_calls,
so every reader of tool_calls stopped seeing them. The chat guardrail handler now passes them to
output guardrails and writes rewrites back to server_tool_calls (stream and non-stream), tool usage
and registry discovery count them, otel, Datadog and the Logs Tools panel list them, and a pending
Claude web search or fetch item replayed to /v1/responses goes back to Anthropic as its own
server_tool_use instead of running again
* fix(anthropic): let guardrails and loggers read Claude's server tool calls again
Guardrails, tool policies, tracing loggers and the Responses guardrail
translation read only client tool calls after server calls moved to
provider_specific_fields. They now read server calls first, then client
calls, through one shared helper, and the Responses guardrail treats
Claude's web_search_call items as web_search or web_fetch calls.
Non-streaming splits server calls by block type, as streaming does.
* fix(anthropic): keep Claude server calls visible to guardrails, redaction and tracing
A guardrail rewrite of a Claude server call now also updates its hosted web_search_calls copy,
streamed or not. Redaction with message logging off hides server call arguments and hosted web
call actions on chat and Responses. Pillar and Pangea scan server calls, transform-mode streaming
guardrails keep server call chunks, and OTEL v2 Responses spans list Claude's web calls.
* fix(anthropic): keep the legacy hook view of streamed /v1/messages web search after the main merge
Main's legacy post-call hooks see Claude's server call as tool_use with its search results first. The streamed /v1/messages response now keeps that view while native scans keep server_tool_use.
* fix(guardrails): let Cisco AI Defense scan Claude's web calls on Responses
Claude's web search and fetch now come back from /v1/responses as web_search_call items, so Cisco AI Defense reads and clears their query or URL the way it did for the old function_call arguments. The helpers that read and rewrite those items move to the shared guardrail utils so the Responses guardrail handler and Cisco use one copy.
* test(guardrails): inject Pillar's HTTP client and type the Cisco web call actions
* test(guardrails): type the fake Pillar client like its Protocol
* fix(anthropic): keep a pending server call and its thinking when the turn also waits on a client tool
Only a turn with no client tool_use is a paused turn. A tool_use turn that also started web_fetch or
web_search still needs that call and its thinking replayed, or Claude loses the fetch result
* fix(guardrails): scan replayed Claude server calls and hosted web calls
Request guardrails now see and rewrite server_tool_calls replayed in chat
messages. Cato and Purview scan Claude web_search and web_fetch calls on
Responses output, and a function tool call with an extra custom key is no
longer dropped from Cisco, Purview, Repello and PANW scans.
* fix(guardrails): scan replayed Claude server tool calls in Lasso, Lakera and Qualifire request checks
* fix(anthropic): apply streamed guardrail tool call rewrites to their own block when a server tool call follows
* test: annotate the new server tool call test locals with Final and type their parameters
* test: type the new guardrail test helpers with litellm's tool call types instead of Any
* test: narrow the remaining new test helper parameters from bare dict and object
* fix(anthropic): keep server calls in Responses streams, guardrail masks, OTEL and Logs
Streamed /v1/responses now emits Claude's tool search and paused web search calls, text guardrail masks rewrite replayed server calls, OTEL gen_ai messages and the Logs pretty view list server calls next to client calls, and integration tests read server calls from server_tool_calls
* fix(guardrails): keep hosted web call copies in sync on request masks and denied server calls, and accept null tool_calls in Lakera
* fix(anthropic): serialize masked hosted web call copies through their adapter so a replay before the OpenAI model is built does not 500
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* Add first-party floating model aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Align Moonshot alias parameter test with its target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Share floating alias proxy budgets and rate limits with targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Prove target-only team budgets count and block floating aliases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Share floating alias quotas with project and organization targets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Coalesce colliding floating alias rate limit counters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Fix latest alias Rust fallback and failure accounting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: adapt latest alias checks to typed rate-limit store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): discover upstream LiteLLM Gateway wildcard models
* fix(proxy): use the typed shared client for gateway discovery
* fix(ui): show the selected LiteLLM Gateway logo
* feat(proxy): hash UI passwords with PBKDF2 in FIPS mode
A FIPS OpenSSL refuses scrypt, so every password write fails there and
nobody can sign in with a password. When OpenSSL enforces FIPS, passwords
now use PBKDF2-HMAC-SHA256 at 600k iterations with a salt from OpenSSL's
DRBG, stored as "pbkdf2:" plus base64 of salt and key. Any other row is
refused with one error log and left as it is, and the boot migration does
nothing, because FIPS mode is for new deployments.
PBKDF2 at 600k takes about five times as long as scrypt, so in FIPS mode the
five request paths run it on one thread instead of the event loop. A single
thread keeps a login burst from using up a CPU quota, which is what stalls
the loop with asyncio.to_thread. A login that queued for that thread asks the
throttle again before it answers, so a burst of guesses can't get past a block
that the failures ahead of it just set.
The breached password check is off in FIPS mode, because it runs the
password through SHA-1.
A standard OpenSSL reads False at every new branch, so standard mode keeps
scrypt, inline, on the same rows.
Resolves LIT-8425
* fix(proxy): take FIPS logins one at a time and recheck the throttle first
A login that waited for the KDF thread asked the throttle again only after
deriving, so a burst for one user kept that thread busy long after the user
was blocked. Logins now wait for their turn on a lock and ask the throttle
when it comes, so a blocked login costs no PBKDF2 and only one login at a time
holds a place in the KDF queue. Password changes and invite claims no longer
wait behind a whole burst of logins.
* fix(proxy): keep PBKDF2 rows on a standard boot and warn when FIPS skips the breach check
A standard boot treated a pbkdf2: row as plaintext and hashed its text, so the
real password stopped working and the old row text signed in. The migration now
leaves a row of that exact shape alone. Any other pbkdf2: text still migrates.
FIPS mode now logs a warning when it skips a breach check that is turned on.
The one-thread test no longer depends on timing, and the new test variables are
annotated Final.
* fix(proxy): keep the breached password check on in FIPS mode
* fix(proxy): choose the password KDF from fips_mode instead of the host OpenSSL
* test(proxy): fake the scrypt probe in the FIPS fixture now that the gate checks scrypt
* feat(redis): route the router, rate limiters and auth through the native store families
* style(redis): satisfy the python lint gates
Format the rate limit store, sort its __all__ and give the routing read batch and two tests concrete list and dict types.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(router): give the registry's default usage store a memory-only handle
RouterUsageStore now takes a handle, so the default passed None and any usage read before RustCore swapped in its store raised AttributeError.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(redis): build stores from native handles and wait for router tasks
Tests still used the pre-refactor store constructors (Store(None), handle= factories, NativeIncrement, tuple scopes). The router outcome tests counted 20 loop ticks, too few now that the allowed-fails increment resolves on the native runtime, so settle() waits for the scheduled tasks instead.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(team): let default_team_params attach access_group_ids to every new team
* fix(types): keep default access_group_ids as Sequence for the type discipline gate
* fix(settings): chunk the default access group existence lookup
* test(team): prove default access groups reach the saved team row and type the settings fixtures
* fix(team): check default access groups on the writer so a fresh group is not rejected behind a read replica
* feat(spend-logs): opt-in latest_turn mode that stores only the current turn of stateless requests
* fix(spend-logs): keep bridged Responses rows and classifier audit in latest_turn mode, reject unknown values at boot
* fix(spend-logs): drop the recursive call in the latest_turn trim and type the stateless test cases
* test(spend-logs): format the stateless latest_turn cases
* test(spend-logs): wrap the classifier audit fixture
* fix(spend-logs): type the latest_turn trim inputs
* fix(spend-logs): leave the raw request body unvalidated before the latest_turn trim
* fix(spend-logs): trim classifier rows and ignore continuation keys off /v1/responses in latest_turn mode
* fix(spend-logs): take the classifier's originating request from the logging payload and redact a copy of vector store results
* fix(spend-logs): redact the body selected for storage, after the latest_turn trim
* feat(proxy): give Codex accurate model entries for every model in /v1/models
* fix(router): read a deployment's upstream model without a not-required key access
* fix(proxy): keep Codex tiers for unpriced deployments and match stock entries by declared provider
A deployment with no token price no longer votes a Codex service tier off,
an undescribed model keeps Codex's default reasoning summary parameter,
a Bedrock deployment naming its provider in custom_llm_provider gets Codex's
stock entry, and Bedrock provider namespaces follow the AWS model id format.
* fix(proxy): import cost tracking and provider helpers lazily in the Codex catalog
lite codex imports the catalog before the SDK is initialized, so importing
cost tracking, the Bedrock helpers or provider resolution at module level hit
a circular import and broke every lite command.
* fix(proxy): drop Codex's fast speed tier wherever its priority tier is dropped
Codex shows /fast for a leftover fast speed tier even when service_tiers
no longer offers priority, so an alias, an empty configured tier list or
a configured list without priority showed a /fast toggle that sends nothing.
One helper now sets service tiers, speed tiers and the default together.
* fix(proxy): give an older Codex a translated entry for models past its version
Codex's own entries name the oldest release that can read them in minimal_client_version. A release older than that can fail to decode the entry, such as a reasoning effort it has no name for, which fails the whole catalog, so it now gets the translated entry, whose levels are trimmed to that release
* feat(proxy): let admins allow specific S3 buckets and endpoints in request bodies, block bucket settings otherwise
Client requests can no longer pick s3_bucket_name, s3_output_bucket_name or allow_legacy_cloud_file_ids unless the
value is on general_settings.allowed_client_side_s3_buckets (buckets) or the admin opted in the usual way.
s3_endpoint_url, already blocked, can now be allowed per value through general_settings.allowed_client_side_s3_endpoint_urls
* refactor(proxy): drive the client-side S3 allowlist from one typed field-to-setting map
* fix(proxy): treat a null client-side S3 allowlist in config as empty instead of failing the request
* fix(proxy): probe scrypt instead of MD5 in the fips_mode OpenSSL check
* test(proxy): fake a working scrypt so the fips_mode check tests pass on FIPS hosts
* fix(types): define the responses compact call types so during_call guardrails and deployment hooks see them
* test: regenerate dashboard api types and skip compact in the azure sdk client sweep
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(interactions): continue requires_continuation long decodes with continuation_token hops
A foreground Interactions request stops at Google's per-hop output cap with
status requires_continuation and a continuation_token. litellm now re-posts the
request with previous_interaction_id and continuation_token until the status
moves on, merging the hops into one interaction (last id, every hop's steps,
summed usage with initial_input_tokens from hop 1), for non-streaming and
streaming calls. continuation_token is accepted as a request param, and
requires_continuation is a known status for billing.
* refactor(gemini interactions): validate streaming chunks with model_validate
* fix(interactions): keep the held completion across stream hops, stop on a repeated continuation_token, follow the spec redirect
* fix(interactions): add grounding_tool_count per tool type, stop on any seen continuation_token, keep streaming past a [DONE] frame when a hop is pending
* ci(interactions): point the spec canary at aistudio.google.com, where Google moved the Interactions OpenAPI spec
* test(interactions): scripted Google ends each long decode hop with a [DONE] frame
* refactor(interactions): pass requires_continuation and continuation_token through instead of looping hops in the proxy
The caller drives the continuation, the same as against Google directly. /interactions accepts continuation_token in place of input so a resume request needs no input.
* test(proxy): assert the accepted /interactions bodies pass the required-param check
* fix(proxy): keep the router default fallback for input on /interactions, continuation_token stands in for it
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(zscaler_ai_guard): scan only the last user turn on requests
* fix(zscaler_ai_guard): keep scanning A2A requests and refuse scan_only_tool_results
* fix(zscaler_ai_guard): join the last user turn's text parts with spaces like before
* fix(zscaler_ai_guard): keep scanning tool results when scan_only_tool_results is set
* fix(zscaler_ai_guard): scan the latest user turn when assistant or tool messages follow it
* fix(zscaler_ai_guard): skip the scan when the latest user turn is only whitespace
A blank latest user turn was sent to Zscaler on its own, which answers
Content is empty, so the proxy returned 500 where main returned 200
---------
Co-authored-by: Arnav Maiti <amaiti@zscaler.com>
* fix(accounting): keep one undecodable delivery from blocking the ledger
A request whose metadata carried a 401-digit tokens_saved stored that number in
its delivery payload. The query engine could not decode it when the worker
claimed the batch, so every later delivery on the proxy stayed pending
The worker now claims the payload as text and decodes each row in Python. A row
that still cannot be decoded is parked alone while the rest of the batch
applies. The payload builder also clamps integers to the JSON-safe range, so
caller-supplied counts can no longer reach the engine at that size
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): take spend locks in one order so the ledger cannot deadlock
A ledger delivery updates many spend tables in one transaction, so it can
deadlock with any writer that locks the same rows in another order. It took the
team advisory lock after the user and team rows, while team roster writers take
it first
The delivery now takes every team advisory lock before any row, then writes
tables in one canonical order with ids sorted. It also runs under the rollup
lock_timeout, applies 500 deliveries per batch by default, and spreads retries
with jitter
The writers that could still collide with it now follow the same order. The
budget reset cascade, model allowlist rename, global spend reset, bulk user
update, org delete and the delete-by-list paths lock their rows in key order
first, and access group endpoints update teams and keys in one sorted pass
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): prove a ledger delivery waits instead of deadlocking
Real Postgres tests hold the first lock of a team roster change, an access group
update and a budget reset cascade, start a delivery, and let the other writer
finish. The delivery must wait and then charge once. Before the lock order fix
each case ends in a Postgres deadlock
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(accounting)!: make the SpendLogs ledger the default accounting path
use_spend_logs_ledger now defaults to true, so every request is charged through
LiteLLM_SpendLogDelivery. Setting it to false rolls back to the in-memory and
Redis spend queues, which stay in place. Tests that cover those legacy writers
pin the setting off, and the rest check the queued charge the ledger stores
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): let the mocked client open a transaction for key and team deletes
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): take team locks first when a model rename rewrites allowlists
A rename locked team rows before user rows while a roster change holds the team
advisory lock and a user row, so the two could deadlock. The rename now takes the
advisory lock for every team that lists the model, sorted, before any row lock.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): drain a backlog in one worker run, one batch per transaction
One run applied a single fresh batch, so at 500 per batch charges fell behind any
load above that rate. The worker now keeps claiming fresh batches while they come
back full, for up to five seconds, and each batch still commits on its own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): queue only the charge when a request's spend log is suppressed
With logging off for a request, the ledger still queued the full row, so a failed
flush or a shutdown could leave its prompt and response in Redis. The queue now
holds only the fields the charge needs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* style(proxy): format the rename team listing query
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): remove expired delivery receipts as fast as the worker applies them
The worker now applies several batches per run but still removed one cleanup batch
of applied receipts, so the table could outgrow its retention window. Cleanup now
repeats while each pass removes a full batch, for up to five seconds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): retry a model rename when a team starts listing the model mid-lock
The team advisory locks come from a snapshot, so a team that gained the old name
before its row lock was locked without its advisory lock. That attempt now commits
without writing and the next one locks the fresh set of teams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: opt-in model deprecation emails for affected owners
* fix: annotate deprecation sender initialization
* fix: clarify deprecation emails and use consistent settings toggles
* fix: exclude former team admins from deprecation emails
* feat: clarify provider deprecations and model migration steps
Adds the litellm-router-core crate, its step-driven PyO3 bridge
(RouterCoreHandle / RouterCall) and RustCore (litellm/router_rust/),
served behind LITELLM_RUST through the ROUTER route rule. The host
call-out is the only boundary: every attempt goes through Host::invoke,
there is no native executor path, and RustCore never builds, wraps or
forwards to PythonCore, so router_python_core.py stays deletable.
Attempt preparation lives in litellm/router_attempt.py so it outlives
router_python_core.py.
RustCore matches PythonCore's behavior, including its known bugs (fixed
in follow-up PRs). Routing decisions (strategy picks, routing groups,
PTU, pre-routing, pre-call checks, fallback walk, health routing, access
groups, weights, cooldowns, allowed fails, usage) live in the crate;
work that needs live Python objects (request shaping, endpoints,
streaming, model info facts, callbacks, alerting, prompt management)
lives in the host-side litellm/router_rust/ package (steps/, invoke/,
surface/) driven by Rust steps.
The crate owns the registry. It runs the deployment build loop (secret
references, ids, organization fan-out, install order) and asks the host
for what needs Python, one deployment at a time so errors surface where
Python raised them: secrets, validate, provider and added steps. It
holds the model list, the index maps and the pattern routers with the
Python core's history rules (host_index.rs), and answers the registry
lookups (lookup.rs). The Python views of that state in
litellm/router_rust/surface/ (model_list, index(), names(), patterns())
read the handle, and every write to them becomes a changeset. Model
info, deployment model info, configured limits and model group info are
merged in the crate (model_info.rs, group_info.rs) from catalog rows and
facts the host answers on a model_info step asked at query time, so
price reloads and register_model stay visible. The usage cache key
format is defined once, in the crate.
The crate records each attempt's outcome itself rather than through
litellm callbacks. A streamed attempt is deferred: it counts its
request when the stream opens, and the host's stream wrappers report
the success with the final usage, or a failure after the first chunk,
when the stream ends, as the Python core's callbacks do.
tests/unit/test_router/core characterizes PythonCore's public behavior
and runs every test on both cores, with nothing pending; only the names
of the Python core's own outcome callbacks are asserted on Python alone.
litellm.Router is now final: subclassing it raises TypeError at the class statement, and the
machinery that kept subclass overrides working is gone. The Rust core never honoured subclass
overrides, so they would have silently stopped running there. Custom routing goes through
register_plugin(), settings through apply() and outcomes through observe(). This is a breaking
change for anyone subclassing Router.
Fuzz parity fixes on top of the rebase: the Rust core keeps num_retries,
max_fallbacks, allowed_fails, retry_after, timeout, the tag settings,
routing_strategy_args and the fallback lists as given (the crate reads
them as numbers, update_settings casts as Python does); a deprecated
`model_list = [...]` write reaches the Rust core; a failed set_model_list
is all or nothing on both cores (the deployments served before the call
are put back and the error raised); config_deployments() is a generator and `del` of a plain
attribute works on the Rust core. Least-busy pick reservations from
main now work on the Rust core: its selectors share one reservation
store, routed calls reserve and free their unstarted picks, and
select() and batch cancel do not reserve. On the Python core a dropped
upsert takes its name back off deployment_names, and
get_deployment_order reads every numeric spelling as one int so mixed
orders no longer raise TypeError.
Rulings on the remaining differences, the Python core changed where it
was the odd one out: sync picks and picks by deployment id apply the
team filter as async picks do, so another team's deployment is refused;
the fallback lists and default_litellm_params in RouterSettings are
copies, so an in-place edit is ignored and settings change only through
apply(); one health record with an unreadable timestamp drops out alone;
an empty fallback entry is skipped; a bool cooldown_time is refused at
construction, a bool or string cooldown_time on a deployment or a call
falls back to the router default, and a failure status that no HTTP
status can be is recorded as 500; an infinite or NaN cost and a whole
number wider than 64 bits (ptu_shares) are refused on both cores with
the same error (litellm/router_utils/deployment_numbers.py); and a
/v1/messages stream whose provider iterator raises cools the deployment
and retries like a non-stream call, with the Rust core waiting for that
cooldown before its retry picks.
Differences the proxy parity fuzz and its regression tests found on the
Rust core: a file upload or a batch on an uploaded file falls back only
within its own model group, as on the Python core, since the file lives
with one provider (the host sends same_group_fallbacks_only and the walk
skips targets in other groups); Decisions calls drop deployments whose
provider has no Decisions config, and deployments that refuse the audio
or videos a call carries, unless that would leave none, as the Python
core does, while the retry count still reads the whole group; a file
upload that fails on one order level is created on the next level's
deployments rather than the first level again; and a response that
cannot be converted (a text_completion body without a message) reports
only its own traceback, so a rate limit handled by the attempt before it
no longer turns the error into a 429, and both cores return 500.
LiteLLM_AccountingEvent and LiteLLM_AccountingDelivery have had no reader since
the SpendLogs projection replaced them, so the migration drops them with their
enums. AccountingDeliveryState stays because LiteLLM_SpendLogDelivery uses it
The schema now spells the delivery_id default the way Postgres stores it, which
ends the drift Prisma reported against the ledger migration. The schema check
compares base types only, so an attribute-only change is no longer reported as
a type change
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(ui): show model deprecation banner on the models page
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(ui): format model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh model deprecations when models change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): address review feedback on model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): cover deprecation role gate and agent builder refresh
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): separate deprecation banner toggle from alerts link
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): dedupe and allow dismissing the model deprecation banner
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): refresh table once and key banner entries on successor
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: quiet successor logs, count banner models, use local dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): pin banner test date and bucket upcoming entries locally
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover mixed-successor log level through the deprecations endpoint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): move deprecation banner cases into integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep deprecation banner when a refetch fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(accounting): add opt-in SpendLogs ledger delivery
* fix(accounting): preserve missing organization daily spend parity
* fix(accounting): preserve ledger receipts when replaying migration
* fix(accounting): preserve callback windows and tune ledger batches
* fix(accounting): preserve window charges and bound ledger statements
* test(accounting): verify budget callback registration order
* test(accounting): verify captured charges after repeated startup
* feat(accounting): store durable payloads independently of spend logs
* fix(accounting): preserve first delivery and trace ledger writes
* fix(accounting): keep SDK imports independent of proxy tracing
* fix(accounting): honor log retention during batch takeover
* test(accounting): cover retained batch alerts and invalid settings
* test(accounting): verify downgrade command against pending work
* fix(accounting): drain retries without starving fresh deliveries
* test(accounting): cover delivery projection, metrics and window key reads
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): read delivery metrics without the shared registry
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): charge a leftover $0 batch row when its claim fails
A failed batch claim buffers the retrieve's row and reports it charged. At flush the
row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back
and the charge never reached the spend tables. The flush now takes that row over in the
same transaction and keeps the delivery.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): trace the batch takeover at flush as a SpendLogs update
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client
The export client is never created, so main always writes SpendLogs to the proxy's own
database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was
set, which dropped those request logs. It now treats a flush as an export only when the
client exists, and batch claims write local rows as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep every request's charge when its log row is shared, evicted or rejected
- Key non-batch deliveries on a per-request delivery_id so a reused client call id is still
charged once per request, while buffer replays stay deduplicated
- Persist the redacted api_key instead of an opaque bearer token
- Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it
- Strip NUL from payloads and skip non-text tags so projection never parks a paid request
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch
A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep charging a request whose tag is too long for the daily tag index
A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): store only guardrail billing fields in the accounting payload
The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): move spend payload stamping out of update_database
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): restore the modules the import-layering guard evicts
The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): keep update_database's payload stamping on its own local
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): type guardrail entries and use a public key redaction helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep charging requests whose end-user id is too long for an index
An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(proxy): type the spend log queue snapshot
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): leave out any projected row Postgres cannot index or store
Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep the guardrail fields compression savings are counted from
Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): cap projected counters so daily totals cannot overflow
Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): let a buffered batch charge take over its zero-cost receipt
A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read a rescued batch row's spend from its accounting payload
Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): read budget windows through the delivery transaction
Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): retry a transient failure while keeping a dropped row's charge
The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): give delivery projection payloads their charged entities
project() now reads entities only from accounting_entities, so the hand-built
delivery payloads carry them like real stored payloads do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): own ledger rows by their payload and key batches on the batch id
Drop the accounting_path stamp and its SpendLogs column. A buffered row that
carries an accounting payload is ledger owned, one without it is legacy, and
spend counters get a plain windows_from_ledger flag. Window seeding tells
applied ledger rows apart by their delivery row.
A ledger batch is plain transaction A keyed by the batch id, so a repeated
completed retrieve conflicts and charges nothing. A $0 retrieve records no
delivery, a batch a legacy retrieve already charged is not charged again, and
the legacy batch path is unchanged from main apart from refusing a batch the
ledger already owns. Batch takeover and its delta bookkeeping are removed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(accounting): drop a projection test duplicated while rebasing
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): drop previous_payload now that deliveries are never replaced
Batch takeover was the only writer of previous_payload, so the worker applies
each delivery's payload as stored and the column leaves the migration.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): drop the ledger downgrade checklist script
The use_spend_logs_ledger flag is the rollback path, so a separate downgrade
gate is no longer needed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): name the window seed reads so their Prisma span resolves
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): remove the unused accounting event types and protocols
types.py, protocols.py and measurements.py were left from the old event
scaffolding. Nothing outside the package imports them, so they go with their
tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): park only deliveries whose own data fails
A delivery that keeps failing on a database outage, lock timeout or deadlock
stays pending and retries with capped backoff, so a recovered database applies
it. Only a data error or an invalid payload parks the isolated row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate a spend log row the insert statement rejects
The raw SpendLogs insert validates columns and JSON before Postgres sees the
row, so a row that fails there is now bisected out and dropped alone while its
charge is kept, instead of failing and replaying the whole flush.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): carry a job's model window on every completed poll in ledger mode
The ledger keeps the poll whose delivery claims the job, which need not be the
poll that won the limiter's in-memory charge. Every completed poll now carries
the job's window, so the delivery that lands always has it, while the live
model budget is still charged once.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): leave model-budget windows on the existing window writer
Model-budget windows are written by the model budget limiter rather than
derived from a request's spend, so they stay on main's window queue like live
budget counters do. This drops window capture from the accounting payload,
the deferred limiter registration and the job-window capture, and the ledger
keeps owning key and team budget windows.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): keep a delivery pending when the database is unreachable
Prisma reports a can't reach database server outage as a DataError, so the
worker now checks for an unavailable database before treating a data error
as the delivery's own fault.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): isolate only rows the raw insert cannot encode
Catching every ValueError also swallowed unexpected insert errors that main
re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column
or non-JSON value, and only that is isolated as a rejected row.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(accounting): store zero savings when caller data makes them uncomputable
A huge caller-supplied tokens_saved overflowed the savings math inside
build_accounting_payload, which dropped the request's charge in ledger mode.
Savings that cannot be computed are now recorded as zero and the charge lands.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): treat Postgres restart and shutdown errors as DB unavailable, not poison spend-log rows
Postgres refuses every new connection with SQLSTATE 57P03 while it starts up, shuts down or replays WAL, and prisma surfaces that as a base DataError with no code. The spend-log writer took it for a poison row, bisected the batch down to single rows and dropped each one. Spend logs now retry and requeue on 57P03 and on server-terminated sessions (57P01, 57P02, 57P05), tool and model usage rollups requeue when the write provably never reached the server, the tool registry requeues its rows, and auth answers 503 instead of 401 while the auth database restarts.
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(proxy): requeue tool discovery rows only when the write never reached Postgres, keep the auth fallback to 57P03
A terminated session (57P01, 57P02, 57P05) may have committed the tool upsert, and the upsert adds call_count on conflict, so replaying it counts the calls twice. The non-Redis tool discovery flush now uses the same never-sent rule as the rollups, which also stops a permanently rejected row (54000) from coming back every tick. One terminated session does not prove the database is down, so it no longer opens the allow_requests_on_db_unavailable fallback, it still answers 503
* fix(proxy): a full server (53300, P2024) also never took the write, so counters and tool rows are sent again
Postgres refuses the connection before any statement runs when it is out of slots, and a prisma pool timeout never reaches the server either, so those failures are as safe to replay as 57P03 and P1001
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Dinu Arateanu <dinu.arateanu@rbinternational.com>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): answer an upstream overload with 503 server_is_overloaded on OpenAI routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass answered overload status to failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve Anthropic route exception handling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): bind the transformed failure without rebinding the parameter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): observe the overload failure hook through a real ProxyLogging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: make four flaky unit tests deterministic
The skill archive fixtures stamped the current time into zip bytes used as parametrize ids, so xdist workers collected different ids and errored the whole shard. They now use a fixed timestamp.
The usage-based v2 override test pinned no clock, so a minute boundary between rpm checks reset the counter. It now pins get_utc_datetime the way #49505 does.
The stalled-burst DB lookup test asserted a 2s wall-clock bound. It now asserts every lookup was waiting at once, which fails if lookups are serialized.
The long tool_search placement test asserted a 1s wall-clock bound. It keeps the placement assertion and drops the bound, as #49505 did for the json accumulator
* test(skills): drop the archive fixture timestamp change now that #50118 removed its test
* feat(billing): support Cognito bearer auth for metrics
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): refetch billing token on 401 and require https
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): import Unpack from typing_extensions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): make billing metrics bearer only and drop mtls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format billing metrics module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
