* feat(ui): rebuild the public AI Hub as a searchable list with a details pane
The public AI Hub at /ui/model_hub_table (and the older /ui/model_hub) now
renders PublicAiHub, built from shadcn parts only: a header with the
gateway's base URL and the admin's links, segmented tabs for models, agents,
MCP servers and skills, and a list beside a details pane on desktop that
becomes a bottom sheet on phones. The open item and tab live in the URL.
It keeps every feature the old page had: server-side search, provider, mode
and feature filters, sorting, paging, health, rate limits, prices, custom
metadata, supported params, usage snippets, the require-auth redirect and
the not-enabled state. cURL shows only for chat models, since the old one
called chat completions for every mode.
The navbar hides the community buttons below the sm breakpoint on every page,
and the public navbar uses a smaller logo on phones, so Docs and Blog no
longer collide with them.
* fix(ui): harden the public AI Hub after adversarial review
- Show "This AI Hub isn't available" only when every hub call fails; a
failed models call now shows an inline error with Try again and leaves
agents, MCP servers and skills usable (a gateway with no models answers
the models list with 400)
- Row links keep the page's other query params, so open-in-new-tab and
copy link no longer 404
- A shared ?open= link resolves models beyond the first page, keeps the
opened item when a search filters it out, and says when it is no
longer listed instead of showing a different item
- Searching, sorting and paging no longer blank the details pane or drop
the Operational badge
- Key MCP servers by their listed name, which is never null
- Keep sub-cent prices and show Free for zero cost, fold evaluation into
Decisions, sort agents, MCP servers and skills by name, restore
word-wise search, and show a skill's source, author and date added
- Drop the invented description for models without a mode, the empty
agent modes row, the stray auth dash on MCP rows and NaN dates
- Do not retry the UI config call, and close the 1023-1024px gap between
the sheet and the details pane
- Update the public hub Playwright spec for the new tabs and list, and
prune the lint suppressions the old route pages needed
* fix(ui): surface hub load errors and keep the open item steady
Agents, MCP servers and skills now load through apiClient so an HTTP error
shows the inline error instead of an empty "published yet" state, and an
all-failing gateway reaches the unavailable page. Try again shows loading
instead of an empty hub, the open model stays in the details pane when a
search or page change hides its row, a zero-result search keeps the pane,
phones get the not-listed sheet for a stale link, an empty open param means
nothing opened, free models read Free in the list, and an invalid skill date
is hidden
* fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call
* fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests
* fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500
* fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message
* fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails
A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever.
A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry.
On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed
* fix(agentic loop): bill retried attempts and clean up what failed or capped turns return
The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend
A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers
Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry
The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints
* fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns
A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call
Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try
* fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served
An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the
router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of
rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run.
The failure row bills every round the loop served before the failure, at any depth, and a later attempt
that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the
response cost headers adding up. The error response's x-litellm-response-cost reports the same amount
its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before.
A refused tool call now matches by id among the refused ids, or by name among refused calls that carry
no id, so a name-only refusal next to an id refusal is no longer kept.
* fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one
* fix(code_interpreter): delete only the session sandbox that failed, never its replacement
* test(code_interpreter): type the session sandbox test helpers
* fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors
* fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success
An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's
plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no
longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too.
A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row.
A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed
agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main.
A code interpreter sandbox that fails while running the code now says the code may have run but its
result is lost, instead of saying the code did not run.
* test(responses): bill a retried MCP request each attempt on its own row again
Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row
* fix(spend): name a failed follow-up attempt the success record already bills
With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off
* fix(spend): count a billed failed follow-up's tokens on the success row
* fix(spend): bill a failed follow-up on a fallback served from cache
* fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown
* fix(spend): keep a failed follow-up's billing state off the request-merged call details
* fix(spend): record the cost breakdown of a successful agentic follow-up
The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
PRs into rc/* branches never ran the OSV dependency scan because the
pull_request filter only listed main and litellm_**. rc branches only
take changes through PRs, so adding rc/** covers every change that
lands on them
* fix(ui): send only shown SSO fields when editing, and keep env-row SSO values behind a blank stored value
* fix(ui): clear every competing SSO provider when the admin switches provider
Only the detected provider's fields were nulled on a switch, so a second stored
provider (e.g. Microsoft behind Google) kept outranking the new one at sign-in
* chore(cost-map): remove the 52 rows whose provider retired them before 2026
* test(utils): pin the proxy function-calling parity case to azure/gpt-4.1
* test(batches): price the Azure batch-logging regression case against gpt-4.1
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(credentials): preserve stored info when patching credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(credentials): type patch test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep a deployment's token limits off the shared backend cost map key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): preserve catalog token limits on shared keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): use catalog for shared token limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drop the class attribute patch from the deployment max_tokens test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): avoid provider auth during catalog key resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): cancel the Admin UI's raw spend-log reads in Postgres after 30s
Prisma's HTTP client gives up on a statement after 30s, so the Logs, Caching,
Cost Optimization and spend report pages already fail at 30s, but Postgres kept
running the scan. Every raw LiteLLM_SpendLogs read behind those pages now runs
under SET LOCAL statement_timeout = 30000 through one shared helper, which
/team/spend/by_user also uses instead of its own copy
* fix(proxy): keep the spend report's 503 and time out the session totals read
The grouped Logs page's session totals query now runs under the same
statement timeout, /global/spend/report re-raises the helper's 503 instead
of turning it into a 400, and the hand-written prisma fakes in the endpoint
tests expose read_db so every guarded route is exercised
* fix(proxy): keep the Logs page when Postgres cancels its session totals read
A cancelled session totals read degrades like any other failed enrichment
read instead of failing the whole page. The span-name test treats the shared
statement-timeout helper as a forwarder, and the spend report test no longer
depends on a license in the environment
* test(proxy): /global/spend/report keeps the 503 hint when Postgres cancels the read
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(main): price per-request custom pricing from the logging object instead of writing litellm.model_cost
* fix(main)!: carry the full per-call price entry on the logging object instead of writing litellm.model_cost
Direct SDK calls build one ModelInfo from the cost map row plus every custom pricing field on the call and price from it, so audio, tiered and directional per-second rates apply without touching litellm.model_cost. Router calls keep their registered deployment entry. completion_cost(completion_response=...) alone no longer sees per-call prices, pass custom_cost_per_second or custom_cost_per_token
* test(cost): assert the exact per-second charge for per-call cost_per_second
* fix(main): price Router request overrides and flat rates over inherited tiers in the per-call entry
A Router call that sets its own price is priced from it over the registered deployment entry, PTU zeroing applied as at registration. Flat token rates on a call drop an inherited tiered_pricing table, which generic_cost_per_token would otherwise prefer
* fix(main): keep tier rates for directions a call does not override and bill per-second-only calls per second
A flat rate set on a call is written into every tier for that direction instead of dropping the tier table, so the other direction keeps its tier price. A call that sets only a per-second rate on a model with catalog token rates clears those token rates for the call so the per-second path applies
* fix(main): apply a call's flat rate to its own tiered_pricing table when it supplies one
* refactor(main): move per-call price entry resolution into litellm_core_utils/per_call_pricing
The resolution logic is pure cost-map work with no dependency on dispatch, so it lives next to ptu_pricing and main.py keeps only the call sites in completion() and embedding()
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): rename per_call_model_info parameter to requested_model_info_override
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(cost-map): record deprecation_source on every dated row that has a known provider notice
* fix(cost-map): keep deprecation_source only where the page lists the row's model and date
* fix(cost-map): drop deprecation_source from 4 rows whose own source page no longer lists the date
* fix(cost-map): give azure_ai/model_router the same deprecation_source as its hyphenated spelling
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The MCP SDK server answers a notification POST with 202 before it hands the notification to the session. When the gateway sends DELETE right after `notifications/initialized`, the session can close in that gap, and the SDK logs an ASGI error that fails the test at teardown. Wrap the upstream with the existing `_acknowledge_notifications_after_dispatch` helper the other SDK peers already use, so the 202 only goes out once the session has the notification
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Replace the hand-rolled combobox in the Cmd/Ctrl+K palette with shadcn's
base-vega Command and CommandDialog (cmdk) and bind the shortcut with
react-hotkeys-hook. cmdk now owns arrow keys, wraparound, Home/End and IME
composition, Base UI keeps the dialog mounted for its closing animation and
returns focus to the trigger, and mod+k is ignored while focus is inside
another dialog
Results keep their fixed Keys, Actions, Pages order with shouldFilter
disabled. The highlight resets to the first row whenever debounced key
results load, so Enter after typing still opens the top key
cmdk 1.1.1 leaves aria-activedescendant empty or stale after it picks a row
on its own (dip/cmdk#413), so a MutationObserver keeps it on the row that is
actually aria-selected until upstream fixes it
* feat(proxy): add LITELLM_FIPS_MODE startup gate
With LITELLM_FIPS_MODE=true the proxy refuses to start when TLS verification
is off, when Python's OpenSSL does not enforce FIPS, or when the startup
password migration fails. A value that is not true/false also refuses.
* feat(proxy): refuse FIPS mode when cryptography's OpenSSL is not FIPS
The stock cryptography wheel bundles its own non-FIPS OpenSSL, which serves
JWT, Fernet and AES-GCM even when Python's OpenSSL enforces FIPS.
* fix(proxy): refuse FIPS mode when cryptography's FIPS provider cannot hash
The stock cryptography wheel reads the host's FIPS config and reports FIPS on,
but its bundled OpenSSL cannot load the host's FIPS provider, so every
operation fails. Require a working SHA-256 alongside the FIPS flag.
* fix(proxy): refuse TLS verification off anywhere in FIPS mode
A model's own ssl_verify: false reached the HTTP client after the startup check
passed. get_ssl_verify now raises in FIPS mode whenever verification resolves
to off, which also covers the startup check.
* refactor(proxy): return FIPS boot refusals as values and harden the FIPS tests
* fix(proxy): stop refusing TLS verification off in FIPS mode
Skipping certificate verification is a security choice, not a FIPS one:
the TLS connection still only uses algorithms the FIPS provider allows.
* feat(fips): refuse a database used without FIPS mode, drop the password migration change
FIPS mode marks a new database with a fips_mode row in LiteLLM_Config and refuses one that already holds
LiteLLM rows without it. Without FIPS mode, the proxy refuses a database carrying the marker.
* test(fips): close the database test clients with aclose so the engine stops
* feat(fips): read FIPS mode from general_settings.fips_mode instead of LITELLM_FIPS_MODE
* refactor(fips): keep fips_mode out of the API-writable general settings, it is read from the config file only
* fix(fips): retry the startup marker and state reads once after a dropped connection
A dropped connection during the non-FIPS marker read used to count as an unmarked
database, so a blip at boot let a non-FIPS proxy start on a FIPS database. Both reads
now go through call_with_db_reconnect_retry, the reconnect-and-retry the other startup
reads use. A read that still fails keeps its old outcome.
* fix(fips): refuse a fips_mode key that has no value
fips_mode: os.environ/X with X unset, or a bare fips_mode: / null, resolved to None and
started without FIPS mode and without a message, while an empty string refused. A key
that is present now has to be a boolean; only a config without the key means off.
* refactor(fips): move the database check out of the boot gate
The boot gate keeps the setting and the two OpenSSL checks. The check that a
database matches the FIPS setting moves to its own PR, so a boot without the flag
runs no extra database read.
* test(fips): pin how a fips_mode string is read through enforce_fips_mode
* feat(model-catalog): add regional_processing_uplift_multiplier_fedramp
* refactor(model-catalog): make the FedRAMP uplift its own fedramp_uplift_multiplier key
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rate-limit): estimate embedding image and audio data URIs as media in TPM pre-call checks
* fix(rate-limit): count embedding media as media only on models that embed it
A data URI sent to a text-only embedding model is read as text, so it keeps
the text estimate there. On models flagged supports_embedding_image_input,
image and audio use the chat media estimates and PDF and video count at most
the model's max_input_tokens. The same estimate now feeds project and key
limits, the usage-based, latency and cost routers' deployment tpm, and the
deployment itpm check.
* fix(proxy): count embedding media only where the backend reads it as media
Vertex AI, Gemini, Azure AI and Bedrock count a strict base64 data URI as
media per item. Cohere counts it as media only when the whole batch is
media, and Voyage, OCI and unknown models keep the text estimate. File
blocks carrying a data URI count like the bare URI.
The router context window pre-call check and the batch limiter now use the
same estimate, resolving each deployment or JSONL row model to its
backends. The cost map gains supports_embedding_image_input on
voyage/voyage-multimodal-3 and oci/cohere.embed-multilingual-image-v3.0.
* fix(router): hold each embedding media item to the context window on its own
A PDF or video data URI alone already reserves at most the model's
max_input_tokens, but the context window pre-call check then ran the
blanked input through the Responses transform and added a few framing
tokens, so a PDF that Vertex accepts got a 400. When every input item is
media, the check now returns the largest item's media estimate, since an
embedding request embeds each item separately
* fix(proxy): leave embedding media out of the context check and narrow Bedrock media scope
The router context window check now counts only the text of an embedding input, so Titan image lists no longer 400. Bedrock Marengo and Nova with a PDF keep the text estimate since they read those inputs as text or reject them, async_invoke Nova counts images as media, and the unused batch limiter change is dropped.
* fix(bedrock): bill Cohere embeddings by Bedrock's input token count header
* fix(embeddings): check data URI base64 in place and keep Bedrock Cohere text usage
The strict base64 check now scans the payload in place instead of decoding
and re-encoding a full copy. Bedrock's billed token header is used only for
image requests, so text-only Bedrock Cohere usage matches main. The deployment
itpm estimator change is reverted, since the router never passes embedding
input to that check.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): let the Admin UI cancel /team/spend/by_user at 60s with an opt-in statement timeout
GET /team/spend/by_user takes an optional timeout_seconds (1 to 600). When set, the raw
LiteLLM_SpendLogs aggregation runs under SET LOCAL statement_timeout and Postgres cancels it,
and the route returns 503 instead of letting the query run for hours. Callers that omit the
param are unchanged. The Admin UI sends timeout_seconds=60 and the Spend Per User Within Team
card shows the server's error instead of an empty table
* fix(proxy): cancel /team/spend/by_user in Postgres after 30s
Prisma's HTTP client already gives up on the statement at 30s, so the
query now runs under SET LOCAL statement_timeout = 30000 for every caller
and the opt-in timeout_seconds param is gone. The Admin UI card shows the
request error instead of the empty-state text.
* fix(proxy): keep /team/spend/by_user on the read replica inside its timeout transaction
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: show user names in access group details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: show user names on project, org, model and prompt details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): look up user names by exact id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back to user list for org admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back when /v2/user/info returns a different user id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): let org admins read their org members in /v2/user/info
Org admins could list a user through /user/list but got a 404 from
/v2/user/info for that same user, so the dashboard fell back to a
substring /user/list lookup that could miss the user. /v2/user/info
now allows an org admin to read users in organizations they administer,
and the dashboard's single-user lookup calls only that endpoint.
Expected 4xx responses from /v2/user/info no longer log a traceback.
* fix(proxy): drop unbounded IN from org admin check in /v2/user/info
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): record the provider's recommended replacement as successor_model on deprecating rows and report it from /model/deprecations
* fix(cost-map): pick the alias row's successor by its own shutdown date when the family has several
* chore(ui): regenerate schema.d.ts for the /model/deprecations successor_model wording
* fix(proxy): prefer a configured successor_model over the cost map's when deployments share a name, and keep the cost map's under a hand-set deprecation_date
* test(cost-map): allow successor_model in the cost map row schema
* feat(cost-map): infer successor_model as the next version in the same family for 136 dated rows with no published replacement
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit cost headers for unpriced responses
* fix(proxy): preserve declared free response costs
* refactor(proxy): share the cost map pricing lookup for the response cost header
The header check now reuses the same raw entry, fill-missing and rule lookup that
cost model selection uses, so the two can't drift apart. It also counts every declared
rate field, so a free model priced only per request keeps its 0.0 header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(ui): move teams, users and orgs text onto i18n
Moves the page-authored English on /teams, /users and /organizations,
plus the shared member table, user search modal, model select and budget
schedule helpers they render, into react-i18next t() calls backed by
en.json. The rendered English is unchanged
Helpers that files outside this batch or untouched tests still call keep
their signatures and take an optional t that defaults to the shared
instance, the same as usageUnits.ts. Text that existing tests pin, values
compared against the API, code samples, CSV column names and date formats
stay as they were
* refactor(ui): escape member ids in team member dialogs and merge split org and team strings
The reset spend and reset budget dialogs interpolate the member id into a Trans string, so an id containing strong, i, p or br tags rendered as markup. Escaping the values and unescaping text nodes keeps it literal, as it was before the move to i18n. Also merges the org budget lines and the team metadata hint into whole sentences and drops two constants nothing imports anymore
* refactor(ui): pass the member id into the team member reset dialogs as an element
Escaping the id still let Trans fill in placeholders like {{budget}} written inside it a second time. Rendering the id through its own component keeps it out of the translated string, so any id shows exactly as it did before the move to i18n
* fix(databricks): mark audio and video input support on the Gemini 3.5 to 3.8 Flash rows
* fix(model_prices): add Vertex AI and Together AI retirement dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): correct perplexity agent api pricing and add gemini-4-argon, nemotron-3.5-super
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): consolidate verified pricing and Aplomb metadata
Absorb verified pricing from #49547 and #49548 and Aplomb 1 metadata from #49539. Preserve provider tiers and remove an identical duplicate retirement key.
Co-authored-by: Liutasil501 <schmid.martin501@gmail.com>
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(model_prices): sync generated EmpirioLabs endpoint metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): verify Nebius and EmpirioLabs model metadata
Verify canonical EmpirioLabs rows from #49695 against published model pages, preserving pricing tiers and excluding unverified variants and specialty billing. Add Nebius models and cached-input prices from the official catalog and correct the Voyage legacy input limit.
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): absorb Bedrock Opus 5 batch prices, Vertex Omni Flash limits and Gemini 2.5 shutdown dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(model_prices): drop the EmpirioLabs rows and endpoint flags from this sync
* fix(model_prices): merge main and dedupe the gemini-2.5 deprecation_date keys
* fix(model_prices): set supports_video_input on the two new Nebius video models
* fix(model_prices): add Azure AI Claude Mythos 5.1 and match Foundry max effort support
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
* fix(proxy): list slash-less wildcard groups like claude-* on /v1/models and /model/info
* fix(proxy): expand team-owned wildcard rows on /model/info through their public name
* fix(proxy): keep listing wildcard names wider than their deployment and skip the Azure placeholder
A group like *gemini-3* over gemini/gemini-3* lists the catalog ids the name matches and routes back, as main did on /model/info, now on /v1/models too. Wildcard listings no longer list the Azure-LLM placeholder. Team-owned wildcard rows on /model/info go back to main's behavior.
* fix(proxy): keep Azure listings on slashed wildcards and list bare deployments through custom_llm_provider
get_provider_models returns main's list again, and only slash-less names skip the Azure-LLM catalog placeholder. A slash-less name over a bare wildcard deployment now lists the catalog of its custom_llm_provider.
* test(proxy): pin custom_llm_provider guards and slashed wider-name listing
* fix(proxy): list bare ids for wider names over a custom_llm_provider deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): carry prompt_cache_breakpoint to cache_control on chat completions to Claude
A content part carrying OpenAI's prompt_cache_breakpoint on a /v1/chat/completions request to Claude (Anthropic, Bedrock Invoke and Converse, Vertex AI, Azure AI) now reaches Claude as cache_control {"type": "ephemeral"} on the same block, so prompt caching works for OpenAI-shaped clients instead of the marker being stripped.
Only parts OpenAI accepts the marker on are translated, a client cache_control on the same block wins, and Converse translates only for models that take cache points. A 30m ttl or a malformed marker answers UnsupportedParamsError without drop_params and becomes a default 5m mark or no mark with it.
* fix(anthropic): hold translated prompt_cache_breakpoint marks to the 4-breakpoint cap
Marks past Claude's 4 cache breakpoints now answer 400 without drop_params and
keep the earliest under it, counting the client's own message, tool, and
top-level cache_control marks. The ttl and malformed-marker errors name
drop_params for proxy users too and no longer name a single provider.
* fix(bedrock): count Converse tool cache points in the breakpoint cap and gate unmapped models
A client tool cache_control or a tool_config injection point becomes a Converse tool
cachePoint, so it now takes one of the 4 slots before a translated marker does.
Converse only translates prompt_cache_breakpoint for a model the cost map says takes
prompt caching. An unmapped model such as an application inference profile ARN opts in
through model_info supports_prompt_cache_breakpoint, so a marker sent to every model of a
mixed fleet no longer turns a non-caching profile's 200 into Bedrock's 403.
Router prompt-caching affinity now treats a marked block like a cache_control one, and
requests without a marker skip the translation after one pass over their messages.
* fix(bedrock): count only the tool cache points Converse sends against the breakpoint cap
* fix(anthropic,bedrock): cap only the cache breakpoints each provider sends, and drop marks Nova cannot take on tool results
* perf(anthropic,bedrock): find the breakpoint cut-off by binary search so trimming many marks stays fast
* fix(anthropic): drop the duplicate drop_params left by the main merge
* fix(bedrock): let an inference profile's cache breakpoint flag win over its base_model
* fix(anthropic): keep the first and last prompt_cache_breakpoint marks past the cap
Under drop_params the translated marks that fit the cap are now split between
the start and the end of the request, so the stable prefix and the newest turns
both stay cached. The cap count also includes a native tool search tool that
keeps its cache_control on the wire.
Bedrock Converse now translates the marker only for the Claude and Nova
families, which Bedrock documents as taking cache checkpoints, unless the
deployment sets supports_prompt_cache_breakpoint. A guarded user turn keeps its
cache mark when the request fits the cap.
* refactor(bedrock): decide guarded cache marks with a typed bool helper
* fix(bedrock): validate guarded blocks before reading their cache mark
* fix(anthropic): let injection points replace client marks and keep non-Claude routes untranslated
When a deployment sets cache_control_injection_points, the client's
prompt_cache_breakpoint marks are dropped on targets that do not read them, so
the operator's injection points decide every Claude breakpoint.
OpenCode Go models served over the Anthropic messages endpoint translate the
marker only for Claude, and guarded Converse text keeps a cache mark only on a
model known to take cache points.
* fix(anthropic): keep client marks unless a message injection point replaces them on a translating route
A tool_config-only injection point no longer strips client prompt_cache_breakpoint marks, a flagged Bedrock profile no longer counts as reading the marker natively, and guarded Converse text skips the tool count when no message carries a cache mark.
* fix(anthropic): keep Bedrock and Claude deployments flagged for the marker on cache_control injection
A supports_prompt_cache_breakpoint flag on a Bedrock profile or a Claude deployment made the injection hook write OpenAI markers and prompt_cache_options, which Bedrock answers 400 for. Routes that translate the marker now never take the OpenAI dialect.
* fix(bedrock): reserve only the tool cache points Converse sends when trimming Claude cache marks under drop_params
* fix(anthropic): name the marker's accepted ttl values in the malformed prompt_cache_breakpoint 400
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API
* fix(logging): log bridged chat completions with the chat response they returned
* fix(logging): log bridged text completions and /v1/messages as served, with the request's options
Bridged rows were translated without the request's options, so structured
output and replay fields could differ from what the client got. Text
completions and non-streamed /v1/messages through the chat bridge also kept
a Responses API object or the encoded id.
* fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions
Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly.
* fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack
* test(integration): look up bridged chat spend rows by the id the client got
Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): spell "environment" right in the allowed_ips and AWS KMS V2 license errors
* fix(proxy): read general_settings.litellm_license before the allowed_ips and enforced_params gates
* fix(proxy): read the config license before callbacks and gates, keeping a valid env license
* fix(proxy): keep an invalid config license reported and sync license data with the license in use
A config license that fails no longer overwrites the license string when the env license is not valid either, so /health/license still reports has_license. A failed boot-time env license is not re-checked. Both license writes go through one helper that also refreshes premium_user_data, so /health/license and the seat limits follow the license in use.
* fix(proxy): keep the env license's boot result when a config license fails, without a second license check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(azure_ai): keep azure_ad_token out of the request body
azure_ad_token is not a litellm-owned param, so get_optional_params routes it
into extra_body and the azure_ai chat and Claude transformations merge it into
the JSON body. Azure AI Foundry then rejects the call with 400 Unknown
parameter / Extra inputs are not permitted. The Authorization header already
comes from litellm_params, so drop the extra_body copy once at the top of the
azure_ai dispatch, as the azure branches do.
Type litellm.open_ai_chat_completion_models and
open_ai_text_completion_models as Set[str] so the basedpyright gate does not
grow from the untyped optional_params read.
* fix(azure_ai): keep Entra credentials out of every azure_ai request body
* fix(azure_ai): keep the Entra token out of the image edit form
* fix(azure_ai): keep Entra credentials out of image edit extra_body too
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000
The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry
An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused
* fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed
When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input
The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts
A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged
The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success
* fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row
A realtime session's success event now reports status failure, with the same
error_information the spend row stores, when the session recorded a provider
failure in its metadata. Usage already streamed stays billed and the router
still counts one cooldown failure
The spend row builder now runs every row's error_information through
sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction),
so the realtime failure rows get the same treatment as the proxy failure hook,
which no longer sanitizes on its own to avoid truncating twice
Drop the selection-failure tag on the committed-session refusal, since the
shared logging object already keeps a retried refusal out of cooldown, and pin
the router's realtime success counting with an OpenAI realtime deployment
* fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops
The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop
* fix(realtime): count a router success only for a session that ended cleanly
A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had
A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers
* fix(bedrock): log a dropped realtime session's usage from its own logging state
The failure event replaced the shared standard_logging_object while the
queued success dispatch was still running its callbacks, so loggers
received $0 and 0 tokens for a billed dropped session.
* fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success
A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed
* fix(rate-limit): settle each token reservation once when a request logs both a failure and a success
A dropped realtime session logs a success for its usage and then a failure. After a success the
limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing
twice. A new reservation written in the same context after a released one now starts clean, so
reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or
28 ITPM for 27 and 20 used
* fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once
* fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge
* test(rate-limit): pin the split hold-release flags across nested MCP pre-calls
* fix(realtime): count a relay that fails on its own as an abnormal end
* test(realtime): pin the relay failure exit with a client frame the provider rejects
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry
Two deployments that share one model string share one key in the live
litellm.model_cost map, and router registration writes each deployment's
supports_prompt_cache_breakpoint into that key. The cache control hook read
the live map, so a flagless deployment picked up a sibling's opt-in and sent
prompt_cache_options and prompt_cache_breakpoint upstream, which the provider
rejects with a 400
When a call carries the routed deployment's model_info, the prompt cache
checks now use its declared supports_prompt_cache_breakpoint, and fall back
to the loaded catalog instead of the live map. This covers the hook on
completion, acompletion, Responses and the /v1/messages route, the azure_ai
chat transform, the OpenAI Responses transform under drop_params, and the
/v1/messages to Responses adapter
* fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers
A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's
prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a
provider other than openai, such as azure_ai OpenAI models and bedrock_mantle.
Bedrock on every route and providers that build Anthropic messages keep
cache_control, which Bedrock Converse turns into cachePoint
The openai chat reader now passes the routed model_info and checks base_model.
The Bedrock cache point reader takes the deployment's flag first and, for a
routed call with no flag, reads the catalog as loaded instead of the live map
that sibling deployments write into. model_info is threaded from the converse
transform, the factory message and tool builders, and the responses bridge
* fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls
A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations.
* fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways
* fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases
* test(anthropic): cover auto caching on an unmapped Claude deployment
* fix(cache_control): keep the merge base's reads where the per-deployment flag changed them
A register_model entry under a provider-prefixed name reaches routed calls again, since the
overlay now resolves the bare and prefixed names the way runtime registrations are stored.
The hook and the openai chat reader no longer judge base_model, as at the merge base, so a
client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint
and get a 400.
A Bedrock deployment flag that is not a real true reads as false again, as at the merge base.
An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again,
as at the merge base. Other wildcards still leave both cache flags to the catalog.
The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag
on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps
its own flag and that a flagless unmapped ARN keeps its cache points after the delete.
* fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name
Routed calls read register_model entries through an overlay that merged the bare
name after the provider-prefixed one whatever order they were registered in, so a
later opt-out under one name lost to an earlier opt-in under the other. Each
registration now moves its key to the end of the kept registrations, and the
overlay merges both names oldest first, which matches what SDK calls read from
the live model map.
Also lists the wildcard cache flag helper in the router coverage script, since
the hook's wildcard tests drive it rather than calling it by name.
* fix(cache_control): keep register_model entries visible while a cost map reload replays them
The previous commit moved a re-registered name to the end of the saved
registrations by removing and re-adding it, and the overlay walked that dict to
order a bare and a provider-prefixed name. A reload replays every saved entry,
so a request on another thread could briefly miss an entry or fail on a dict
that changed size mid-walk. Entries are written in place again, a per-name
sequence records which registration came last, and both the overlay and the
reload replay order by it, so the later registration still wins after a reload.
* fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix
A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(realtime): support latest OpenAI audio models
* fix(transcription): type SDK stream duration metadata explicitly
* fix(realtime): preserve transport settings and translation duration
* refactor(realtime): reuse SDK client initialization
* fix(audio): bill finalized streams and authorize transcription models
* refactor(audio): keep transcription pricing selection together
* fix(realtime): reconcile model metadata and test contracts
* fix(realtime): preserve provider duration and model capabilities
* fix(realtime): keep transcription validation compatible with cost map guard
* fix(realtime): address transcription and billing review findings
* fix(realtime): preserve input usage and provider routing
* fix(realtime): bill retained output and route Azure aliases
* fix(realtime): reject malformed provider durations
* fix(realtime): preserve batch cache rates and Azure v1 routes
* fix(transcription): reject nonfinite provider durations
* refactor(realtime): align annotations with current lint rules
* fix(realtime): preserve translation deployment pricing and input usage
* test(transcription): verify synchronous stream finalization
* fix(realtime): retain missing translation input usage at close
* fix(transcription): reject streams that bypass output guardrails
* fix(realtime): reconcile audio protocols billing and current CI
* fix(realtime): trust confirmed formats and authorize default models
* fix(realtime): preserve absent models and reject unsupported guardrails
* fix(realtime): reject translation policy pipelines
* chore(ci): rerun transient infrastructure timing failure
* fix(realtime): reconcile audio protocols with current upstream
* fix(audio): validate provider responses at SDK boundaries
* fix(realtime): preserve minimum OpenAI SDK compatibility
* fix(realtime): preserve deferred Live session authorization
* fix(realtime): preserve upstream authentication and transcription behavior
* fix(audio): scope provider duration precedence to configured models
* fix(typing): avoid Final declarations inside transcription cost loop
* refactor(audio): reuse transcription mapping and duration resolution
* fix(realtime): enforce model budgets and preserve translation routing
* fix(audio): log streamed transcripts whose done event reports detected languages
* fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers
Providers without a streaming transcription handler got UnsupportedParamsError
for stream=True, languages and keywords, where the merge base forwarded them as
provider-specific params. Only openai and azure validate them now.
* fix(audio): bill streamed gpt-transcribe from the provider's reported duration
The stream path computed cost after logging had replaced the transcription's
duration usage, so it fell back to the measured file length while the
unstreamed call billed the provider's seconds.
* fix(transcription): log streamed transcription usage from the provider response
The stream logging path built the spend payload from the response after
its duration usage was transformed away, so whisper-1 with stream=true and
streamed gpt-transcribe logged an empty usage object. Bill and log from the
provider's TranscriptionResponse in both the sync and async paths.
* fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes
- Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape
- Use the input transcription model as the session model only for transcription sessions
- Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main
- Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set
- Return x-litellm headers on streamed transcriptions and mark the request done
- Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure
- Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing
- Bill translation sessions on output audio seconds only, at the single published rate
- Document the 201 SDP answer on the translation calls routes and the transcription 200 response
* fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients
Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the
guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the
default dated version on the deployments route for every model. The SDK client secret keeps
session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata
keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two
unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the
gpt-transcribe format check only reads the request provider's cost map entry.
* fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it
Transcription reaches every provider as on main again, sends stream only when asked, forwards language and
languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the
new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a
mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from
one source and spend rows carry the billed seconds.
Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward
explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths
are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later
session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA
get beta-shaped session events, and transcription sessions send the deployment model in both shapes.
* fix(realtime): address review findings on transcription billing, streaming and realtime auth
Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata
* fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only
* fix(realtime): preserve prior merge resolutions after rebase
* fix(realtime): enforce translation policies and normalize Azure transcription URLs
* fix(cookbook): stop translation once output idles after the upload ends
* fix(realtime): bill translation on input audio and drop pre-read stream cost headers
Translation spend now uses the input audio seconds the session sent, priced
through input_cost_per_second, instead of the translated output duration.
A streamed transcription no longer returns cost headers before the stream is
read, since the cost is only known after the last event. Also removes an
unused realtime guardrail helper.
* fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields
Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop
* test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription
* fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA
Bill translation sessions from the deployment's own per-second rates, add per-second prices for
Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map
mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to
the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation
calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models
* fix(realtime): bill a translation rate the deployment omits from the session model
* test(e2e): check translated speech is not English instead of exact French wording
* fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices
Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending
* fix(realtime): keep the proxy's realtime session gate off translation sessions
A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects
* fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model
A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params.
* fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree
* fix(model-catalog): accept the realtime translation endpoints in the cost map
* fix(cookbook): close the translation session and read until session.closed
* fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard
An empty `stream` form value runs the transcription unstreamed again, as
OpenAI does, instead of answering 400. The four translate cost-map entries
drop the translation routes from supported_endpoints until the main
branch's catalog parser accepts them, and the nested transcription model
lookup uses the router registry entry point.
* test(proxy): drop the mid-stream error frame pin that main's error shape change replaced
* fix(transcription): bill uploaded seconds when a stream fails mid-transcript
* fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream
* fix(audio): keep transcription's positional arguments and redact streamed error details
- Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning
- Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do
- Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite
* fix(audio): stream the guarded transcript to SDK callers that asked for a stream
* fix(logging): keep a streamed transcript out of callbacks that turned off message logging
A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does
* fix(proxy): stream a transcription the deployment streamed when the client sent no stream field
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): send an https PDF image_url to Claude as a document
A PDF link sent as image_url to Claude on Anthropic went out as an image
block and Anthropic answered 400. A link whose path ends in .pdf is now
downloaded like Vertex links are, and the served Content-Type picks the
document or image block, so CDN links that serve a PNG from a .pdf path
keep working. An explicit format of application/pdf sends a url document
with no download. PDF media types match regardless of case or parameters,
and count_tokens picks the same block as the request.
* fix(anthropic): download https PDF links off the event loop
The proxy downloaded an https link ending in .pdf synchronously while
building the Anthropic request, so on an async call every other request
on that worker waited for the download. Those links are now inlined
before the request is built, the same way http links already are, and
the /v1/messages guardrail write-back does the same. A non-string
format on an image_url is ignored the way it already was inside the
image_url mapping, instead of raising an AttributeError.
* fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent
* test(anthropic): expect the PNG media type the served bytes carry at a .jpg path
* fix(anthropic): bound the PDF link download and keep query-string PDF names
Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the
client's 6000 s default, fall back to main's whole-URL extension check so
download?file=report.pdf still counts as a PDF, treat application/x-pdf as
a PDF, and set the url document's application/pdf format only for the
guardrail write-back so other /v1/messages providers send what main sends
* fix(anthropic): probe a .pdf link and send it to Claude as a url document
Instead of downloading a .pdf link and inlining it as base64, ask the host for
its first kilobyte and send the link as a url document when it serves a PDF.
Every other link goes out as the url image main sends, so robots.txt handling,
caching and the wire format stay as they are on main.
Also restore the guardrail adapter and the content type inference to main,
keep the client's url document blocks (title, context, citations,
cache_control) through the guardrail write-back, treat a PDF filename in the
query string as a PDF link, probe links concurrently on the sync path, cap the
async probe's read at the download limit, and catch only transport, URL and
SSRF failures in the probe
* fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place
The probe now streams the response and keeps the first 1 KB, so a host that
ignores Range no longer fills memory with up to 20 parallel bodies. The
guardrail write-back now matches url documents inside a tool_result and outside
one separately, since the write-back moves tool results ahead of other content
* fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke
* fix(anthropic): keep a PDF probe's error status for a minute, not ten
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Mistral lowered Large 4 to $0.70 in, $0.07 cached in and $2.10 out per million tokens, so both rows in both cost map files move to that price and point their source at the model page that lists it
Breaking change: Large 4 calls are logged at about half the old cost, so budgets last longer and cost-based routing may favor it
* fix(embeddings): honor the caller's timeout on Cohere sync and HuggingFace embedding requests
Bedrock Cohere embeddings ran the sync request with the client default instead of the timeout
passed to litellm.embedding, and HuggingFace embeddings dropped it on both the sync and async paths.
Pass the timeout through to the HTTP request on each path.
Fixes#47301Fixes#47299
* fix(huggingface): pass the caller's timeout to the embedding model lookup too
Without input_type the handler first fetches the model's pipeline tag from huggingface.co, and that GET still used the default timeout. Thread the caller's timeout through both lookup helpers and the transform step so every request the embedding makes honors it.
Refs #47299
* fix(huggingface): keep the unreachable async lookup path untouched
The async model lookup helper and the call_type="async" transform branch are never reached, since the async embedding path transforms through the sync lookup. Leave them as they were so the change only touches requests that actually run.
