Only a guardrail header this proxy recorded gives way to a callback header that leaves it no room, so a stream
whose proxy recorded nothing passes the upstream's headers through unchanged, as the buffered path does.
- Drop a client's user_api_key_* metadata, non-string headers, non-object requester_metadata and non-string
litellm_call_id, litellm_trace_id, custom_llm_provider, api_base, api_key and proxy_server_request headers
before the merged guardrails run, as the native routes overwrite them
- Keep a guardrail the body turns off with {name: false} out of the merged list
- Put back a body's empty guardrails list when the merge finds nothing to run
- Leave a non-list applied_guardrails record alone instead of failing the request
- Skip key, team and project policy entries that aren't names
* feat: show user names in access group details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat: show user names on project, org, model and prompt details
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): look up user names by exact id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back to user list for org admins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): fall back when /v2/user/info returns a different user id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): let org admins read their org members in /v2/user/info
Org admins could list a user through /user/list but got a 404 from
/v2/user/info for that same user, so the dashboard fell back to a
substring /user/list lookup that could miss the user. /v2/user/info
now allows an org admin to read users in organizations they administer,
and the dashboard's single-user lookup calls only that endpoint.
Expected 4xx responses from /v2/user/info no longer log a traceback.
* fix(proxy): drop unbounded IN from org admin check in /v2/user/info
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost-map): record the provider's recommended replacement as successor_model on deprecating rows and report it from /model/deprecations
* fix(cost-map): pick the alias row's successor by its own shutdown date when the family has several
* chore(ui): regenerate schema.d.ts for the /model/deprecations successor_model wording
* fix(proxy): prefer a configured successor_model over the cost map's when deployments share a name, and keep the cost map's under a hand-set deprecation_date
* test(cost-map): allow successor_model in the cost map row schema
* feat(cost-map): infer successor_model as the next version in the same family for 136 dated rows with no published replacement
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): omit cost headers for unpriced responses
* fix(proxy): preserve declared free response costs
* refactor(proxy): share the cost map pricing lookup for the response cost header
The header check now reuses the same raw entry, fill-missing and rule lookup that
cost model selection uses, so the two can't drift apart. It also counts every declared
rate field, so a free model priced only per request keeps its 0.0 header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): list slash-less wildcard groups like claude-* on /v1/models and /model/info
* fix(proxy): expand team-owned wildcard rows on /model/info through their public name
* fix(proxy): keep listing wildcard names wider than their deployment and skip the Azure placeholder
A group like *gemini-3* over gemini/gemini-3* lists the catalog ids the name matches and routes back, as main did on /model/info, now on /v1/models too. Wildcard listings no longer list the Azure-LLM placeholder. Team-owned wildcard rows on /model/info go back to main's behavior.
* fix(proxy): keep Azure listings on slashed wildcards and list bare deployments through custom_llm_provider
get_provider_models returns main's list again, and only slash-less names skip the Azure-LLM catalog placeholder. A slash-less name over a bare wildcard deployment now lists the catalog of its custom_llm_provider.
* test(proxy): pin custom_llm_provider guards and slashed wider-name listing
* fix(proxy): list bare ids for wider names over a custom_llm_provider deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
- Drop a top-level guardrails or guardrail_config the merge can't combine,
which returned 500 where main stripped the field.
- Count only a pre_call policy pipeline when withholding the Anthropic user_id.
- Read /bedrock invoke replies whole for generic guardrails, as on main.
- Keep the provider's body for logging when a hook returns a non-dict.
- Settle a no-cost blocked response at 0 so its failure row keeps its usage.
- Leave custom endpoints uncharged on a failed post-call hook, as on main.
- Keep a legacy Lakera string list's empty entries.
* fix(proxy): spell "environment" right in the allowed_ips and AWS KMS V2 license errors
* fix(proxy): read general_settings.litellm_license before the allowed_ips and enforced_params gates
* fix(proxy): read the config license before callbacks and gates, keeping a valid env license
* fix(proxy): keep an invalid config license reported and sync license data with the license in use
A config license that fails no longer overwrites the license string when the env license is not valid either, so /health/license still reports has_license. A failed boot-time env license is not re-checked. Both license writes go through one helper that also refreshes premium_user_data, so /health/license and the seat limits follow the license in use.
* fix(proxy): keep the env license's boot result when a config license fails, without a second license check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): fail a Nova Sonic realtime session whose Bedrock stream drops instead of closing 1000
The Bedrock SDK ends the output stream the same way for a dropped connection as for a session that sent sessionEnd, so the handler treated a dropped stream as a clean finish. The client got a 1000 close with no error event and the router logged 200 OK with no retry
An output end now counts as clean only after the client left and the handler began closing the Bedrock input. An earlier end raises a 503 BedrockError, so the client gets an error event and a 1011 close, the failure handlers run, partial spend is still logged, and a retry of a committed session is still refused
* fix(bedrock): fail a dropped Nova Sonic session right away and log it as failed
When Bedrock's output side ends mid-session, by a drop or a modeled stream error, the client side no longer spends the close timeout sending close messages into the dead input stream, so the error reaches the app right after the drop. A session the app ends itself still sends Bedrock its close messages and closes the input
The replay refusal now leads with the committed cause, so the error event and the cut 1011 close reason both start with it. It stays a 400 and is tagged as a failure outside the selected deployment, so router retries and fallbacks of a dropped session no longer add extra failure counts
A clean realtime session now counts once toward its deployment's cooldown success counter, so cooldown reads the real failure rate instead of 100 percent. Realtime tpm and rpm counting is unchanged
The spend row of a session the provider ended now reads failure with its error information, while the events already forwarded are still billed. A session the app ended keeps logging success
* fix(bedrock): agree on a dropped realtime session's failure across loggers and the spend row
A realtime session's success event now reports status failure, with the same
error_information the spend row stores, when the session recorded a provider
failure in its metadata. Usage already streamed stays billed and the router
still counts one cooldown failure
The spend row builder now runs every row's error_information through
sanitize_error_information_for_spend_logs (DB size cap plus prompt redaction),
so the realtime failure rows get the same treatment as the proxy failure hook,
which no longer sanitizes on its own to avoid truncating twice
Drop the selection-failure tag on the committed-session refusal, since the
shared logging object already keeps a retried refusal out of cooldown, and pin
the router's realtime success counting with an OpenAI realtime deployment
* fix(bedrock): count realtime router successes only for sessions that logged one, and log first-turn drops
The router counts a realtime success only when the session's success handlers ran, so a refused handshake or an upstream close before any frame no longer offsets its failure. The success event's payload status stays success again, so each logger gets one failure record per dropped session, while the spend row still reads failure from the metadata the Bedrock handler records. A committed Bedrock session that fails before any logged event now gets one zero-cost Logs row. After an output failure the input stream is closed in the background, so the SDK body writer finishes without delaying the client's error. Every input send failure reports the same 503 stream-ended cause as an output drop
* fix(realtime): count a router success only for a session that ended cleanly
A realtime session the upstream closes with 1011 or 1006 after its first frame now marks itself as ended abnormally where that close is seen, in the shared relay and in the OpenAI Live handler, so the router no longer counts it as a cooldown success. The success callbacks still run, so the proxy keeps the budget and parallel-slot settlement it had
A Bedrock session that fails before Bedrock sends any usage now logs no success event, as on base. The spend-row failure fields go into a new metadata mapping instead of the request's own dict, validated with a pydantic TypeAdapter instead of an unchecked TypeIs guard. The handler tests drop a wall-clock bound and type their helpers
* fix(bedrock): log a dropped realtime session's usage from its own logging state
The failure event replaced the shared standard_logging_object while the
queued success dispatch was still running its callbacks, so loggers
received $0 and 0 tokens for a billed dropped session.
* fix(proxy): settle a rate-limit reservation once when a request logs a failure and a success
A dropped realtime session fires a failure event and then a success event for the same call. The v3 limiter refunded the TPM reservation on the failure and released it again on the success, so the key's TPM counter went negative. The success path now skips a reservation the failure already released and marks it settled for a later failure. The realtime handler test callbacks are typed
* fix(rate-limit): settle each token reservation once when a request logs both a failure and a success
A dropped realtime session logs a success for its usage and then a failure. After a success the
limiter now clears every reserved amount (TPM, PTU, ITPM, OTPM), so a later failure refunds nothing
twice. A new reservation written in the same context after a released one now starts clean, so
reserve, fail, reserve, succeed charges the second request's usage once instead of 78 TPM or
28 ITPM for 27 and 20 used
* fix(rate-limit): take a token hold before awaiting so concurrent success and failure callbacks settle it once
* fix(rate-limit): track the ITPM/OTPM hold's release apart from TPM so a nested pre-call keeps the request's project IO charge
* test(rate-limit): pin the split hold-release flags across nested MCP pre-calls
* fix(realtime): count a relay that fails on its own as an abnormal end
* test(realtime): pin the relay failure exit with a client frame the provider rejects
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(realtime): support latest OpenAI audio models
* fix(transcription): type SDK stream duration metadata explicitly
* fix(realtime): preserve transport settings and translation duration
* refactor(realtime): reuse SDK client initialization
* fix(audio): bill finalized streams and authorize transcription models
* refactor(audio): keep transcription pricing selection together
* fix(realtime): reconcile model metadata and test contracts
* fix(realtime): preserve provider duration and model capabilities
* fix(realtime): keep transcription validation compatible with cost map guard
* fix(realtime): address transcription and billing review findings
* fix(realtime): preserve input usage and provider routing
* fix(realtime): bill retained output and route Azure aliases
* fix(realtime): reject malformed provider durations
* fix(realtime): preserve batch cache rates and Azure v1 routes
* fix(transcription): reject nonfinite provider durations
* refactor(realtime): align annotations with current lint rules
* fix(realtime): preserve translation deployment pricing and input usage
* test(transcription): verify synchronous stream finalization
* fix(realtime): retain missing translation input usage at close
* fix(transcription): reject streams that bypass output guardrails
* fix(realtime): reconcile audio protocols billing and current CI
* fix(realtime): trust confirmed formats and authorize default models
* fix(realtime): preserve absent models and reject unsupported guardrails
* fix(realtime): reject translation policy pipelines
* chore(ci): rerun transient infrastructure timing failure
* fix(realtime): reconcile audio protocols with current upstream
* fix(audio): validate provider responses at SDK boundaries
* fix(realtime): preserve minimum OpenAI SDK compatibility
* fix(realtime): preserve deferred Live session authorization
* fix(realtime): preserve upstream authentication and transcription behavior
* fix(audio): scope provider duration precedence to configured models
* fix(typing): avoid Final declarations inside transcription cost loop
* refactor(audio): reuse transcription mapping and duration resolution
* fix(realtime): enforce model budgets and preserve translation routing
* fix(audio): log streamed transcripts whose done event reports detected languages
* fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers
Providers without a streaming transcription handler got UnsupportedParamsError
for stream=True, languages and keywords, where the merge base forwarded them as
provider-specific params. Only openai and azure validate them now.
* fix(audio): bill streamed gpt-transcribe from the provider's reported duration
The stream path computed cost after logging had replaced the transcription's
duration usage, so it fell back to the measured file length while the
unstreamed call billed the provider's seconds.
* fix(transcription): log streamed transcription usage from the provider response
The stream logging path built the spend payload from the response after
its duration usage was transformed away, so whisper-1 with stream=true and
streamed gpt-transcribe logged an empty usage object. Bill and log from the
provider's TranscriptionResponse in both the sync and async paths.
* fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes
- Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape
- Use the input transcription model as the session model only for transcription sessions
- Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main
- Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set
- Return x-litellm headers on streamed transcriptions and mark the request done
- Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure
- Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing
- Bill translation sessions on output audio seconds only, at the single published rate
- Document the 201 SDP answer on the translation calls routes and the transcription 200 response
* fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients
Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the
guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the
default dated version on the deployments route for every model. The SDK client secret keeps
session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata
keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two
unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the
gpt-transcribe format check only reads the request provider's cost map entry.
* fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it
Transcription reaches every provider as on main again, sends stream only when asked, forwards language and
languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the
new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a
mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from
one source and spend rows carry the billed seconds.
Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward
explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths
are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later
session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA
get beta-shaped session events, and transcription sessions send the deployment model in both shapes.
* fix(realtime): address review findings on transcription billing, streaming and realtime auth
Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata
* fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only
* fix(realtime): preserve prior merge resolutions after rebase
* fix(realtime): enforce translation policies and normalize Azure transcription URLs
* fix(cookbook): stop translation once output idles after the upload ends
* fix(realtime): bill translation on input audio and drop pre-read stream cost headers
Translation spend now uses the input audio seconds the session sent, priced
through input_cost_per_second, instead of the translated output duration.
A streamed transcription no longer returns cost headers before the stream is
read, since the cost is only known after the last event. Also removes an
unused realtime guardrail helper.
* fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields
Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop
* test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription
* fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA
Bill translation sessions from the deployment's own per-second rates, add per-second prices for
Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map
mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to
the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation
calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models
* fix(realtime): bill a translation rate the deployment omits from the session model
* test(e2e): check translated speech is not English instead of exact French wording
* fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices
Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending
* fix(realtime): keep the proxy's realtime session gate off translation sessions
A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects
* fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model
A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params.
* fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree
* fix(model-catalog): accept the realtime translation endpoints in the cost map
* fix(cookbook): close the translation session and read until session.closed
* fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard
An empty `stream` form value runs the transcription unstreamed again, as
OpenAI does, instead of answering 400. The four translate cost-map entries
drop the translation routes from supported_endpoints until the main
branch's catalog parser accepts them, and the nested transcription model
lookup uses the router registry entry point.
* test(proxy): drop the mid-stream error frame pin that main's error shape change replaced
* fix(transcription): bill uploaded seconds when a stream fails mid-transcript
* fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream
* fix(audio): keep transcription's positional arguments and redact streamed error details
- Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning
- Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do
- Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite
* fix(audio): stream the guarded transcript to SDK callers that asked for a stream
* fix(logging): keep a streamed transcript out of callbacks that turned off message logging
A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does
* fix(proxy): stream a transcription the deployment streamed when the client sent no stream field
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): send an https PDF image_url to Claude as a document
A PDF link sent as image_url to Claude on Anthropic went out as an image
block and Anthropic answered 400. A link whose path ends in .pdf is now
downloaded like Vertex links are, and the served Content-Type picks the
document or image block, so CDN links that serve a PNG from a .pdf path
keep working. An explicit format of application/pdf sends a url document
with no download. PDF media types match regardless of case or parameters,
and count_tokens picks the same block as the request.
* fix(anthropic): download https PDF links off the event loop
The proxy downloaded an https link ending in .pdf synchronously while
building the Anthropic request, so on an async call every other request
on that worker waited for the download. Those links are now inlined
before the request is built, the same way http links already are, and
the /v1/messages guardrail write-back does the same. A non-string
format on an image_url is ignored the way it already was inside the
image_url mapping, instead of raising an AttributeError.
* fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent
* test(anthropic): expect the PNG media type the served bytes carry at a .jpg path
* fix(anthropic): bound the PDF link download and keep query-string PDF names
Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the
client's 6000 s default, fall back to main's whole-URL extension check so
download?file=report.pdf still counts as a PDF, treat application/x-pdf as
a PDF, and set the url document's application/pdf format only for the
guardrail write-back so other /v1/messages providers send what main sends
* fix(anthropic): probe a .pdf link and send it to Claude as a url document
Instead of downloading a .pdf link and inlining it as base64, ask the host for
its first kilobyte and send the link as a url document when it serves a PDF.
Every other link goes out as the url image main sends, so robots.txt handling,
caching and the wire format stay as they are on main.
Also restore the guardrail adapter and the content type inference to main,
keep the client's url document blocks (title, context, citations,
cache_control) through the guardrail write-back, treat a PDF filename in the
query string as a PDF link, probe links concurrently on the sync path, cap the
async probe's read at the download limit, and catch only transport, URL and
SSRF failures in the probe
* fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place
The probe now streams the response and keeps the first 1 KB, so a host that
ignores Range no longer fills memory with up to 20 parallel bodies. The
guardrail write-back now matches url documents inside a tool_result and outside
one separately, since the write-back moves tool results ahead of other content
* fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke
* fix(anthropic): keep a PDF probe's error status for a minute, not ten
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A custom guardrail that records a value JSON can't encode, or a mixed-key dict in a list field, raised while
the guardrail headers were built and turned the provider's reply into a 500. The guardrail headers are now
left out on that response.
* feat(decisions): add Parallel AI web-grounded decisions
* fix(decisions): name the Parallel model decision
* fix(parallel_ai): restrict environment credentials to trusted decision endpoints
* fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence
Only fields a provider names reach the OpenAI response, so existing providers keep
their base shape. A malformed api_base answers 400 on both credential paths. The
complexity router classifier and the Playground handle a provider that omits
confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types
* chore(cost-map): declare the Decisions routes on parallel_ai/decision
The provider endpoint files are generated from the cost map, so the row must
name both routes for the generated System One flag to hold
* fix(decisions): report no classifier confidence when a provider omits it
A choice answer with no confidence is a valid classifier answer, so the
complexity router picks its tier and leaves classifier_confidence and the
confidence signal out instead of reporting a different metric in its place.
Also show the Parallel AI logo in the provider picker and add a Parallel
translation fixture captured from the live API for both Decisions routes.
* fix(decisions): keep a token-priced row on token billing when it also sets a query price
A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR
* fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route
A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions.
* fix(decisions): keep per-query billing and optional confidence on Parallel only
A query price on any other Decisions provider moved that provider off token
billing, and a choice or score answer with no confidence became a 200 with a
null for every provider. Both now apply to Parallel alone, so other providers
bill and fail as they do on main.
A Parallel deployment with explicit zero token rates and a query price read as
free, so every budget check was skipped while it billed per call. It now counts
as priced.
The dashboard key for Parallel is now PARALLEL_AI so a served bundle that
predates the provider still sends parallel_ai as the provider slug, and the
README table lists the provider.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): expand wildcard models for providers with model info but no cost map entry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): stop requiring an api key to list vllm models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): preserve model ids during wildcard expansion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): forward api key for model discovery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): support bearer auth for model discovery
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vllm): list hosted_vllm models from the server chat uses, with a bounded wait
The hosted_vllm listing now resolves HOSTED_VLLM_API_BASE and HOSTED_VLLM_API_KEY like hosted_vllm chat, instead of the /vllm pass-through's VLLM_API_BASE, and waits at most NON_LLM_CONNECTION_TIMEOUT rather than the completion timeout. vllm/* keeps listing nothing, since its chat runs in process. The router pass-through now forwards the deployment's api_key.
* fix(vllm): skip the placeholder key, honor deployment TLS and headers, and retry failed listings after a minute
* fix(discovery): cache only timed-out model listings, so a fast failure is tried again on the next call
* fix(proxy): list wildcard models off the event loop, so a hung provider no longer stalls other requests
* fix(utils): keep a concurrent call's model listing when another call times out
* fix(proxy): run model discovery on its own thread pool and keep credential lookups off the event loop
* fix(proxy): stop the file credential lookup at the first match before listing wildcard models
The fallback lookup now walks the allowed model names lazily, in the same
order as before, so a hung vLLM server is listed only when no configured
name serves the requested provider. The vLLM auth headers no longer
mutate the caller's headers dict.
* fix(proxy): try every configured wildcard route before listing any wildcard's models in the file credential lookup
* fix(proxy): answer model lists from cached listings first and share one listing per server
* test(vllm): give the self-signed cert fixture fixed validity dates
* fix(proxy): skip wildcard servers a key cannot reach on /v1/model/info
/v1/model/info listed every wildcard deployment and filtered by the key
afterwards, so a key limited to one model waited on every wildcard server and
sent listing calls to servers it cannot use. The listing now skips a wildcard
whose patterns match none of the key's models before it asks the server.
A key or team granted hosted_vllm/* with no deployment of that name now lists
no models, like openrouter/*, since chat answers 400 for every id the
environment server would list.
* fix(proxy): match a wildcard's own row pattern before asking its server
The /v1/model/info pre-filter matched a key's models against both the
deployment name and its litellm_params.model, so a key granted
hosted_vllm/* still asked a vllm-secure/* -> hosted_vllm/* server whose
rows it can never see. It now matches only the pattern expansion names
the rows with, picked by one shared helper.
* test(vllm): cover the vLLM listings and no-slash wildcard aliases the patch left unrun
Adds tests for hosted_vllm get_models with and without a server, vllm/*
never asking a configured server, and /model/info pre-filtering of
wildcard aliases whose name has no slash. Drops a test comment that
only repeated a method signature.
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): reproduce comparison response ID ownership and wire failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): guard stale retries and malformed comparison envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): authorize and unwrap comparison response ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): require comparison IDs to follow response security policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): deny raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): reject raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): enforce managed ownership with unmanaged ID opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost-map): give the bare gpt-4o, gpt-4o-mini, gpt-4.1 and gpt-4.1-mini aliases their 2027-01-14 shutdown date
* fix(proxy): report a deployment's own provider retirement date before the bare model's
An Azure deployment declared as model gpt-4o with custom_llm_provider azure, or as
azure/<deployment> with base_model gpt-4o, now resolves azure/gpt-4o's date instead of
the bare OpenAI row's. The GPT-4 series alias test now reads both cost maps with typed rows
* fix(proxy): read Azure's retirement schedule for azure_ai deployments
An azure_ai deployment with no azure_ai row of its own now reports the azure/<model> date, and a cost-map row wins over the provider-less copy the router registers under the deployment's key, which carries the bare model's date.
* fix(proxy): let custom and registered base_model rows keep their deprecation date
Only the copy the router writes at startup, which has no provider and inherits the bare row's date, gives way to a later candidate
* fix(proxy): report a pinned snapshot's own earlier deprecation date over its base_model's
* fix(proxy): recognize the router's copied cost-map rows by the loaded catalog
* fix(proxy): read Azure's schedule for a provider-prefixed azure_ai base_model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The opt-out check reran should_run_guardrail on metadata alone, so a streaming-only or tag-scoped global
guardrail looked opted out and skipped the Gemini body. It now reads the key's and team's opt-out fields.
A list input of Responses messages now reaches legacy Lakera instead of being dropped, and a pass-through
body with no messages or input skips the empty vendor call. Lakera post_call on /gemini has nothing to
read, so the rig's Gemini block cells use a generic post_call guardrail. Moves the regression cells into
each guardrail's own test file.
* fix: Return Anthropic error envelopes on Messages auth failures
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through
The pass-through relays upstream errors in Anthropic's shape, but the proxy's
own refusals there (bad key, model access, budget) still answered the OpenAI
body, so an Anthropic SDK pointed at /anthropic could not read them
* fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes
A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the
Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The
central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route.
The dashboard playground toast titles Anthropic error types when the body has no code.
* fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes
Error bodies on the Messages routes now carry error.litellm_error_type when the
LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep
causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error
per Anthropic's errors page, model-list requests that send anthropic-version get
the Anthropic envelope, and the admission, decompression and request-size
middleware errors use it on these routes too.
* test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests
* fix(proxy): keep handler error bodies unchanged and timeouts as api_error
Errors the Messages and count_tokens handlers already returned in the Anthropic
shape keep their original bodies, so litellm_error_type now appears only on
errors the proxy raises before the handler. A 408 stays api_error as before,
and the playground toast titles the new Anthropic types and ignores long
numbers in front of an embedded error
* fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies
The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error,
other unlisted 4xx invalid_request_error) now applies only to errors the proxy
raises before the handler. Errors inside the Messages handler, streamed error
frames and the SDK streaming adapters keep the merge base's table.
The dashboard toast reads a status before embedded JSON and titles by Anthropic
type only when the body is an Anthropic error envelope, so other pages keep
their titles.
* test(anthropic): mark the new test variables Final
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Every MCP route (/mcp, /<server>/mcp, /toolset/<name>/mcp, /mcp/proxy) ran LiteLLM auth before looking at the HTTP method, so a bare OPTIONS probe got the RFC 9728 401 OAuth challenge and clients that probe with OPTIONS looped on re-auth. The Streamable HTTP handler now returns 204 with Allow: GET, POST, DELETE, OPTIONS before auth runs. GET, POST and DELETE are unchanged.
Resolves LIT-10231
* fix(proxy): record the route's call type on failed speech spend rows
A failed /v1/audio/speech request never puts a logging object on the request
data, so the failure lift returned nothing and the spend row's call_type came
out empty (or whatever the client sent in the body). Without a logging object,
lift the call type the route already resolved, so a failure row records the
same call type a success row on that route records
* fix(proxy): keep the call type of a failure logged under its own route
An MCP tool listing that fails inside a Responses, chat, or messages call
is logged under /mcp/list_tools, so its spend row keeps list_mcp_tools
instead of taking the outer request's call type
* fix(proxy): take a failed row's call type from its endpoint, not a route path guess
* fix(proxy): log the endpoint call type on failed image generation and rerank rows
* fix(proxy): estimate failed-call ledger usage from the route's call type again
The failure usage snapshot now gets the same call type the spend row uses for
its estimate, so a failed Responses call counts input from the newest
compaction item again, as on main.
The endpoint call type tests now go through real routers whose provider
answers 400, instead of asserting on a mocked failure hook.
* test(proxy): type the failed spend row tests and fixture
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
