* feat(decisions): add EmpirioLabs AI as a decisions provider
empiriolabs/aplomb-1 now works with litellm.decisions and litellm.systemone, the proxy /v1/decisions and /v1/systemone routes, the decision model guardrail and the Admin UI. Requests go to POST /v1/decisions on the EmpirioLabs AI API with the System One body, keyed by EMPIRIOLABS_API_KEY with an optional EMPIRIOLABS_API_BASE
Chat models keep routing through the existing OpenAI-compatible JSON provider, and a test pins that
* ci: run the EmpirioLabs AI decisions tests in the llms unit shard
* fix: align EmpirioLabs registration with CI gates
---------
Co-authored-by: Adam Dalloul <47503782+Adam-Dalloul@users.noreply.github.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(accounting): replace event scaffolding with SpendLogs projection
* test: preserve spend timestamps without removed capture callback
* fix: guard accounting event removal across schema update paths
* refactor(accounting): retain schema and prepare durable payloads
* fix(accounting): preserve dates in serialized payloads
* fix(accounting): retain schema push compatibility
* fix(accounting): retain span registration for the legacy schema
* test(accounting): remove obsolete capture acknowledgement test
* fix(accounting): date every daily projection row the way the model rollup does
A space-separated startTime kept the whole timestamp as the daily key, while the model usage rollup took the calendar date. Every daily row now uses the same date helper
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore(ui): regenerate dashboard API types after rebasing on main
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(accounting): project only from the stored accounting payload
project() now reads entities only from the payload's accounting_entities, so
the raw SpendLogs fallback branches are gone. build_accounting_payload() builds
the metadata once instead of dumping it to JSON and parsing it back. Guardrail
usage dates use a tolerant UTC parse, so a malformed startTime drops only the
guardrail rows instead of failing the whole projection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(e2e): retire the quiet_stack lock
quiet_stack held an exclusive host-wide lock for six tests, draining every
other xdist worker while they ran. On the full e2e lane that left 35-38% of
worker time idle. Each of those tests now lives where it can run without
pausing the suite
- the two memory tests move to perf/ behind the perf marker, so the perf lane
runs them on its own fresh stack. Their refusing deployments use a literal
key because the perf stack carries no provider keys
- the stored-request size check from the memory test becomes an integration
test on an owned proxy that stores prompts
- the discount and margin spend-row tests are dropped from e2e. Their
breakdown assertions join the existing integration pricing test, which
already owns the discount and margin repricing flow
- the Bedrock guardrail cost test stays in e2e without the pricing reset
fixture, since nothing else on the stack writes pricing config now
- the MCP semantic tool filter test turns on a proxy-wide setting, so it gets
a new proxy_wide_config opt-in marker and runs after the main run
* test(e2e): register the claude_code deployments as one batch
The claude_code session fixture registered its 24 deployments one at a time on every xdist worker, and each one waited out the 15 s propagation window on its own, about 6 minutes per worker. ProxyClient.register_models posts the whole batch, polls each model until every replica serves it, then waits out propagation once from the last write. A model that fails to post or never becomes servable comes back as ModelNotRegistered, and an unservable one is deleted, so the fixture keeps reporting per-deployment failures the way it did
* fix(agentic): surface a failed agentic follow-up call instead of the pre-loop tool call
* fix(agentic): surface failed follow-ups from the OpenAI chat dispatcher and update pinned integration tests
* fix(agentic): end a capped agentic turn on chat and responses instead of a retried 500
* fix(openai): run agentic hooks after the first call's error handling so a follow-up error keeps its own type and message
* fix(agentic loop): end capped turns on every chat path and bill the first call when a follow-up fails
A turn refused at max_agentic_loops now drops the internal tool calls of every hook that claims calls in it, while a call the client declared survives. The hosted_vllm and openai chat paths now carry the loop depth, fingerprints and max_agentic_loops into the follow-up, so they stop at the cap instead of looping forever.
A failed follow-up call now records the first call's usage and cost on the request's failure log before the error reaches the client. A code interpreter sandbox that is missing or fails becomes the tool call's output for the model to handle, so the turn completes without a retry.
On native streaming /v1/messages a hook or follow-up failure now reaches the client as an Anthropic error event with the mapped type, and compresr declares its retrieve tool so the stream holds it back. Comments and docstrings the branch added against the repo rule are removed
* fix(agentic loop): bill retried attempts and clean up what failed or capped turns return
The router's retry loop now carries each failed attempt's paid first call into the request's billing, so a request whose follow-up keeps failing bills every first call on its failure row, and one that recovers bills the failed attempt's first call on its success row. A failure row no longer adds unpriced estimated tokens on top of attempts that were billed, so its tokens match its spend
A sandbox failure now answers the model with a fixed sentence and marks the code_interpreter_call as failed, without the sandbox's error text. Code interpreter on /v1/responses with Claude now runs its loop, because the bridged inner call no longer strips the hook's own markers
Dropped internal tool calls leave the kept calls numbered from 0, also on a replayed stream. A capped native /v1/messages stream now says the loop hit its limit and that max_agentic_loops raises it, instead of asking the client to retry
The agentic refusal and hook error logs no longer include the model or the exception text, and the agentic loop helpers take typed messages, params, and read-only fingerprints
* fix(agentic loop): keep a recovered retry's cost breakdown on the response it returns
A retried agentic request that recovers no longer adds the failed attempt's first call to its success row. The proxy keeps the follow-up's own row for that request id, so the added spend never showed up in the stored row, and that duplicate row is tracked separately. When every attempt fails, the failure row still bills each attempt's first call
Pricing a first call for the failure row no longer replaces the request's cost breakdown. Before, a recovered retry answered with x-litellm-response-cost-original, -input, and -output from the failed attempt's first call. Now its headers and logged breakdown match a request that succeeds on its first try
* fix(agentic): stop retrying a request whose agentic follow-up failed and bill every round it served
An error raised by an agentic follow-up after the first call was served now carries a typed mark, so the
router's retry loops, the mid-stream retry, and litellm's num_retries wrapper raise it as is instead of
rerunning the whole request. A failure of the first call itself still retries, and fallbacks still run.
The failure row bills every round the loop served before the failure, at any depth, and a later attempt
that succeeds bills what the failed attempts paid on its success row, with the cost breakdown and the
response cost headers adding up. The error response's x-litellm-response-cost reports the same amount
its failure row bills. Failure rows outside agentic hooks keep the prompt-token estimate they had before.
A refused tool call now matches by id among the refused ids, or by name among refused calls that carry
no id, so a name-only refusal next to an id refusal is no longer kept.
* fix(code_interpreter): delete a session sandbox whose code run failed so the next call starts a fresh one
* fix(code_interpreter): delete only the session sandbox that failed, never its replacement
* test(code_interpreter): type the session sandbox test helpers
* fix(proxy): report what a failed agentic follow-up's failure row bills on /v1/messages errors
* fix(agentic): keep the no-retry mark through error mapping and bill only failed follow-ups on success
An agentic loop error that is not a provider error, such as the streaming chat cap or a crash in a hook's
plan build, now keeps its no-retry mark when litellm maps it to an APIConnectionError, so the router no
longer reruns every paid round. The mark is stored on the exception, so a built-in error can carry it too.
A crash in a hook's plan build on /v1/messages and /v1/responses bills the first call on its failure row.
A recovered attempt adds a failed attempt's usage to its success row only when that attempt was a failed
agentic follow-up. Other failed attempts, such as a dropped stream, log their own row as on main.
A code interpreter sandbox that fails while running the code now says the code may have run but its
result is lost, instead of saying the code did not run.
* test(responses): bill a retried MCP request each attempt on its own row again
Round 4 stopped adding a non-agentic failed attempt to the success row, so this test goes back to main's assertions: the success row bills only its own rounds and the failed attempt's first round stays on its own failure row
* fix(spend): name a failed follow-up attempt the success record already bills
With the usage meter on, a failed agentic follow-up attempt gets its own charge-eligible record, and the request's success record also carries that attempt's usage and cost once the success row bills it. The success record now lists the attempt's record ids in included_attempt_ids, so a reader can tell the success record already covers it. Nothing changes when the meter is off
* fix(spend): count a billed failed follow-up's tokens on the success row
* fix(spend): bill a failed follow-up on a fallback served from cache
* fix(spend): show a billed failed follow-up in a cached fallback's cost breakdown
* fix(spend): keep a failed follow-up's billing state off the request-merged call details
* fix(spend): record the cost breakdown of a successful agentic follow-up
The chat agentic loop returned the follow-up response with the cost its own internal call had already stamped, so the request's logging reused that cost and never recorded cost_breakdown on the spend row. The loop now drops that stamped cost unless the follow-up was served from the cache, and the request prices the response itself
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): answer a 503 as 529 overloaded_error on /v1/messages
Anthropic's error contract has no 503, so a client built against it (Claude Code) treats a 503 from /v1/messages as a plain 5xx and retries it ten times instead of taking its overload path. A Bedrock 503 now leaves /v1/messages and /v1/messages/count_tokens as 529 overloaded_error, for both a JSON answer and a stream that fails before its first byte.
An Anthropic 529 overloaded_error now maps to litellm.ServiceUnavailableError (503) instead of InternalServerError (500), so OpenAI-shaped routes answer it as 503, the status OpenAI documents for an overloaded engine, and router retry and allowed-fails policies key it as ServiceUnavailableError.
* test(proxy): expect overloaded_error on a 503 stream error after a keepalive ping
* fix(proxy): record the 529 the Anthropic routes answer and type the chat stream error frame
The messages route stamps the answered status on the exception before post_call_failure_hook runs, so the spend log row, the OTel server span, the failed-request metric, and alerting record the 529 the client saw instead of the deployment's 503, while the deployment failure metric and cooldown keep the deployment's own status. A chat completions stream error frame names the type its status stands for instead of the null .type ServiceUnavailableError carries. An Anthropic-shaped 503 envelope passes through as 529 overloaded_error through a validated TypedDict, and count_tokens answers every ProxyException as the Anthropic envelope. Five integration tests that pinned 503 on the messages route now expect 529
* test(proxy): expect the top-level Anthropic envelope from count_tokens errors
count_tokens now answers a ProxyException as a JSONResponse carrying the
Anthropic error envelope instead of raising HTTPException, which FastAPI
wrapped in {"detail": ...}. The three legacy error-format tests now read
the response body and check that no detail wrap is left.
* test(proxy): cover the 529 overloaded_error answer across routes, cooldown, and retry budgets
* fix(anthropic): keep Anthropic overloads as InternalServerError and answer 529 from the provider status
An upstream Anthropic 529 stays litellm.InternalServerError (status 500) so
InternalServerErrorRetries, InternalServerErrorAllowedFails, SDK handlers,
and the OpenAI-shaped routes behave as on main. The error now carries
provider_status_code, and the /v1/messages routes answer 529
overloaded_error from it.
* refactor(proxy): unwrap a mid-stream fallback error once instead of recursing for the provider status
* fix(anthropic): keep count_tokens' detail body, end keepalive streams with an Anthropic error event, and log any 529 as an overload
* test(integration): cover the audit round 3 cells and expect 529 on the count_tokens 503 cases
* fix(proxy): re-raise an unchanged stream HTTPException as itself and walk nested mid-stream fallbacks for the provider status
* fix(proxy): walk nested mid-stream fallbacks without recursion
* fix(anthropic): preserve 503s and answer upstream 529s on messages routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): answer an upstream Anthropic 529 as 529 on count_tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): use integration coverage for keepalive errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(cost-map): remove the 52 rows whose provider retired them before 2026
* test(utils): pin the proxy function-calling parity case to azure/gpt-4.1
* test(batches): price the Azure batch-logging regression case against gpt-4.1
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(credentials): preserve stored info when patching credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(credentials): type patch test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep a deployment's token limits off the shared backend cost map key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): preserve catalog token limits on shared keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): use catalog for shared token limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drop the class attribute patch from the deployment max_tokens test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): avoid provider auth during catalog key resolution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The MCP SDK server answers a notification POST with 202 before it hands the notification to the session. When the gateway sends DELETE right after `notifications/initialized`, the session can close in that gap, and the SDK logs an ASGI error that fails the test at teardown. Wrap the upstream with the existing `_acknowledge_notifications_after_dispatch` helper the other SDK peers already use, so the 202 only goes out once the session has the notification
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): add LITELLM_FIPS_MODE startup gate
With LITELLM_FIPS_MODE=true the proxy refuses to start when TLS verification
is off, when Python's OpenSSL does not enforce FIPS, or when the startup
password migration fails. A value that is not true/false also refuses.
* feat(proxy): refuse FIPS mode when cryptography's OpenSSL is not FIPS
The stock cryptography wheel bundles its own non-FIPS OpenSSL, which serves
JWT, Fernet and AES-GCM even when Python's OpenSSL enforces FIPS.
* fix(proxy): refuse FIPS mode when cryptography's FIPS provider cannot hash
The stock cryptography wheel reads the host's FIPS config and reports FIPS on,
but its bundled OpenSSL cannot load the host's FIPS provider, so every
operation fails. Require a working SHA-256 alongside the FIPS flag.
* fix(proxy): refuse TLS verification off anywhere in FIPS mode
A model's own ssl_verify: false reached the HTTP client after the startup check
passed. get_ssl_verify now raises in FIPS mode whenever verification resolves
to off, which also covers the startup check.
* refactor(proxy): return FIPS boot refusals as values and harden the FIPS tests
* fix(proxy): stop refusing TLS verification off in FIPS mode
Skipping certificate verification is a security choice, not a FIPS one:
the TLS connection still only uses algorithms the FIPS provider allows.
* feat(fips): refuse a database used without FIPS mode, drop the password migration change
FIPS mode marks a new database with a fips_mode row in LiteLLM_Config and refuses one that already holds
LiteLLM rows without it. Without FIPS mode, the proxy refuses a database carrying the marker.
* test(fips): close the database test clients with aclose so the engine stops
* feat(fips): read FIPS mode from general_settings.fips_mode instead of LITELLM_FIPS_MODE
* refactor(fips): keep fips_mode out of the API-writable general settings, it is read from the config file only
* fix(fips): retry the startup marker and state reads once after a dropped connection
A dropped connection during the non-FIPS marker read used to count as an unmarked
database, so a blip at boot let a non-FIPS proxy start on a FIPS database. Both reads
now go through call_with_db_reconnect_retry, the reconnect-and-retry the other startup
reads use. A read that still fails keeps its old outcome.
* fix(fips): refuse a fips_mode key that has no value
fips_mode: os.environ/X with X unset, or a bare fips_mode: / null, resolved to None and
started without FIPS mode and without a message, while an empty string refused. A key
that is present now has to be a boolean; only a config without the key means off.
* refactor(fips): move the database check out of the boot gate
The boot gate keeps the setting and the two OpenSSL checks. The check that a
database matches the FIPS setting moves to its own PR, so a boot without the flag
runs no extra database read.
* test(fips): pin how a fips_mode string is read through enforce_fips_mode
* fix(proxy): omit cost headers for unpriced responses
* fix(proxy): preserve declared free response costs
* refactor(proxy): share the cost map pricing lookup for the response cost header
The header check now reuses the same raw entry, fill-missing and rule lookup that
cost model selection uses, so the two can't drift apart. It also counts every declared
rate field, so a free model priced only per request keeps its 0.0 header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(logging): keep the caller's call_type when chat completions are bridged to the Responses API
* fix(logging): log bridged chat completions with the chat response they returned
* fix(logging): log bridged text completions and /v1/messages as served, with the request's options
Bridged rows were translated without the request's options, so structured
output and replay fields could differ from what the client got. Text
completions and non-streamed /v1/messages through the chat bridge also kept
a Responses API object or the encoded id.
* fix(logging): redact reasoning summaries on bridged rows and log bridged text completions as text completions
Redaction now drops reasoning_items on chat messages and deltas, so a bridged row keeps no reasoning summary when message logging is off. A non-streamed bridged text completion logs the text completion the client got, priced like the chat response it came from. The bridge handler's input TypedDict fields are ReadOnly.
* fix(logging): redact bridged text completions for opted-out callbacks and pin the sync logged shape and usage ack
* test(integration): look up bridged chat spend rows by the id the client got
Bridged chat spend rows are keyed by the client-facing id and logged with the caller's call type, so the helpers that decoded the old resp_<base64> key or pinned call_type responses now assert the new contract
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cache_control): read the routed deployment's own prompt-cache flag, not a sibling's shared cost-map entry
Two deployments that share one model string share one key in the live
litellm.model_cost map, and router registration writes each deployment's
supports_prompt_cache_breakpoint into that key. The cache control hook read
the live map, so a flagless deployment picked up a sibling's opt-in and sent
prompt_cache_options and prompt_cache_breakpoint upstream, which the provider
rejects with a 400
When a call carries the routed deployment's model_info, the prompt cache
checks now use its declared supports_prompt_cache_breakpoint, and fall back
to the loaded catalog instead of the live map. This covers the hook on
completion, acompletion, Responses and the /v1/messages route, the azure_ai
chat transform, the OpenAI Responses transform under drop_params, and the
/v1/messages to Responses adapter
* fix(prompt caching): read each deployment's own breakpoint flag in the openai and bedrock readers
A deployment's own supports_prompt_cache_breakpoint now picks OpenAI's
prompt_cache_breakpoint only where the request goes out OpenAI-shaped on a
provider other than openai, such as azure_ai OpenAI models and bedrock_mantle.
Bedrock on every route and providers that build Anthropic messages keep
cache_control, which Bedrock Converse turns into cachePoint
The openai chat reader now passes the routed model_info and checks base_model.
The Bedrock cache point reader takes the deployment's flag first and, for a
routed call with no flag, reads the catalog as loaded instead of the live map
that sibling deployments write into. model_info is threaded from the converse
transform, the factory message and tool builders, and the responses bridge
* fix(prompt caching): keep gateways on cache_control and read register_model opt-ins on routed calls
A declared supports_prompt_cache_breakpoint picks OpenAI's breakpoint only on providers hosting OpenAI's API or for a model the cost map lists under the provider, so OpenRouter, LiteLLM proxy, and vLLM deployments keep cache_control as before. Routed calls read the loaded catalog plus the user's own register_model entries, never router registrations.
* fix(prompt caching): honor a deployment's own supports_prompt_caching on Bedrock and its declared litellm_provider on gateways
* fix(prompt caching): keep wildcard cache flags off matched models, read deployment flags for auto caching and base_model aliases
* test(anthropic): cover auto caching on an unmapped Claude deployment
* fix(cache_control): keep the merge base's reads where the per-deployment flag changed them
A register_model entry under a provider-prefixed name reaches routed calls again, since the
overlay now resolves the bare and prefixed names the way runtime registrations are stored.
The hook and the openai chat reader no longer judge base_model, as at the merge base, so a
client base_model sent under allow_client_pricing_override cannot pick OpenAI's breakpoint
and get a 400.
A Bedrock deployment flag that is not a real true reads as false again, as at the merge base.
An azure_ai wildcard's supports_prompt_cache_breakpoint speaks for each model it matches again,
as at the merge base. Other wildcards still leave both cache flags to the catalog.
The s10/s11 Bedrock wire test asserted that a flagless deployment inherits its sibling's flag
on a shared ARN, which is the leak this PR removes. It now asserts that each deployment keeps
its own flag and that a flagless unmapped ARN keeps its cache points after the delete.
* fix(cache_control): let the later register_model entry win between a bare and a provider-prefixed name
Routed calls read register_model entries through an overlay that merged the bare
name after the provider-prefixed one whatever order they were registered in, so a
later opt-out under one name lost to an earlier opt-in under the other. Each
registration now moves its key to the end of the kept registrations, and the
overlay merges both names oldest first, which matches what SDK calls read from
the live model map.
Also lists the wildcard cache flag helper in the router coverage script, since
the hook's wildcard tests drive it rather than calling it by name.
* fix(cache_control): keep register_model entries visible while a cost map reload replays them
The previous commit moved a re-registered name to the end of the saved
registrations by removing and re-adding it, and the overlay walked that dict to
order a bare and a provider-prefixed name. A reload replays every saved entry,
so a request on another thread could briefly miss an entry or fail on a dict
that changed size mid-walk. Entries are written in place again, a per-name
sequence records which registration came last, and both the overlay and the
reload replay order by it, so the later registration still wins after a reload.
* fix(cache_control): read a string or numeric supports_prompt_caching as the bool it spells and find register_model entries under any route prefix
A deployment flag stored as "true", 1, "false" or 0 (POST /model/new and quoted YAML keep strings) now means that bool on the Bedrock cachePoint reader and Anthropic auto caching, and any other value falls to the catalog. register_model records which names wrote to each catalog row, so an opt-out under bedrock/ or bedrock/converse/ reaches routed calls like one under anthropic/.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(realtime): support latest OpenAI audio models
* fix(transcription): type SDK stream duration metadata explicitly
* fix(realtime): preserve transport settings and translation duration
* refactor(realtime): reuse SDK client initialization
* fix(audio): bill finalized streams and authorize transcription models
* refactor(audio): keep transcription pricing selection together
* fix(realtime): reconcile model metadata and test contracts
* fix(realtime): preserve provider duration and model capabilities
* fix(realtime): keep transcription validation compatible with cost map guard
* fix(realtime): address transcription and billing review findings
* fix(realtime): preserve input usage and provider routing
* fix(realtime): bill retained output and route Azure aliases
* fix(realtime): reject malformed provider durations
* fix(realtime): preserve batch cache rates and Azure v1 routes
* fix(transcription): reject nonfinite provider durations
* refactor(realtime): align annotations with current lint rules
* fix(realtime): preserve translation deployment pricing and input usage
* test(transcription): verify synchronous stream finalization
* fix(realtime): retain missing translation input usage at close
* fix(transcription): reject streams that bypass output guardrails
* fix(realtime): reconcile audio protocols billing and current CI
* fix(realtime): trust confirmed formats and authorize default models
* fix(realtime): preserve absent models and reject unsupported guardrails
* fix(realtime): reject translation policy pipelines
* chore(ci): rerun transient infrastructure timing failure
* fix(realtime): reconcile audio protocols with current upstream
* fix(audio): validate provider responses at SDK boundaries
* fix(realtime): preserve minimum OpenAI SDK compatibility
* fix(realtime): preserve deferred Live session authorization
* fix(realtime): preserve upstream authentication and transcription behavior
* fix(audio): scope provider duration precedence to configured models
* fix(typing): avoid Final declarations inside transcription cost loop
* refactor(audio): reuse transcription mapping and duration resolution
* fix(realtime): enforce model budgets and preserve translation routing
* fix(audio): log streamed transcripts whose done event reports detected languages
* fix(audio): pass stream, languages and keywords through unchecked for non-OpenAI transcription providers
Providers without a streaming transcription handler got UnsupportedParamsError
for stream=True, languages and keywords, where the merge base forwarded them as
provider-specific params. Only openai and azure validate them now.
* fix(audio): bill streamed gpt-transcribe from the provider's reported duration
The stream path computed cost after logging had replaced the transcription's
duration usage, so it fell back to the measured file length while the
unstreamed call billed the provider's seconds.
* fix(transcription): log streamed transcription usage from the provider response
The stream logging path built the spend payload from the response after
its duration usage was transformed away, so whisper-1 with stream=true and
streamed gpt-transcribe logged an empty usage object. Bill and log from the
provider's TranscriptionResponse in both the sync and async paths.
* fix(realtime): keep main's routing, protocol and transport behavior around the new translation routes
- Send OpenAI realtime and translation client_secrets and calls through litellm's httpx client again, so ssl_verify and custom CA settings apply and upstream errors keep main's shape
- Use the input transcription model as the session model only for transcription sessions
- Drop the new /openai/v1 translation aliases and record each lazy feature's own routes, so the /openai pass-through and the MCP routes resolve as on main
- Let a configured Azure realtime_protocol win for every model, with GA as the default only when none is set
- Return x-litellm headers on streamed transcriptions and mark the request done
- Refuse translation sessions before connecting upstream when realtime guardrails apply or the provider is not OpenAI or Azure
- Accept json and text for gpt-transcribe, and drop other formats under drop_params instead of failing
- Bill translation sessions on output audio seconds only, at the single published rate
- Document the 201 SDP answer on the translation calls routes and the transcription 200 response
* fix(realtime): restore main behavior for guardrails, Azure transcription, and realtime clients
Streamed transcription runs unstreamed when a deployment post_call guardrail applies, so the
guardrail still sees the transcript. Azure transcription maps v1, latest and preview to the
default dated version on the deployments route for every model. The SDK client secret keeps
session.model ahead of model, realtime WebSockets connect to api_base again, realtime metadata
keeps only guardrails, and managed agents need an explicit transcription model. Reverts the two
unrelated streamed Responses error codes and the duplicate cached_tokens_details field, and the
gpt-transcribe format check only reads the request provider's cost map entry.
* fix(realtime): keep main's transcription, client secret and route behavior where the PR changed it
Transcription reaches every provider as on main again, sends stream only when asked, forwards language and
languages together, keeps form key order, bills provider seconds only when above zero, honors base_model for the
new flags, keeps main's Azure api_version on the proxy path, and streams carry chat's headers, an error event on a
mid-stream failure, and a spend row when the stream ends early. Translation cost reads both per-second rates from
one source and spend rows carry the billed seconds.
Realtime transcription WebSockets keep model in the upstream URL outside OpenAI's hosts, client secrets forward
explicit nulls, accept conversation and delay, and answer 400 on a malformed body, the /openai translation paths
are back to main, translation refuses only for guardrails realtime sessions run, nested aliases resolve and later
session.update models are authorized, translation client secrets run per-model auth, beta clients bridged to GA
get beta-shaped session events, and transcription sessions send the deployment model in both shapes.
* fix(realtime): address review findings on transcription billing, streaming and realtime auth
Per-second pricing replaces token billing only when the priced entry sets no token rates, translation rates fall back per key to the cost map, transcription cache keys include languages and keywords, a stream request a post_call guardrail forces unstreamed comes back as redacted SSE, Azure beta clients are bridged only for GA-only deployments, nested transcription models on the realtime WebSocket are checked against the key, and translation spend carries key and team metadata
* fix(azure): pick the GA realtime protocol for deployments whose model_info.base_model is GA-only
* fix(realtime): preserve prior merge resolutions after rebase
* fix(realtime): enforce translation policies and normalize Azure transcription URLs
* fix(cookbook): stop translation once output idles after the upload ends
* fix(realtime): bill translation on input audio and drop pre-read stream cost headers
Translation spend now uses the input audio seconds the session sent, priced
through input_cost_per_second, instead of the translated output duration.
A streamed transcription no longer returns cost headers before the stream is
read, since the cost is only known after the last event. Also removes an
unused realtime guardrail helper.
* fix(realtime): drop top-level session nulls, bill dropped transcription streams, and bridge beta response fields
Client secret, translation and transcription session bodies drop top-level nulls again, as main did, while nested nulls such as turn_detection still pass through. Transcription client secrets send the deployment model as session.model. A streamed transcription the client drops still runs its success callbacks, and the stream ends with data: [DONE] like OpenAI's. The beta to GA bridge maps max_response_output_tokens, speed and noise reduction, remaps response.create, and drops temperature under drop_params. The Entra token for realtime HTTP routes is fetched off the event loop
* test(e2e): cover realtime translation, live transcription, client secrets, and streamed transcription
* fix(realtime): close review gaps in translation billing, transcription streams, and Azure GA
Bill translation sessions from the deployment's own per-second rates, add per-second prices for
Azure gpt-4o transcribe models, keep stream-scoped guardrails seeing the caller's stream flag, map
mid-stream transcription errors to LiteLLM exceptions with a [DONE] frame, map GA responses back to
the beta shape for bridged beta clients, honor the minting key's guardrail opt-out on translation
calls, restore the Azure transcription session URL, and flag GA-only Azure realtime models
* fix(realtime): bill a translation rate the deployment omits from the session model
* test(e2e): check translated speech is not English instead of exact French wording
* fix(realtime): keep main's behavior for non-bridged backends, stream error frames, gateway URLs and Azure transcribe prices
Remap beta events to GA only for bridged beta clients and translation sessions, bridge beta clients on GA-pinned Azure deployments, log error-only upstream sessions as failures, flag dated Azure GA realtime models, drop the unpublished Azure per-second transcribe rates, log a mid-stream transcription failure once, keep main's SSE error frame outside transcription, drop unsupported transcription params under drop_params, keep the caller's Azure auth headers, keep a gateway /v1 path, let key opt-outs release string-callback guardrails on translation, and keep the cookbook waiting while a long upload is still sending
* fix(realtime): keep the proxy's realtime session gate off translation sessions
A key that opts out of a legacy callback guardrail now gets a working translation session, since the transcription-guardrail gate no longer sends a realtime session.update the translation upstream rejects
* fix(realtime): keep websocket model limits, bill measured transcription length, route transcription sessions on the allowed model
A WebSocket opened without a query model now authorizes as an empty model, transcription cost prefers the measured upload length over the provider's reported seconds, transcription sessions overwrite nested models with the authorized top-level model again, and a non-boolean transcription stream value answers 400 or is dropped under drop_params.
* fix(ci): copy the rust bridge extension into the openapi docs gate's base worktree
* fix(model-catalog): accept the realtime translation endpoints in the cost map
* fix(cookbook): close the translation session and read until session.closed
* fix(realtime): treat an empty transcription stream as unset and pass the cost-map guard
An empty `stream` form value runs the transcription unstreamed again, as
OpenAI does, instead of answering 400. The four translate cost-map entries
drop the translation routes from supported_endpoints until the main
branch's catalog parser accepts them, and the nested transcription model
lookup uses the router registry entry point.
* test(proxy): drop the mid-stream error frame pin that main's error shape change replaced
* fix(transcription): bill uploaded seconds when a stream fails mid-transcript
* fix(proxy): bill uploaded seconds when a streamed transcription breaks mid-stream
* fix(audio): keep transcription's positional arguments and redact streamed error details
- Move the new languages, keywords and stream options to the end of transcription's parameters, so a prompt or format passed by position keeps its meaning
- Redact internal hostnames, paths and tracebacks from a streamed transcription's mid-stream error event, as chat streams already do
- Move the CA-bundle check for the realtime HTTP routes, which serves real TLS on loopback, from the unit suite to the SDK integration suite
* fix(audio): stream the guarded transcript to SDK callers that asked for a stream
* fix(logging): keep a streamed transcript out of callbacks that turned off message logging
A streamed transcription's assembled response reached success callbacks with its text intact under turn_off_message_logging or a callback's message_logging=False, because the shared streaming redactor had no transcription branch. It now redacts the text, as the unstreamed call already does
* fix(proxy): stream a transcription the deployment streamed when the client sent no stream field
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(videos): serve custom_provider_map BaseVideoConfig providers on /v1/videos
A provider registered in custom_provider_map with a BaseVideoConfig handler now resolves on every
video call type (create, status, content, list, remix, edit, extension, characters), the same way
CustomLLM handlers already resolve on /v1/images/generations. The video entry points and the HTTP
handler stop casting the provider name to LlmProviders, which raised before any lookup ran, and a
provider without a video config gets a 400 instead of a 500.
* fix(videos): type the custom provider lookups so the type gate holds
Pick the CustomLLM for a custom provider through ProviderConfigManager.get_custom_llm_handler,
which narrows the now wider custom_provider_map handler union, read custom_provider_map instead
of the private _custom_providers list, and annotate the untyped custom_llm_provider parameters
on the video entry points
* test(videos): cover a CustomLLM shadowing a built-in video provider name and type the new tests
Mark custom_handler ReadOnly on CustomLLMItem, type the new tests, and assert that a CustomLLM
registered under openai still leaves openai video on the built-in config
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add Parallel AI web-grounded decisions
* fix(decisions): name the Parallel model decision
* fix(parallel_ai): restrict environment credentials to trusted decision endpoints
* fix(decisions): keep Parallel evidence fields off other providers and handle missing confidence
Only fields a provider names reach the OpenAI response, so existing providers keep
their base shape. A malformed api_base answers 400 on both credential paths. The
complexity router classifier and the Playground handle a provider that omits
confidence. Regenerates the lazy OpenAPI snapshot and the dashboard API types
* chore(cost-map): declare the Decisions routes on parallel_ai/decision
The provider endpoint files are generated from the cost map, so the row must
name both routes for the generated System One flag to hold
* fix(decisions): report no classifier confidence when a provider omits it
A choice answer with no confidence is a valid classifier answer, so the
complexity router picks its tier and leaves classifier_confidence and the
confidence signal out instead of reporting a different metric in its place.
Also show the Parallel AI logo in the provider picker and add a Parallel
translation fixture captured from the live API for both Decisions routes.
* fix(decisions): keep a token-priced row on token billing when it also sets a query price
A plain input_cost_per_query bills flat only on a row with no token price, and input_cost_per_query_decisions always bills flat, so a custom Decisions row that already bills tokens costs the same as before this PR
* fix(decisions): bill a deployment's query price and accept Parallel bases that end in a route
A proxy request logs through an object built before the router picks the deployment, so a query price on litellm_params never reached Decisions billing. Parallel Search accepts a base ending in /v1/search and shares PARALLEL_AI_API_BASE with Decisions, so Decisions now accepts that base and one ending in /v1/decisions.
* fix(decisions): keep per-query billing and optional confidence on Parallel only
A query price on any other Decisions provider moved that provider off token
billing, and a choice or score answer with no confidence became a 200 with a
null for every provider. Both now apply to Parallel alone, so other providers
bill and fail as they do on main.
A Parallel deployment with explicit zero token rates and a query price read as
free, so every budget check was skipped while it billed per call. It now counts
as priced.
The dashboard key for Parallel is now PARALLEL_AI so a served bundle that
predates the provider still sends parallel_ai as the provider slug, and the
README table lists the provider.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(spend): bill web search interception follow-ups once under the outer request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): keep web search rounds inside interception follow-ups billed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): keep the outer request as parent for nested interception follow-ups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(spend): drop parent override for nested interception follow-ups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): reproduce comparison response ID ownership and wire failures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): guard stale retries and malformed comparison envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): authorize and unwrap comparison response ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): require comparison IDs to follow response security policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): deny raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): reject raw comparison response ids by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): enforce managed ownership with unmanaged ID opt-in
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send native structured output for json_schema on Claude models that cannot force a tool
Opus 5.5 and Sonnet 5.5 on Bedrock cannot force the json tool, so a json_schema request without thinking
sent an unforced json tool the model ignored. The opt-in native output key now also applies when the model
map says the model cannot force a tool, and the Opus 5.5 commercial Bedrock rows carry the opt-in key. The
GovCloud Opus 5.5 rows carry it as false, like the Sonnet 5.5 GovCloud rows
* fix(bedrock): keep json tool with thinking on Opus 5.5 and Fable 5.1, and GovCloud on the tool
Native output without thinking now applies to every Bedrock Claude model that cannot
force the json tool unless supports_bedrock_native_structured_output is false, so Opus 5.5
needs no opt-in row and keeps the json tool when thinking is requested, as #46870 chose.
The invoke chat path keeps the native output_format it built instead of inlining it, and
GovCloud region-path ids read their own false rows.
* fix(bedrock): clean caller output_format schemas before sending them natively on invoke
* fix(bedrock): keep Mantle Claude on the json tool and send a caller's output_config.format natively on invoke
* fix(bedrock): keep the faked Converse stream for native output that replaces the json tool
A json_schema stream that goes native only because the model can't force the json tool now keeps the json tool's faked stream, so it still needs only bedrock:InvokeModel. /v1/messages builds a native-output retry only where it sends native output, and the Mantle test drops its mutable capture.
* test(bedrock): cover native output and the opt-out in the invoke json-mode refusal tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(anthropic): pick beta allowlist row per deployment, add mid-2026 beta rows, surface input_transformations
Foundry-hosted anthropic deployments now filter anthropic-beta by the azure_ai row on /v1/messages and chat completions, the bundled config gains the mid-conversation-system-clear-at-2026-08-21 row plus the tool-changes and output-config betas on azure_ai and vertex_ai from a live probe, and Anthropic's input_transformations array reaches chat completions in provider_specific_fields (non-streaming and the first streaming chunk).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): type the provider_specific_fields source dict and beta row test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): input_transformations reaches chat completions and /v1/messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): read the Anthropic response as a Mapping when building provider_specific_fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): fold deployment-row tests into the filtering file and mirror the new beta in the Rust table
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): drop the duplicated beta tuple and mark filter results Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): forward the seven betas Foundry accepts on the azure_ai beta row
A Foundry-hosted anthropic deployment now filters through the azure_ai row, which
dropped compact-2026-01-12, fast mode, fine-grained tool streaming, inline tools,
oauth, output-128k, and token-efficient tools. Foundry answers 200 to each, and
without compact-2026-01-12 a compaction request answers 400, so the row now allows
all seven on both the Python map and the Rust table
* fix(anthropic): keep pass-through and beta-less requests off the Foundry beta row
An openai/ deployment that passes /v1/messages through to a Foundry host lost
every beta, and a request without betas read the beta table from GitHub.
Only an anthropic deployment on a Foundry host now takes the azure_ai row,
and the check reads the provider and host before the table.
* test(anthropic): point the beta-gated wire legs whose row lacks clear-at at a native Databricks deployment
* style(rust): wrap the azure_ai beta header test case the way rustfmt does
* fix(anthropic): forward unlisted betas to Foundry and keep Foundry calls off the Rust messages route
An anthropic deployment on a Foundry host now drops only the betas the azure_ai row marks
unsupported on /v1/messages, and forwards betas the table does not list as sent, as main did.
cognitiveservices.azure.com counts as a Foundry host. The Rust messages route sends an async call
with a Foundry api_base to Python, which filters by row. A sync call stays on Rust, since Python has
no sync /v1/messages handler. The json_mode chat response keeps input_transformations,
and the first /v1/messages call with betas loads the beta table off the event loop.
* fix(anthropic): run the advisor loop for an anthropic deployment on a Foundry host
The advisor interceptor treated every anthropic deployment as native, so a Foundry-hosted one
received the advisor_20260301 tool and Foundry answered 400. Interceptors now get the
deployment's beta row from resolve_beta_provider, so that deployment runs litellm's advisor
loop the same way an azure_ai deployment does
* test(anthropic): drive the Foundry advisor loop test through recorded HTTP
The test now intercepts the Foundry and Anthropic HTTP calls with respx instead of patching litellm's own sub-call helper, so it checks what Foundry receives: no advisor_20260301 tool and no advisor beta
* fix(anthropic): leave the advisor loop to the providers it ran for on main and tidy the PR's tests
Reverts 083990f6d7, since review found it reaches a pre-existing gap that is now with security engineering.
The beta row tests drive a real HTTP transport, the beta table fetch test checks the fetch thread instead of a 2 second deadline, and the touched test files keep main's formatting
* test(anthropic): give the Foundry beta row logging fixture a fixed datetime start time
* test(anthropic): check that chat completions pick the beta row by api_base
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: Return Anthropic error envelopes on Messages auth failures
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix: Answer Anthropic envelopes on Messages auth failures without relabelling unmapped statuses
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
* fix(proxy): answer the Anthropic error envelope on the /anthropic pass-through
The pass-through relays upstream errors in Anthropic's shape, but the proxy's
own refusals there (bad key, model access, budget) still answered the OpenAI
body, so an Anthropic SDK pointed at /anthropic could not read them
* fix(proxy): answer the Anthropic envelope for 404, 405 and unhandled 500 errors on Messages routes
A disabled Claude Code gateway 404, a wrong-method 405 and an unhandled 500 on the
Messages and /anthropic/ routes kept FastAPI's detail body or the OpenAI shape. The
central HTTP and unhandled-exception handlers now pick the Anthropic envelope by route.
The dashboard playground toast titles Anthropic error types when the body has no code.
* fix(proxy): keep LiteLLM error types and Anthropic envelopes on Messages routes
Error bodies on the Messages routes now carry error.litellm_error_type when the
LiteLLM type differs from the Anthropic type, so clients and the Admin UI keep
causes like budget_exceeded. Unlisted 4xx statuses map to invalid_request_error
per Anthropic's errors page, model-list requests that send anthropic-version get
the Anthropic envelope, and the admission, decompression and request-size
middleware errors use it on these routes too.
* test(proxy): expect litellm_error_type in Messages integration tests and type middleware tests
* fix(proxy): keep handler error bodies unchanged and timeouts as api_error
Errors the Messages and count_tokens handlers already returned in the Anthropic
shape keep their original bodies, so litellm_error_type now appears only on
errors the proxy raises before the handler. A 408 stays api_error as before,
and the playground toast titles the new Anthropic types and ignores long
numbers in front of an embedded error
* fix(proxy): keep the base error types inside the Messages handler and scope toast titles to Anthropic bodies
The wider status table (402 billing_error, 409 conflict_error, 504 timeout_error,
other unlisted 4xx invalid_request_error) now applies only to errors the proxy
raises before the handler. Errors inside the Messages handler, streamed error
frames and the SDK streaming adapters keep the merge base's table.
The dashboard toast reads a status before embedded JSON and titles by Anthropic
type only when the body is an Anthropic error envelope, so other pages keep
their titles.
* test(anthropic): mark the new test variables Final
---------
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): resend once with the placeholder signature when the other backend rejects a replayed thought signature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): gemini tool-call signature rejected by the backend is resent once with the placeholder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): keep the placeholder resend inside the cache rebuild so either rejection order ends in an accepted body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): resend a cached-prefix signature inline and cover native generateContent
When the rejected signature sits inside a cachedContents prefix, the
placeholder resend now carries the history inline, since replacing the
signature in the request body cannot reach the cache. The native
generateContent handlers get the same one-time resend.
* fix(gemini): send the history inline when Vertex rejects a signature while creating the cache
* fix(gemini): keep the context cache on a signature resend and go inline only when that is rejected too
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(managed_files): keep the upload the managed id encodes when two target models share a deployment
* fix(proxy): skip cost tracking for a file delete routed to a deployment
* fix(managed_files): delete repeat uploads after the upload answers
Run the repeat-copy deletes together in a background task so a slow
provider delete no longer holds the upload's response, log the unbilled
file-delete cost skip at debug, and type the new test helpers
* fix(proxy): run managed file repeat-upload cleanup as a drained internal call
Delete repeat uploads after the record is saved, through the shared background
task list the shutdown drain waits on, and as an internal call so logging,
spend tracking and alerts skip it. Leave litellm_proxy deployments on the old
keep-last behavior, since their per-target copies are distinct files. Revert
the cost tracker skip change, which the internal call makes unnecessary.
* fix(managed_files): delete the copy a managed id names on DELETE and isolate cleanup metadata
* fix(managed_files): read the copy DELETE removes from the id's gateway-written fields
* fix(managed_files): delete the copy a managed id carries as an internal call
* fix(managed_files): pass the parent call id to the cleanup deletes
* fix(managed_files): file each repeat-upload delete under the upload that made its copy
POST /v1/files sets no request call id, so the cleanup delete passed none
and its usage ledger row had no parent. Each copy now keeps the call id of
the router upload that made it, and its delete runs under that id.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Apply database general settings only for managed keys, warn about ignored copies, and allow admins to delete stale stored keys
Move intended database-only settings into config.yaml before upgrading, including authentication and access restrictions
* fix(responses): log the encrypted response id on streamed /v1/responses spend rows
With response id security on, the stream hook encrypts the resp_ id on every
frame, but logging had already received the SDK's own copy of the terminal
response, so the spend row, the standard logging payload and callbacks got the
managed id (or the chatcmpl id on a bridged stream) the client never saw. The
hook now records the exact id it advertises on the request's logging object,
and the logging layer stamps that id on the logged streamed response
* fix(responses): return the id the client sent on GET and cancel, and keep read rows off the create row's id
Every GET /v1/responses/{id} re-encrypted the response id, so a client that continued a bridged conversation with the
id a GET returned referenced a ciphertext no spend row was filed under. The follow-up lost turn 1 and opened a new
session. GET and cancel now answer with the exact id the client sent when it is an id this proxy issued and it decodes
to the response the provider returned. A create with previous_response_id still gets a fresh id for its own response,
and a raw provider id still gets a managed id
Because a read now carries the create's id, a GET or cancel spend row is filed as <id>_retrieve_<call id> or
<id>_cancel_<call id>. That keeps one row per read, as before, and a read row can never take the create row's
request_id, so the create row survives whichever row is written first and a follow-up replays turn 1 once. The
background settlement row keeps <id>_background_cost, and the cost poll's own status checks keep the response id
* fix(responses): log a fresh id for each read and cancel, echo the held id only to its owner
A GET or cancel of a stored response now logs a new encrypted id of its own,
so callbacks and spend rows never reuse the create's id, while the owner of
the id still gets back the exact id they sent. An admin reading someone
else's response gets an id bound to the admin, as before. The spend row id
suffix for reads and cancels is gone, so with response id security off the
rows and session history match the default branch. A streamed GET follows
the same rules.
* fix(responses): echo the sent id to every reader that passes the response access check
A GET, streamed GET or cancel of a response now answers with the id the caller sent whenever the caller passed
the access check, not only when the caller is the user or team the id was issued to. An admin, the master key
or a teammate who read someone else's streamed response got a fresh id back, and the session lookup could not
reach the first turn from it, so a bridged follow-up through that id lost the conversation history
Every read and cancel still logs a fresh id bound to the reader, never the create's id
* fix(responses): keep a read or cancel row in session history only when it brings new output
A GET, streamed GET or cancel sent with a session header joins the
create's session, and its stored response repeats the create's output.
The replay then sent the same assistant turn once per read.
Session history now keeps an aget_responses or acancel_responses row
only when its stored Responses output has an item id that no earlier
kept row carried. A polled background response keeps the first poll
that returns the finished output, since its queued create row has none,
and later polls of the same output are dropped. Every other row keeps
today's rule
* fix(responses): replay each history item once, echo the sent id on delete, keep background cost files apart
A create row now owns its output items in session history, and a read or cancel row adds only items no
create and no earlier read carried, in one linear pass, so a background response polled while it grows,
or a read that ends before its streamed create, replays the answer once and after its prompt
DELETE /v1/responses/{id} answers with the id the caller sent, on a copy, and logs a fresh encrypted id
like a read does
Azure Data Lake Storage per-event files and unbatched GCS objects are named after the event's spend row
id, so a streamed background create and its cost event, which now share the client's id, keep separate
files
The integration tests that decoded the managed id now match the exact id the client received
* fix(responses): keep main's answer for a raw-id delete and main's storage names outside background cost events
A DELETE by a raw provider id returns the provider's id as main did, and logging still records a fresh
encrypted id. A streamed GET matches the held id by upstream response id, like a plain GET
Only a background response's cost event gets the _background_cost suffix on its Azure Data Lake Storage
file and unbatched GCS object, so batch retrieves and events with no id keep main's names
An item a session read replays takes the newest read's copy at the first read's place, so an item a
poll caught half written replays finished
Tests pin the client's id over the router's continued response id, and the storage test no longer reads
the clock after the upload
* fix(spend): give background cost events their own OpenMeter and DynamoDB ids, replay id-less read items
A streamed background create and its later cost event share the client's
id, so OpenMeter dropped the second event and DynamoDB overwrote the
create's item. Both now key the cost event as <id>_background_cost, like
its spend row and storage file.
Session history again replays a read's output items that carry no id,
as main did.
* fix(integrations): keep OpenMeter and DynamoDB logging working without the proxy extras
* fix(integrations): give background cost events their own ClickHouse request id and DeepEval span id
A streamed background create and its later cost event now share the id the
client received, so ClickHouse and DeepEval saw both events under one id.
Route both through logged_event_key, the way the Azure storage, GCS,
OpenMeter and DynamoDB loggers already do, so the cost event's ClickHouse
request_id matches its spend row's `<id>_background_cost` and its DeepEval
span gets its own uuid. ClickHouse response_id stays the raw provider id.
* test(deepeval): annotate the background cost span test's locals with Final
* test(responses): run the DELETE wire test that pins the echoed client id
DELETE /v1/responses/{id} now answers with the id the caller sent, so the
skip that marked it as a known bug no longer applies. The test passes on
the /v1, /openai/v1 and bare routes.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): honor model_info.max_input_tokens in the Codex catalog for stock models
A Codex model catalog entry copied from Codex's stock entry (a stock slug, or an alias whose upstream is one) now takes the max_input_tokens every routable deployment of the group sets as its context_window, capped at the stock entry's max_context_window like Codex's own model_context_window. With nothing set, the stock window stays.
* feat(proxy): honor the smallest max_input_tokens a group's deployments set in the Codex catalog
A deployment without a limit counts as Codex's own window, so a limit on only some deployments can lower the window but never raise it
* fix(proxy): skip a zero or negative max_input_tokens when sizing a Codex catalog window
* test(proxy): stub the max_input_tokens group field and type the Codex window tests
* fix(proxy): skip a max_input_tokens a saved /model/info response copied from the cost map
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(count_tokens): count grok-4-1, grok-4.3 and DeepSeek V4.1 tool requests like their chat templates
Render grok-4-1, grok-4.3 and DeepSeek V4.1 tool schemas the way each chat
template does, and add a per-deployment token_count_tool_framing next to
token_count_framing for the hidden tool preamble a host bills. The Azure AI
Foundry grok-4-1, xAI grok-4.3 and Fireworks DeepSeek V4.1 cost map rows carry
the measured framing, overridable in model_info
* fix(count_tokens): count Azure grok-4-1 Responses requests with their own framing and use the tool framing in place of the base framing
* fix(count_tokens): keep operator framing on tool and Responses counts, frame bridged Responses like chat, follow base_model for tool rendering, count grok tool_choice
* fix(token_counter): count Azure Responses strict default, skip false strict and built-in-only tools, per-kind operator framing
* fix(token_counter): count an explicit null strict like false on Azure Responses
* fix(count_tokens): follow the chat completions bridge flag and Azure's strict default on azure/ deployments
Keep Azure grok-4-1 chat tool counts at or over the bill by raising its tool framing to 302
* fix(token_counter): keep strict false for non-grok templates, coerce the bridge flag, frame DeepSeek API counts
* fix(proxy): frame Responses counts for the api_base router defaults send inference to
* fix(proxy): keep the api_base helper's name so main's /v1/messages count caller resolves
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): resolve Invoke Claude default max_tokens from the base model id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): build Invoke Claude request params without mutating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): type tool-search normalizer return to clear unknown-type gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): resolve Invoke and Mantle Claude default max_tokens from the deployment id, ARN base_model, and Mantle row
* fix(bedrock): default Invoke Claude max_tokens from the deployment's own cap and honor AnthropicConfig
Read the called deployment's model_info.max_output_tokens before the base model's
cost row, skip the shared regional key the router fills from every deployment, and
leave the default to litellm.AnthropicConfig(max_tokens) when it is set.
* fix(bedrock): send the mapped default max_tokens on bedrock_mantle Claude chat
* fix(bedrock): bound a global AnthropicConfig max_tokens by the deployment's own cap
* test(bedrock): annotate the new Mantle default max_tokens tests with Final
* refactor(bedrock): inject the AnthropicConfig max_tokens reader into the Invoke Claude config
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): join plain text blocks for non-Claude json_object requests
* fix(databricks): join only the last message for json_object and drop cache_control under drop_params
Databricks checks for "json" only in the last message, so earlier turns now go out as sent and only the
last message, plus any system messages merged into it, is joined into a string. Under drop_params a
text block's cache_control is dropped so that message can be joined too, and without it the request
still goes out as a list.
* fix(databricks): skip dropped assistant turns and blank parts when joining for json_object
The json_object join picked the last message before the request transform drops bare assistant turns,
so a trailing thinking-only or reasoning-only turn (a Responses input ending in a reasoning item included)
left the real last user message as a list and Databricks still returned its 400. The pick now skips the
turns that transform drops.
The join also ran when only one part had text, which sent whitespace-only parts the existing empty block
filter used to drop on requests that already worked. It now joins only two or more non-blank parts, so a
request with one non-blank part goes out exactly as before. Without drop_params, a message with a
message-level cache_control now goes out as sent instead of being rewritten into one cached block.
* fix(databricks): join json_object text parts only on models Databricks hosts
Databricks checks for the word "json" only on the models it hosts
(databricks-* pay per token names and system.ai.* AI Gateway names). An
external model endpoint skips that check and takes a list of text parts
today, so its request now goes out as sent instead of joined.
The drop_params read now goes through drops_unsupported_params, and a
test pins that a text block with a truthy non-string text is kept.
* fix(databricks): reject a text part holding a non-string locally with a clear 400
* refactor(databricks): reject a non-string text part at the transform_request boundary
* fix(databricks): read pydantic message objects in the non-string text check
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openai): trim a malformed prompt_cache_breakpoint under drop_params on chat completions
* fix(openai): leave a non-object content part for OpenAI to reject under drop_params
* fix(openai): trim an unsupported prompt_cache_breakpoint ttl on every path and on azure_ai chat under drop_params
* fix: trim prompt_cache_breakpoint to what Azure and Mantle accept under drop_params
Azure OpenAI chat now gets the same content part trim as OpenAI chat, and
the Azure and Foundry gates also read a deployment's declared base_model.
Each Responses config keeps only the ttl its API takes: Azure keeps 30m,
and Mantle keeps no ttl. A trimmed ttl is logged at debug.
* fix(responses): trim prompt_cache_breakpoint only on APIs known to read it
Other Responses providers get the marker whole or not at all, as on main.
Azure chat now trims for a deployment with no base_model unless the cost
map marks the model as not reading breakpoints, and Azure AI Foundry
Responses keeps Azure's 30m-only shape.
* fix(azure): trim prompt cache breakpoints only for deployments known to read them
Azure and Azure AI Responses keep main's whole-or-nothing rule for unmapped deployment names, an opted-in custom OpenAI host trims the ttl and keeps the mode, and Azure chat strips every marker for a known non-reader under drop_params
* fix(providers): keep main's marker handling where the target won't trim it
A /v1/responses model with unknown breakpoint support on OpenAI's host gets a marker whole or not at all, as on main. The chat to Responses bridge keeps main's whole-marker check when a non-bool global drop_params leaves the target's trim off. A hyphenated name like gpt-5-6-sol now reads as gpt-5.6.
* fix(providers): read a non-version hyphen suffix like -32k as part of the model name
* fix(providers): keep main's reading of a dotted minor version like gpt-5.6o
* fix(providers): keep main's breakpoint shape for unmapped Mantle models and skip the Azure lookup without drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Scope the Prisma migration lock test's advisory lock check to its own database, so a lock another xdist worker holds in its database no longer trips it
Keep in-process MCP peers' idle connections open for 30s instead of 1s, so the gateway's pooled HTTP client doesn't reuse a connection the peer is closing and hang the tool call
Delete database guardrails in the logging-only scope tests through DELETE /guardrails/{id} on the shared gateway instead of raw SQL, so a default-on guardrail stops applying to the next test on the same worker right away
Bound the wedged coordination Redis /customer/update check by the shortest Redis wait the proxy uses, like the burst check, so a slow runner can't fail it while a handler that waits on Redis still does
* fix(anthropic): sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept
* test(e2e): send the azure_ai beta check without the code execution tool
* test(anthropic): require a vertex_ai and azure_ai decision for every anthropic beta
* fix(anthropic): forward the betas Claude Code sends that anthropic, vertex_ai, and azure_ai accept
* test(e2e): send Vertex's header-only betas in the beta acceptance case
* fix(anthropic): keep forwarding Vertex computer-use betas and point beta tests at platforms that still lack them
A live probe found each Claude 4.5 and 4.6 model on Vertex needs one of the two
computer-use betas for the computer tool, so dropping both broke it. The e2e
acceptance test leaves those two out of its all-at-once Vertex call, and the
integration and unit tests that relied on a beta being dropped now use one
the platform still lacks.
* test(anthropic): point unsupported-beta tests at betas the platforms still drop
The Azure AI handler test used claude-code-20250219 as the beta Azure drops, and the
Rust parse test used it as an unknown value. This PR forwards it on Azure AI and maps it,
so both now use values that stay dropped or unknown.
* fix(anthropic): add computer-use-2025-11-24 for the computer_20251124 tool
The computer_20251124 tool fell through to the default computer-use-2024-10-22 beta, which the
provider rejects for that tool version. Map it to computer-use-2025-11-24, which the beta map now
forwards on anthropic and azure_ai
* fix(anthropic): send no computer use beta for an unknown computer tool version
An unknown computer tool version fell back to computer-use-2024-10-22, which the anthropic and azure_ai columns now forward. It now gets no beta, as every platform effectively sent before. Every computer tool version also skips defer_loading and allowed_callers, computer_20251124 included
* test(azure_ai): read the branch's beta map in the auto-injected beta wire test
* fix(anthropic): keep defer_loading and allowed_callers on the computer_20251124 chat tool
* test(anthropic): read the branch's beta map in the azure_ai safeguards beta wire test
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): keep encrypted reasoning on a hop between same-key deployments on the provider's default endpoint
* test(router): type the default-endpoint boundary test helpers
* fix(guardrails): restore the google_text_moderation embeddings fix an earlier main merge dropped
* fix(router): match encryption boundaries on the key, account, and region the request really uses
A deployment's named credential now fills only the values the deployment leaves out, through the
same helper the request path uses, so the boundary names the key and base the request sends.
A deployment with no api_key takes the provider's default key, the boundary carries the OpenAI
organization and project a deployment sends, and a Bedrock Mantle region named in the model
routes through the provider resolver before the Responses URL builder
* fix(router): keep header-signed accounts apart on a shared boundary and skip boundary work no reasoning needs
Headers the request logger masks as secrets (Authorization, api-key, x-api-key) now join the
boundary's account scope, with extra_headers overriding headers the way the request merges them,
so deployments that sign in through a header on one base never share encrypted reasoning.
Target boundaries are built only when a reasoning item from another deployment needs the
comparison, and the Responses input strip asks only about encrypted reasoning items, so a pinned
turn or fallback hop with nothing to compare makes no secret-manager reads.
* fix(router): resolve default boundaries only for openai and drop the unsent organization param
Aggregators and Bedrock Mantle serve several vendors on one key, so a missing key or base there no longer yields a shared boundary. The Responses request never sends the organization param, so it no longer splits the boundary, while a Cookie header now does.
* fix(router): match explicit pairs as on main and key default boundaries by model
An explicit api_base plus api_key pair is the whole boundary again, so tag
headers no longer split deployments that main kept together. The default
OpenAI boundary now carries the model, since one OPENAI_BASE_URL can front
several vendors, and a deployment that declares another provider gets no
boundary before any secret lookup.
* fix(router): match an explicit key and the environment key on one api_base
A deployment with an explicit api_base and api_key now shares a boundary with one on the same api_base that takes the same key from OPENAI_API_KEY, when the model and account headers also match. Two explicit pairs still match on base and key alone, as on main.
* fix(router): skip a deployment whose params cannot be read in boundary checks
A named credential with a value of the wrong type, or an extra header whose name is not a string, made the boundary check raise and fail the whole request. That deployment now has no boundary, so the others still keep the reasoning they can decrypt. Two explicit api_base plus api_key pairs also compare without resolving the provider, so they read no secrets, as on main.
* fix(router): ask a secret manager for default boundaries at most once a minute per deployment
* fix(router): resolve a changed named credential at once in the cached boundary and take an injected clock
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Deletes the A2A e2e test that registered a third-party agent, which depended on a site the project does not control, along with its helpers and its coverage registry row. Adds an integration test where a scripted A2A agent receives a skill call as a data part and returns a data artifact, asserting the agent and the caller each get theirs unchanged
