* fix(responses): keep the proxy's error body on litellm.responses errors from litellm_proxy models
litellm.responses() with a litellm_proxy/ model raised BadRequestError with
body None when the proxy answered a 4xx, while litellm.completion() on the
same proxy carries the decoded error object. The litellm_proxy Responses
config now decodes the error JSON into the exception body the way the
OpenAI SDK does, so body, code, type and the message match chat completions
on sync, async and streaming calls.
* fix(proxy): relay a downstream proxy's provider_specific_fields in error responses
A LiteLLM proxy routing to another LiteLLM proxy now copies the downstream's
provider_specific_fields (which guardrail fired) from the exception body into
its own error answer, on chat and Responses alike. Adds tests for the two
_openai_sdk_error_body branches the existing tests left open.
* fix(proxy): redact internal details from relayed provider_specific_fields
* fix(proxy): redact relayed provider_specific_fields per key so credential-named values stay hidden
* fix(proxy): redact numeric credentials in relayed provider_specific_fields
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(ui): pin the Workflows page copy with characterization tests
Locks the header, table, filters drawer, detail drawer, relative time and duration text on the Workflows page so the upcoming move onto i18next can be checked against it without editing any test
* feat(ui): add i18next plumbing and move the Workflows page copy onto it
Adds an English-only i18next instance with typed keys from src/i18n/locales/en.json, wraps the root layout in I18nProvider, and loads the instance in the test setup. Every string the Workflows page authors now renders through t(), while backend values such as run status, roles, event types and metadata keys stay raw. The rendered English is unchanged
* fix(anthropic): retry /v1/messages without strict when Claude rejects strict tools under drop_params
The native /v1/messages path sent tools with `strict` straight through, so
when Claude's strict-tool limits rejected them with a 400 the caller got that
400 even with drop_params set. Under drop_params, a 400 that matches Claude's
strict-limit phrases on a request with a strict tool is now sent once more
with `strict` removed from every tool. Without drop_params nothing changes.
The Claude strict-limit matcher moves from bedrock/common_utils.py to
anthropic/common_utils.py so Bedrock converse and the base messages config
share it, and the Rust bridge routes strict tools under drop_params to Python.
* fix(anthropic): retry without strict only when a tool sends strict true
A tool with only strict false no longer triggers a resend of an identical body, the warning no longer
blames tool schemas when the rejected schema may be the output format, and requests with only
additional_drop_params stay on the Rust messages route since the Python retry ignores them
* fix(anthropic): keep sync messages calls with strict tools on Rust
Python has no sync /v1/messages handler, so routing a sync call with a
strict tool to Python raised ValueError where Rust served it before.
The strict-tools detector now applies to async calls only. The retry
warning also names the strict tool count without a plural.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill streamed /v1/completions without stream_options like chat
* test(proxy): mark the text completion stream usage test variable Final
* fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage
* test(spend): drop the module docstring from the text completion stream usage test
* fix(proxy): keep the internal stream usage strip flag out of the logged request body
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openapi): build native extension for base counts
* test(openapi): exercise native import in base worktree
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
GenericGuardrailAPI.get_supported_event_hooks only listed pre_call,
post_call and during_call, so event hook validation rejected the MCP
modes and the guardrail was skipped at proxy startup instead of being
attached to MCP tool calls. Add pre_mcp_call, during_mcp_call and
post_mcp_call. during_mcp_call runs alongside the tool call, so it can
fail the call but cannot rewrite its arguments.
#48718 added TestLangfuseDestinationFields while #48723 made the unconfigured
stdout placeholder opt-in (console_default=True, the otel callback's setting).
Both merged within a minute, so main's integrations job fails six tests that
read the placeholder's exporter. Build those providers with console_default=True
so the operator-exporter checks keep their target.
* fix(caching): key cached async clients by a per-loop token instead of id(loop)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): drop deferred closes queued on a collected event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): pin that a close queued on a live loop waits for that loop
* test(caching): run loop-reuse tests until a loop address repeats
The collected-loop tests ran a fixed 50 loops and passed with the fix reverted whenever no loop reused an earlier address. They now stop at the first reuse and skip if none happens, through a shared helper.
* test(caching): cover the cache key fallback for loops without weak references
Also corrects the evicted client closer docstring: a dropped bucket's clients are left to the garbage collector.
* fix(http): free short-lived event loops whose handler finalizer close finished as the loop stopped
* fix(caching): share sync clients across event loops and catch a failed finalizer close
* fix(agentops): stop closing the shared sync http client after the auth call
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Without drop_params, every copy of a tool from additional_tools input items
is forwarded as sent, so a provider that rejects duplicate tools returns its
own 400. Codex keeps the merge because the proxy turns drop_params on for it.
* test: add a manually dispatched integration trial on larger runners
Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers
run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one
* test: run the integration trial on push to this branch
* test: give integration trial workers a local redis-server
* test: run the providers trial on a 64-core runner with 32 workers
The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4
* test: run every integration group in the trial on sized runners
* test: add a scan that refuses to upload integration results holding the license
* test: drop the member-name pass the archive byte scan already covers
* test: use a secret whose base64 and base64url encodings differ
* ci: run integration groups on run-ci pull requests and scheduled main runs
* ci: scope the integration license to a protected environment
* fix(otel): stop OTel v2 vendor presets from printing every span to stdout
With LITELLM_OTEL_V2 on and a vendor callback such as langfuse_otel or arize
holding operator credentials, the proxy printed every span as JSON to stdout,
prompts included, on the request path. Nobody set OTEL_EXPORTER, but the
console placeholder that OpenTelemetryV2Config folds in for an empty exporter
list stayed next to the vendor's own exporter.
The provider now skips that placeholder when the config has any other
exporter. A console exporter the operator names with OTEL_EXPORTER=console is
kept, and so is the placeholder when it is the only exporter.
* test(otel): type the console-placeholder tests and drop inherited OTEL_* env from the stdout check
* fix(otel): print the console default only for the otel callback
The proxy's background V2 logger (published when no callback built one, which is
what key-only and team-only Langfuse destinations run under) also kept the
unconfigured console exporter, so it printed every span with prompts to stdout.
The console default now prints only for a listed otel callback or an exporter the
operator named.
* fix(otel): keep the otel callback's console output on service-name routes
A key or team otel_service_name moves the call onto a provider the tenant
tracer cache builds, which never got the console choice, so the plain otel
callback stopped printing those calls. The no-operator-keys integration rig
also drops inherited Langfuse keys.
* feat(guardrails): scan sampled stream windows in the background (LIT-9338)
streaming_scan_in_background runs each sampled post_call scan as a task so
the stream keeps flowing; a block ends the stream before the next chunk.
* fix(guardrails): hold the stream end for the background verdict and cancel the scan on upstream errors (LIT-9338)
* fix(guardrails): wait for the cancelled background scan on upstream errors (LIT-9338)
* test(guardrails): cover /v1/responses, a failing scan peer and a mixed burst for background scans (LIT-9338)
* test(guardrails): build the burst calls without a nested comprehension (LIT-9338)
* fix(otel)!: keep a credential-less langfuse_otel on OTel v2 so key destinations get the Langfuse mapper
With LITELLM_OTEL_V2 on, langfuse_otel in callbacks and no operator
LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY, the preset raised at startup and
the callback fell back to the legacy logger. Key and team destinations were
then served by a generic v2 logger without the Langfuse mapper, so their
traces lost tags, user and cost, and the legacy logger added an orphaned
raw_gen_ai_request span.
The preset now always degrades to the credential-gated exporter when the
operator has no keys, the same way arize_preset does, so the startup logger
stays on v2, exports nowhere for traffic without a destination, and carries
the Langfuse mapper to every key and team destination.
* test(otel): type the langfuse tenant-only test helpers
* fix(otel): give key/team Langfuse destinations the Langfuse fields when langfuse_otel is not a callback
With langfuse_otel only in a key's or team's logging, the startup logger is the
generic one and its spans reached the key's Langfuse project without user,
session or tags. The emitter now adds the Langfuse mapper for a request with a
Langfuse destination, and every non-Langfuse exporter strips the Langfuse
request metadata, since any logger can now write it.
* fix(otel): stop a key's Langfuse logger from re-sending spans to the operator's collector
With langfuse_otel only in key or team logging, the proxy publishes the generic
V2 logger at startup and it already exports to the operator's OTLP collector.
The Langfuse logger a key's request builds kept the same base exporter, so the
collector got every model span twice. A logger built after the publish now keeps
only its preset's own exporters, as it already did beside a configured V2 logger,
and a later call that would rebuild one exporting nowhere reuses it.
* test(otel): assert the collector gets one copy instead of reading the key logger's exporters
* test(otel): count the collector's spans with one comprehension clause
* fix(otel): keep the legacy langfuse_otel logger and stand it down for requests a Langfuse destination carries
Reverts the preset degrade and the published-provider exporter trim, so every
request without a key or team Langfuse destination keeps main's legacy logger.
For a request with one, the V2 fan-out already delivers the whole trace with
the Langfuse fields, so the legacy logger emits nothing for it.
* test(otel): pin the legacy langfuse_otel paths a tenant's project still gets without a destination
* fix(otel): shed a destination's Langfuse request metadata before its other Langfuse fields
At its attribute limit a span sheds the Langfuse fields only a destination asked
for first, in the mapper's order, so the generation marker went before the
request metadata blob. Put the request metadata first, as a Langfuse-configured
logger already does.
* fix(otel): keep the legacy langfuse_otel span for a destination's call rejected before any provider
The OTel v2 fan-out records no call for a request the proxy rejects before a
provider (rate limit, pre-call hook), so standing the legacy logger down there
lost the user, session and tags main sent the tenant. Stand it down only for a
call the fan-out records, using the same check the v2 logger uses.
The tenant and collector integration tests now hold their exact counts for a
few seconds, so a late duplicate copy fails them.
* fix(otel): give a key's Langfuse destination the langfuse_environment the legacy logger wrote
The destination's resource carries langfuse.environment from the key's or team's langfuse_environment, else LANGFUSE_TRACING_ENVIRONMENT, so a key's project files its traces under the environment it chose instead of default. Only that destination's export carries it.
* fix(otel): keep the Langfuse fields written for a key's Langfuse destination off every other exporter
* fix(otel): give a key's Langfuse destination the prompt and response the legacy logger sent, only where it ran
* fix(otel): keep a key's Langfuse destination on the global capture setting when it sets none, with or without global Langfuse keys
* fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask
on_flagged block now returns 400 when Lakera flags only PII, instead of
masking it and forwarding the request. on_flagged mask keeps the old
behavior: mask PII-only detections and block anything else.
* test(guardrails): cover lakera_v2 on_flagged on every endpoint, hook and a Lakera outage
* test(guardrails): type the lakera_v2 on_flagged unit test helpers
* docs: link README to Python SDK and autorouter guides
* docs: give auto routing its own README feature section
* docs: enlarge README feature headings and clarify A2A
* docs: add compact auto routing examples and preserve feature label size
* docs: put autorouter docs first and trim examples
* fix(proxy): honor litellm_settings.drop_params for Claude Code and Codex requests
The agentic CLI drop_params default checked the operator's config with
isinstance(config, dict), but ProxyConfig.config is a MappingProxyType, so the
check never matched and litellm_settings.drop_params could not turn the
default off. Read it as a Mapping so an operator who sets drop_params, true or
false, keeps the decision on /v1/messages, chat, and count_tokens
* fix(proxy): keep the Claude Code and Codex drop_params default unless the admin's value reads as false
A drop_params: true saved through /config/update never reaches the global setting, so treating the key's presence as an opt-out made Claude Code and Codex requests answer 400 there. Only a value that reads as false now turns the default off; true, empty, or missing keeps it.
* test(proxy): cover string drop_params values in the Claude Code and Codex default rows
Adds rows for "false", " False ", "true" and an unreadable "maybe" under litellm_settings, so swapping the value check for a plain is-not-False comparison fails the suite
* docs(proxy): say only a false operator drop_params turns off the agentic CLI default
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): integration cells for PANW include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): reject non-boolean include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(guardrails): update PANW tool-call comment for include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): build PANW kwargs without mutation and clean up DB-created test guardrails
Restore main's single-expression kwargs in the PANW initializer. The two DB-backed integration tests now create their guardrail inside the scenario and delete it on exit.
* fix(guardrails): type raw PANW boolean setting before validation
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep Mistral thinking text on streaming /v1/messages
* test(anthropic): type the signed thinking stream test helpers
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(vertex_ai): use HTTP/2 httpx client for search_api vector store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(httpx): exercise configured request path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(httpx): verify HTTP/2 request negotiation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(httpx): preserve HTTP/2 for sync proxy mounts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): add h2 to proxy extra so vertex search HTTP/2 is on by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): isolate ssl context for http2 clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): add an HTTP/2 opt-out and harden the Vertex AI Search HTTP/2 client
litellm.http2 is now unset by default, so http2: false or LITELLM_HTTP2=False
turns the provider's HTTP/2 default off. A caller's own SSLContext keeps the
search on HTTP/1.1, the HTTP/2 async client honors DISABLE_AIOHTTP_TRUST_ENV,
forced IPv4 transports get the CA bundle and client cert, and vector store
search retries once when the server drops the connection with a GOAWAY.
* fix(http): retry vector store search only on HTTP/2 GOAWAY and keep aiohttp pool and proxy behavior
Scope the one retry to a connection Google closes with an HTTP/2 GOAWAY and
log it. When HTTP/2 replaces the aiohttp transport, keep aiohttp's idle
keepalive and connection cap, and only then skip env proxies under
DISABLE_AIOHTTP_TRUST_ENV, so the global LITELLM_HTTP2 toggle keeps HTTPS_PROXY
* test(http): cover the vector store search GOAWAY retry in unit tests and drop duplicate SSL cache comments
* fix(http): open another HTTP/2 connection once 100 streams are in use
httpcore keeps every request to an origin on one HTTP/2 connection and caps it at 100 concurrent streams, so request 101 and later waited with no timeout. The handlers now spread requests over extra connections, opening one only when every open connection has 100 streams in flight. The force_ipv4 wire tests now cover ssl_verify=False and the client certificate.
* fix(vertex_ai): reuse one Vertex search OAuth token across searches
Every vector store search built a new search config, so the token cache it carries was thrown away and each search fetched a new Google OAuth token with a blocking call on the event loop. Under a burst of searches that blocking fetch held back every HTTP/2 response. The provider config is now shared, so searches with the same credentials reuse one token until it expires.
* fix(http): keep the HTTP/2 connection spread within the connection cap
* fix(http): keep sync vertex search on HTTP/1.1, since httpcore's sync HTTP/2 is not thread-safe
httpcore 1.0.9 hands two threads the same HTTP/2 stream id, so threaded sync
searches failed with protocol errors. Only the async client, which the proxy
uses, now defaults to HTTP/2, and the sync connection spread is removed.
* fix(http): split the connection cap across HTTP/2 lanes
Each lane had its own pool with the full connection cap, so an origin answering
HTTP/1.1, or several origins on one client, could exceed max_connections once
more than 100 requests were in flight. Lanes now open only while lanes times
100 streams stay under the cap, and each lane pool gets at most 100 of it.
* test(http2): scope wire-test peer logs to each fixture instance
* fix(http2): keep TE, force_ipv4 trust, evicted-client and opt-out behavior on the HTTP/2 path
- drop TE values other than trailers on https requests through the HTTP/2 spread, since h2 refuses them
- force_ipv4 transports keep httpx's default trust store (SSL_CERT_DIR) unless SSL is configured
- the evicted client closer asks the HTTP/2 spread whether a request is in flight
- LITELLM_HTTP2 set to 0, no or off, and a quoted "false" http2 setting, opt out of HTTP/2
- the wire test's held-search barrier waits on an event instead of polling
* fix(http2): count proxy-mounted transports when checking an evicted client for in-flight requests
* test(http2): move the HTTP/2 eviction cases that open a loopback server into the integration wire tests
* fix(http2): read an unquoted http2: 0 as off and free the stream slot of a response dropped without closing
* fix(http2): free the stream slot of a partly read response dropped without closing
* test(http2): check that the evicted client and its stream close after a partly read response is dropped
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Code outside the router now reaches it only through the entry points
(registry, settings, health, select, apply, observe, register_plugin, retire)
or the endpoint methods; the outside-member allowlist shrinks from 139 to 45.
litellm/router.py no longer imports litellm.proxy: it asks the installed
RouterHost (litellm/router_host.py) for vertex pass-through credentials,
member auto-router authorization and auto-router prompt compression, and the
proxy installs its host on import. Call sites where a model name can be
missing keep the old answer instead of reaching an entry point with None.
* fix(proxy): default the uvicorn worker healthcheck timeout to 60 s so multi-worker boots survive a slow import
* fix(proxy): warn on uvicorn below 0.37 when multi-worker boots cannot get the longer healthcheck
uvicorn 0.33 to 0.36 has no timeout_worker_healthcheck, so the 60 s default never reaches it and a slow worker start still crash-loops with no hint. Print an upgrade warning when the proxy runs more than one uvicorn worker on such a version, keep granian out of the uvicorn-only path, and type the new tests.
* fix(proxy): name --run_granian in the healthcheck help text and add per-worker grace guidance to the helm chart
* fix(proxy): keep the litellm command light so spawned workers answer uvicorn's first health ping
Each worker uvicorn spawns re-runs the parent's main script before it starts
answering health pings. The litellm console script imported the whole SDK at
the top, so on a busy or CPU-limited host a new worker missed uvicorn's 5 s
ping and was killed in a loop. The script now runs litellm_cli.main:proxy_server,
which imports the proxy only when called, and gateway.launch imports LiteLLM
inside main the same way.
This replaces the earlier 60 s healthcheck default, so uvicorn's 5 s
healthcheck and SIGTERM timing stay as they are.
* fix(proxy): load the worker app before its log config and without GC pauses
* fix(gateway): ship litellm_cli in the gateway image
gateway.launch imports without_gc_pauses from litellm_cli, which the
gateway image never copied, so python -m gateway.launch failed at import.
Also build the gateway image when the launcher or litellm_cli changes.
* fix(proxy): apply the worker log config before the proxy app import, as earlier releases did
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve reasoning fields on streaming cache hits
* fix(proxy): validate cached streaming delta fields
Signed-off-by: jesco-absolut <team@srswti.com>
* fix(caching): replay cached audio, images, reasoning and citations in stream order
A cached answer replayed as a stream now ends with its own empty finish chunk, so a chunk that
carries only audio or images keeps its payload and role. Reasoning fields lead on a chunk with no
text, and annotations arrive once on the last text slice, matching a live stream's order.
* fix(caching): replay a cached tool call after the text, like a live stream
Only the role, reasoning fields and provider fields lead a replayed answer. Tool calls now arrive on
their own chunk after the last text slice, and logprobs ride the first text chunk, so a cached
answer with reasoning and a tool call keeps a live stream's reasoning, text, tool call order.
* fix(caching): replay the reasoning before a cached tool call that has no text
A cached answer that thinks and then only calls a tool now replays its reasoning on its own chunk before the
tool call, the order a live stream sends, instead of one chunk carrying both.
* fix(caching): keep blank-only text when replaying a cached answer with reasoning
A cached answer that thinks, sends only whitespace as text, and then calls a tool now replays that whitespace
between the reasoning and the tool call, instead of dropping it.
---------
Signed-off-by: jesco-absolut <team@srswti.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Router gains eight entry points (registry, settings, health, select, apply/aapply,
observe, register_plugin, retire) in litellm/router_api.py. The 123 public members
outside code used become DeprecatedRouterMembers (litellm/router_deprecated.py):
same signatures and results, a DeprecationWarning naming the replacement and a use
counter for callers outside the router. Router internals move to _core_ names;
subclass overrides of old members are still called by the router.
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations
* feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger
* test(proxy): cover the dry run CLI and PrismaManager wrapper without a database
* fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly
* ci(codecov): stop carrying forward the roi-postgres and redis-compat flags
Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: drain shutdown callbacks and persist pending tag spend
* fix: catch shutdown timeouts on Python 3.10
* fix: share shutdown callback budget with HTTP draining
* fix: type shutdown cancellation tasks and document its clock
* test: retain callbacks after bounded worker cancellation
* test(proxy): inspect the lifespan body after the SIGTERM wrapper split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): keep the success callback enqueue on its own task
The synchronous enqueue spawned the logging worker loop inside the awaited
call and ran callbacks that per-call loops used to drop, which hung the SDK
callback tests on the autouse worker flush. The shutdown worker flush covers
the queued callback on its own, so the one-tick task is restored
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback
The aggressive clear ran the rejected callback and marked it done against
the queue it was never put in, so flush() could return while a real
callback was still running and the shutdown drain cancelled it early. The
rejected callback is now re-enqueued once the clear made room, and only
runs uncounted when the queue is still full
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging_worker): build the aggressive-clear run set immutably
Final tuple instead of an appended list, with the re-enqueue in a
_requeue helper, and Final annotations on the overflow regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): track overflow callbacks through shutdown
* chore(logging): explain shared callback ownership
* test(logging): isolate deferred callback completion
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dashscope): derive protocol bases from one domain
Accept the DashScope domain root, the OpenAI-compatible base, the Anthropic
Messages base, or a native /api/v1 base and derive every surface from it
Adds a native Anthropic Messages config for /apps/anthropic so a single
deployment serves chat, messages, embeddings, rerank, image generation, and
video against the same credential
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* fix(dashscope): keep Anthropic credentials off DashScope messages
Require a DashScope key or a caller auth header before the Messages path
runs, and opt out of workload identity, so the base class cannot fall back
to ANTHROPIC_API_KEY or ANTHROPIC_AUTH_TOKEN
Also format the image generation base selection
* style(dashscope): keep the messages config within the format gate
* test(dashscope): cover caller Authorization on messages
* fix(dashscope): keep the chat bridge for protocol-named bases and the image default for unconfigured calls
* fix(dashscope): keep configured image and rerank bases, normalize blank keys, derive brand chat bases
- image calls fall back to the provider's own chat base env (DASHSCOPE_API_BASE, QWENCLOUD_API_BASE, QWEN_AI_PLATFORM_API_BASE) before the image default
- an explicit /compatible-api/v1/reranks URL wins over DASHSCOPE_API_BASE_RERANK again
- a blank DashScope key counts as unset on /v1/messages, so it never falls back to ANTHROPIC_API_KEY
- QwenCloud and Qianwen AI Platform chat and embeddings derive their compatible-mode base from a domain root
- tests annotate new variables with Final and drop the config-class assertion
* fix(dashscope): keep an explicit compatible-api rerank base as given
* fix(dashscope): derive chat from compatible-api bases, keep the caller's rerank base, and let explicit bases beat per-endpoint env vars
* fix(dashscope): keep a full DashScope image endpoint URL as given and type the brand tests
* fix(dashscope): let the image env var beat any shared DashScope base
* fix(dashscope): keep a full DASHSCOPE_API_BASE image URL ahead of the image env var
* fix(dashscope): keep a native DashScope endpoint URL as given for chat and the api base header
---------
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines
Gives OpenAITextCompletionHandler a process_output_streaming_response that
assembles the buffered TextCompletionResponse chunks per choice index at end
of stream, runs the guardrail over the assembled text, and writes text
rewrites back into the first text-carrying chunk of each choice. Tests and
live proof still pending.
* fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails
Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs.
* test(guardrails): cover streamed /v1/completions under post_call guardrails
* fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through
* fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter
* fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent
The content filter's streaming hook held back the end of a choice until its
finish_reason, which litellm sends only for the first choice of an n>1
stream, so choice 1 lost its last characters on text and chat streams. A
closing chunk now releases every unfinished choice when the stream ends.
A blocked text stream's closing chunk reported 0/0 usage when the upstream
sent none; it now carries usage only when the upstream reported it.
* fix(guardrails): send no usage on the content filter's end-of-stream chunk
* fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions
* fix(guardrails): stream Responses API events through the content filter as they arrive
* fix(guardrails): treat a null text completion choice index as choice zero when assembling streams
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(secret_managers): cover oidc/google api_key references resolved per request
Unit tests send a metadata server ID token for an admin oidc/google/<audience> api_key, reuse it within its
lifetime, refresh it before it expires, fetch it off the event loop, fail closed with a 401 that names only the
kind, and honor GCE_METADATA_HOST. The integration file scripts a metadata server and checks the bearer the
upstream receives for config, stored credential and DB credential sources, plus a metadata failure
* feat(secret_managers): resolve oidc/google api_key references per request
An admin-set oidc/google/<audience> api_key now resolves on each call to the GCE metadata server's ID token for that audience, cached until a minute before it expires. Async callers fetch on a cache miss in a worker thread, so the event loop never waits on the metadata server. A failed fetch answers 401 naming only the kind, and GCE_METADATA_HOST points the fetch at another metadata server.
* fix(oidc): bound the Google metadata fetch and answer 503 when it fails
A Google OIDC api_key fetch now waits at most GCE_METADATA_TIMEOUT seconds
(default 3), runs on its own thread pool instead of the loop's default
executor, and sends the token read from the cache at call start even if it
expires mid-call. A metadata timeout, network error, or 5xx answers a
retryable 503, while a 4xx or empty token stays a 401. The expiry and
empty-token warnings log the secret name on one line.
* test(oidc): expect 503 when the metadata server answers 5xx in the integration test
* fix(oidc): answer 503 on token-count routes during a metadata outage and fall back on a bad GCE_METADATA_TIMEOUT
* fix(secret_managers): keep get_secret's metadata wait, share concurrent google token fetches, refuse oidc/google on pass-through
A model's api_key fetch keeps the 3 second GCE_METADATA_TIMEOUT wait while get_secret callers keep the handler default. Any metadata transport error answers a retryable 503. Async cache misses for one reference share one fetch, and a call the reference may not go to is refused before any fetch. Pass-through routes answer 401 for an oidc/google credential instead of sending the reference text. The metadata client is closed after each fetch, and the unused sync resolver is removed.
* fix(utils): refuse an oidc/google api_key on pass-through calls before any fetch
A router-model or credential-name pass-through call with an oidc/google
reference now fails before the metadata fetch and sends nothing upstream,
since the pass-through config never sends the resolved key. The
single-flight test also waits on the fetch-start signal with a bounded
window instead of a fixed sleep.
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints
A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior.
* fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check
The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions
* fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog
* fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags
* fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket
Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released
* fix(proxy): walk fallback layers without recursion and gate the compact Responses route
The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes
* fix(proxy): keep chat on rows whose cost map entry serves chat
A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s
* fix(proxy): check chat sent to a deployment id or backend model name
* fix(proxy): check a model the read-through loads from the database
* fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name
A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400
* fix(proxy): ignore a client model_group that is not a string when building rate-limit headers
A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells.
* fix(proxy): keep a non-string client model_group out of spend rows
* fix(proxy): validate standard fallbacks before the typed fallback-group lookup
* feat(router): opt in to non-chat deployment filtering
* fix(router): skip unresolved local rate-limit fallbacks
* fix(router): ignore caller-provided endpoint selector values
* fix(router): normalize declared chat endpoint prefixes
* test(router): cover preflight error-value behavior
* test(router): derive replay metadata defaults from model info
* fix(router): preserve fallback aliases and check synchronous text calls
* test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission
* test(routing): keep the endpoint gate's owned proxies on the worker's database
* fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries
A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the
field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None
so an omitted flag keeps the stored value
The same-group retry of a stream dropped before content called the provider without the endpoint selection
context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside
endpoint_selection like every other router attempt
* fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow
Main's router fence lets code outside the router use only listed Router members and
router_utils modules. The proxy's fallback probe now calls the listed
get_routable_deployments with the request's call type, and the gate's 400 moves to
litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's
UnsupportedEndpointError, which the router retries on the group's other deployments,
while the gate refuses the whole group
* test(responses): cover the WebSocket error frame for a model that refuses the endpoint
* fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet
The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup
* fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame
* fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path
* test(router): type the load-warning test's caplog and router
* test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket
---------
Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>
* feat(vector_stores): add Amazon OpenSearch as a vector store provider
* feat(ui): add Amazon OpenSearch to the vector store provider dropdown
* feat(vector_stores): add OpenSearch k-NN search as a vector store provider
* test(vector_stores): hold the OpenSearch provider tests alone for the first review
The provider source comes back in the next commits on this PR
* ci(tests): run the OpenSearch vector store unit tests
* fix(vector_stores): sign on.aws and FIPS OpenSearch hosts and reject missing AWS credentials before embedding
* fix(vector_stores): regenerate provider endpoint support for opensearch
* fix(vector_stores): keep opensearch retrieve, update, and delete off the index and refuse half-set AWS keys and per-account endpoints
* fix(vector_stores): return only scalar opensearch attributes and keep credential source output out of 401s
* test(vector_stores): inject a failing opensearch credential source instead of running a shell
* fix(vector_stores): sign opensearch for the endpoint's region, refuse list, add opensearch_index_name, honor drop_params
* fix(vector_stores): refuse a caller-chosen opensearch_index_name on search and rag query
* fix(vector_stores): refuse opensearch_index_name only on OpenSearch stores, keep the old 400 text
Other providers drop the key again as they did before OpenSearch support, and the four embedding keys keep their original error sentence.
* fix(vector_stores): refuse a caller index on slash-form OpenSearch stores
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts
* refactor(anthropic): hold the chat completions document change for a later commit
* fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai
* fix(snowflake): turn file parts into document blocks for Cortex Claude
* fix(prompt_security): sanitize chat file parts that carry file_data
* fix: answer 400 or drop PDFs for models that cannot read them
Models whose model_info or cost map entry sets supports_pdf_input to
false, and Ollama, now get the PDF dropped under drop_params and a 400
from litellm without it. GigaChat uploads file parts, SAP keeps sending
them as image_url data URLs, azure_ai passes dict content through, and
annotations other than url_citation move to provider_specific_fields
instead of failing the response.
* fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys
Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and
litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider
annotations to provider_specific_fields, and split them the same way when a stream is rebuilt.
* fix(vertex_ai): send PDF file parts to Vertex partner models as image_url
Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages
reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as
unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL.
* fix(cost-map): leave native DeepSeek PDF support unset
/v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there
and would only change chat completions, which this PR does not otherwise touch.
* fix(main): drop a user turn that held only a PDF under drop_params
Dropping the PDF from a PDF-only user turn left an empty content list, which
Foundry and OpenAI-shaped providers reject with a 400, so drop_params still
failed for that request. The emptied user turn is now removed.
* fix: read OCI PDFs, gate deployment base models, and stop empty drops
OCI generic chat sends file parts as image data URLs again, so OCI Gemini
keeps reading PDFs from /v1/messages and chat file parts.
The PDF gate now reads model_info.base_model, drops PDFs under drop_params
for azure_ai deployments that never declared PDF support (the old flattening
behavior), and answers 400 when dropping the PDF leaves no user message.
The cost map marks the Vertex partner text models that a live probe showed
answering without reading the PDF.
* test(integration): cover the PDF input gate, messages PDF file parts and provider annotations
Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed
* test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells
* fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload
* fix(main): PDF check reads the deployment's top-level base_model
* fix(main): keep roles alternating after drop_params removes a PDF-only turn
When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other
* test: type the PDF and annotation tests the PR added
* test: type the GigaChat, SAP and Snowflake PDF tests
* fix: openai exception mapper ignored status code for invalid_request_error
_map_openai_exception coerced anything whose body carried type
invalid_request_error into BadRequestError, guarded only by excluding the single
phrase Incorrect API key provided, and this ran before the status-code checks.
OpenAI and openai-compatible providers return 401 and 404 bodies with type
invalid_request_error, so an auth failure like Invalid API key mapped to a 400
instead of AuthenticationError, and a 404 mapped to BadRequestError instead of
NotFoundError. That changes the status returned to proxy clients and router
retry, cooldown, and fallback handling. Mirror the Azure mapper by only coercing
to BadRequestError when status_code is None or 400, so 401 and 404 fall through
to their existing handlers.
* fix(exceptions): map OpenAI-family invalid_request_error 404s to NotFoundError
Move the contributor's test to the mapped unit test file, assert the raised
class, status code and message, and cover the 400 and status-less cases that
must still raise BadRequestError
Add a regression test for the flow the bug was found on. Retrieving a missing
OpenAI vector store, sync or async, now raises NotFoundError with status 404
The router test that pinned an OpenAI 404 typed invalid_request_error as
BadRequestError now expects NotFoundError
* fix(a2a): answer a provider NotFoundError as JSON-RPC invalid params
* fix(exceptions)!: limit the invalid_request_error status guard to 401 and 404
Every other status keeps raising BadRequestError as on main, so 402-415, 429 and 5xx bodies typed invalid_request_error keep their class and router retry behavior. A2A goes back to main's mapping, which answers NotFoundError and AuthenticationError as JSON-RPC internal errors.
* fix(exceptions): keep the provider error fields on mapped 401 and 404 invalid_request_errors
Raise AuthenticationError and NotFoundError straight from the invalid_request_error
branch with the provider body, so code, param and type survive and the proxy's
error.type matches main. The Rust mapper copy skips 401 and 404 the same way.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(soniox): add realtime speech-to-text on /v1/realtime
Maps OpenAI realtime transcription sessions onto the Soniox websocket API
(stt-rt-v5): server VAD uses Soniox endpoint detection, commit sends a
manual finalize, deployment litellm_params pass Soniox options through,
translation sessions transcribe the translated tokens, idle streams get
keepalives, and cost is tracked from processed audio duration.
* fix(soniox): bill realtime usage from integer milliseconds
* fix(soniox): bound session-start queue and cover realtime backend edge paths
* fix(soniox): bill realtime stream time, start sessions on connect and surface client protocol errors
Soniox charges for the whole open stream, so session close now bills the connection's open time. The backend announces the session on connect, which flushes audio sent before any session.update, and a single upstream pump makes recv safe to cancel. Invalid client frames now get an OpenAI error event instead of a silent disconnect
* fix(soniox): skip deferred setup so client frames reach Soniox in order
Drops the connect marker frame, which counted as an upstream frame and hid refusals, and could start a buffered stream at the default rate before the client's session.update. Soniox now opts out of deferred setup, the handler announces a synthetic session.created whenever no setup went upstream, a session.update that would change a started stream gets an error event, and stream time is billed only once a stream started
* fix(soniox): judge a follow-up session.update only on the settings it sends
A partial follow-up session.update was rebuilt with defaults for the settings it left out, so restating part of a 16 kHz or VAD-off stream was rejected as a change
* fix(soniox): treat an omitted rate in a follow-up session.update as unchanged
* chore(soniox): drop comments that restate the code
* fix(soniox): return speaker labels, log refused sessions as failures, and probe realtime health
Deltas carry speaker and language when diarization or language identification is on, and the
completed event lists segments per speaker and language. A session Soniox refuses before any
audio is processed is logged as a failure with no spend. Billing starts at the start request.
Deferred-setup providers (Meta, Vertex Chirp) send the error event and still end the session.
Soniox realtime models get a websocket /health probe and cost rows for the stt-rt-v5 aliases.
* fix(soniox): bill failed streams, probe health with deployment options, and keep token accumulation linear
Only request rejections (400, 401, 402, 403, 429) before any processed audio count as a refused
session, so a 408 or 5xx still bills the open stream. The /health probe starts Soniox with the
deployment's options, sends 100 ms of silence, and passes only on Soniox's finished frame.
A turn's final tokens append in constant time instead of copying the tuple per token.
* fix(soniox): log every Soniox error before any processed audio as a failed session
* fix(soniox): keep untranslated speech, support input_audio_buffer.clear, ignore an empty language
* fix(soniox): reject frames without a type, keep strict language hints, queue finalizes in a deque
* fix(soniox): answer a commit with no speech with an empty completed turn
* fix(soniox): let a server VAD turn answer a commit that is still in flight
* fix(soniox): let a server VAD turn answer only the oldest pending commit
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(fallback): read stored router_settings from the database on fallback writes
POST /fallback and DELETE /fallback/{model} read the current rules through the
config cache. When Redis refuses DEL, the eviction before that read fails and
Redis keeps serving the old copy, so each new rule dropped the earlier ones
The write path now reads the router_settings row straight from the writer
database and stores only that row plus the changed key, so config-file router
settings no longer get copied into the database row
* fix(fallback): edit the config file's fallback list when it owns the key
A delete of a rule only the config file holds answers the config-owned 400 again instead of 404, and restating a rule the file already has answers 200 without copying the file's list into the database row. Adds a writer-read regression test and an integration test against a Redis whose ACL refuses DEL
* fix(fallback): re-read the config file's fallbacks ownership on every fallback write
* test(fallback): run the real config reload in the fallback write tests
* test(fallback): key the refused-DEL Redis double by cache key
* fix(fallback): keep the config-owned 400 for an unset env reference and replace a non-object stored row
* fix(fallback): replace a stored router_settings row that holds a non-JSON string
* fix(fallback): compare a restated config-file rule against its resolved env value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* refactor(types): replace Any with proven types in 57 files
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): hide input values in validation errors and drop secret-bearing seams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert the Phoenix prompt seam that rejects string content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert files that break mocked tests and Python 3.10 imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert pegasus response validation to main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(types): revert azure search, milvus and onyx response validation to main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep accepting numeric keys in /cache/delete
* fix(caching): answer 400 for a non-list /cache/delete keys and accept null Stability lists
Return pydantic errors without the caller's input on a bad /cache/delete body, let the Bedrock Stability
image edit schema take null finish_reasons and images like the old reads did, declare user_id and user_role
on the plugin session claim, wrap the retyped Vertex fields in ReadOnly, and drop unused Guardrails AI types
* refactor(types): leave provider reply parsers for a follow-up with live provider calls
Restores the Bedrock Stability, Bedrock KB, AgentCore, Replicate, Recraft, Aleph Alpha and Guardrails AI parsers to main, since no live provider call could prove the stricter schemas. Also types the cache delete route tests.
* refactor(types): keep /cache/delete on main's behavior
Restores the cache delete route and its test to main, so this types PR changes no response. Rejecting a non-list keys is left for its own change, since an object keys deletes its keys on main today.
* fix(types): keep text completion, batch ids and guardrail callback on main's code
* fix(types): keep policy suggestion and auto router parsing on main's code
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: run the dashboard API types check on every change that can alter the OpenAPI spec
* chore: shorten the spec_pattern comment to a pointer at the CI filter
* chore: drop the spec_pattern comment
* docs: name the full ui-api-types trigger set in the make check header and dashboard AGENTS.md
* test: check make check and CI pick the same files for the API-type sync
* ci: run the Terraform provider endpoint audit on enterprise route changes
* test: keep the Terraform trigger test in the mapped workflow test file
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta
When a provider leg drops an anthropic-beta value (the chat handler's allowlist
filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields),
also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates,
mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the
request degrades on that leg instead of failing with a provider 400. Copy-on-write so a
fallback leg still sends the full request, and each strip is logged at WARNING.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove a leg that drops a beta sends neither the beta nor clear_at
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): strip beta-gated fields against the final body anthropic_beta list
Converse counts the body anthropic_beta list as active, Invoke strips after
its auto-injected betas are known and injects thinking-binding-controls for
thinking.block_binding, test helpers are fully typed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn
Models without supports_mid_conversation_system send no mid-conversation beta,
so tool_addition and tool_removal blocks are dropped (and logged) before the
turn is converted instead of riding along as user content. The Invoke
output_config regression test reads the real cost map instead of patching
_bedrock_model_supports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): annotate every new test local with Final and type the fixtures and returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): strip beta-gated body fields only under drop_params
Without drop_params the request now goes out as sent and the provider
answers its own 400, like any other unsupported param. inline-tools-2026-09-15
also unlocks tool_addition and tool_removal blocks, so a leg that carries it
keeps them.
* fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params
Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field.
* fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking
Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes.
* fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown
Vertex messages and Claude Platform chat now add the thinking betas when thinking carries
block_binding or display updates. Converse sends only the caller's header betas when it has them.
A provider with no row in the beta config forwards beta-gated fields instead of dropping them,
an emptied trailing system turn that would leave an assistant prefill is sent as is, and the
prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta
that unlocks each field
* fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks
* fix(anthropic): validate messages only when a beta-gated drop runs
A non-object messages entry on a Claude model without mid-conversation
system support raised a raw pydantic ValidationError before the
beta-config and drop_params checks. The strip now passes the messages
through untouched unless it drops something, so the provider's 400
reaches SDK callers as BadRequestError again
* fix(anthropic): keep system blocks with a non-string type instead of raising
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): send region-prefixed embedding model ids to that region without the prefix
* chore(bedrock): hold the embedding fix back until the risk stamp
* fix(bedrock): send region-prefixed embedding model ids to that region without the prefix
* fix(bedrock): keep a model_id's region and price embeddings at the region the call goes to
* fix(bedrock): read a model_id ARN's region for embeddings and keep unlisted regions mappable
* fix(bedrock): route async_invoke embeddings whether the region comes before or after it
* fix(bedrock): price embeddings by region_name and keep the path region with a plain model_id
* fix(bedrock): price a model_id ARN at its region's row
* fix(bedrock): keep chat's region rules for embeddings and leave ARN pricing as on main
A plain model_id skips the region in the model, as chat does, so a deployment that
worked on main keeps calling the same region. An empty aws_region_name fails region
validation again instead of falling back. This reverts the ARN regional pricing, since
many regional rows lack the bare row's priority, batch, and long-context rates
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(tests): keep foreign asyncio task noise out of the budget limiter caplog asserts
The two gc.collect()-under-caplog tests in test_budget_limiter_hotpath.py asserted
on every captured record, so a LoggingWorker task another test had parked on the
session fixture loop got destroyed inside their window and asyncio's "Task was
destroyed but it is pending!" error failed them. They now assert only the limiter
logger's records and park a pending task on a closed loop in the window themselves,
so the noise is exercised on every run
The edenai and sail spend_capture fixtures no longer start the worker from the
session loop: the test's own call and its flush bind the worker to the test loop
* test(router_strategy): drop only asyncio records in the limiter caplog asserts and drain the response cache worker on the test loop
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(passthrough): forward OpenAI-Beta and default agents=v1 on the OpenAI passthrough
The OpenAI-shaped passthrough now forwards the client's own OpenAI-Beta header (agents, chatkit, realtime and any future beta), keeps an operator's extra header over it, and injects agents=v1 on /v1/agents and /v1/vaults paths when the client sent none. Unbilled Agents API calls now log the agent's model on the spend row instead of unknown.
* fix(passthrough): keep OpenAI-Beta forwarding on OpenAI only and name the model on streamed and saved agent rows
Azure and Azure AI keep main's assistants=v2 rule. ChatKit paths get no route default. Streamed agents rows read session.agent.model from the session events instead of defaulting to gpt-4o, and saved agents name their top-level model.
* fix(passthrough): forward the client's OpenAI-Beta only on agents, vaults, and ChatKit paths
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(caching): audit #44542 response cache model group key
* fix(caching): key the response cache by the router's metadata bucket only
* fix(caching): read the router's model group before the caller's metadata
* fix(caching): read caching groups only beside the router's model group
A caller's own metadata.caching_groups on /v1/responses and /v1/messages could
still pick the cache key, so a string group let one model group fill another's
entry. The key now reads caching_groups only from the metadata that holds the
router's model_group.
* test(caching): type the planted caching groups parameter
* fix(caching): read the model group from the metadata that carries the call's deployment
A router call can carry a litellm_metadata it did not write, from the auto router's classifier embedding or an SDK caller, while the router writes its own group into metadata. The cache now picks the metadata source whose model_info matches the call's deployment, then falls back to the previous order.
* fix(caching): keep main's model group lookup for calls outside the router
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): stream a guardrail block on /v1/messages as Anthropic events
* fix(proxy): keep post-call guardrails off the streamed messages block
The streamed block is sent as-is, like the Responses endpoint's block, so post-call guardrails and streaming hooks never scan the proxy's own block message. Tests cover a post-call rail on the stream and the call id on the JSON block
* test(proxy): expect Anthropic events from a streamed messages guardrail block
The legacy guardrail suite pinned the old single bare data: line, which is the
malformed block this branch replaces. The test now checks the six Anthropic
stream events and the block text, so it fails if the bare line comes back
* test(proxy): block messages through a real pre-call guardrail in endpoint tests
The block tests replaced ProxyBaseLLMRequestProcessing.base_process_llm_request on the class to raise the block. They now send a real request through the endpoint with a pre-call guardrail that raises the passthrough block, and check that the x-litellm-call-id header carries the request's call id
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): honor per-second custom pricing on chat completions for every provider
* test(cost): assert a per-second priced deployment bills instead of staying at $0
The zero-cost diagnostic tests from #42345 used a per-second-only entry as their
misconfigured fixture, which this branch now bills. Switch that fixture to a
per-query-only entry, which is still selected as the deployment's own pricing and
still prices chat usage at $0, and add a per-second test asserting the call
duration is billed with no diagnostic
Also let a caller's explicit total_time outrank the logging window in
completion_cost, so the SDK precedence stays stamped response, caller, logging
* test(response_metadata): move the per-second pricing regression into the mapped tests/unit file
* fix(cost): bill a deployment's per-second rates on /v1/responses and /v1/messages
The proxy builds the request's logging state before the router picks a
deployment, and only the chat path re-read the deployment's
input_cost_per_second / output_cost_per_second afterwards. The shared
update step every other route runs (Logging.update_from_kwargs) now
seeds those custom pricing keys from the router-merged kwargs too, so
custom_pricing flips on and the cost calculator prices the call against
the router-registered deployment id on every surface.
* fix(cost): price a /v1/messages call off the response time the proxy reports
A /v1/messages result is a plain dict, so the proxy keeps its response time on
the logging object. The cost step now reads it there for a non-streamed call,
so the cost header and the spend log bill the same duration.
* fix(anthropic): bill a deployment's per-second rate only on the native /v1/messages route
Move the custom pricing copy from the shared logging refresh into the native
Messages handler, reusing the helper the Responses handler already had, so
generateContent and other routes keep their current billing
* fix(cost): pass only per-second rates to native /v1/messages cost tracking
Other deployment overrides keep billing as on main, matching chat's per-second gate
* test(anthropic): pin per-second-only pricing on native /v1/messages with a long-context-rate deployment
* fix(cost): bill native /v1/messages from logging setup, as main did, to the end the proxy reports
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
