* fix(router): cap the default OTPM reservation for requests without max_tokens
* fix(router): reconcile ITPM and OTPM reservations after every successful call
* fix(router): read the default OTPM reservation from the served model and make its fraction configurable
The default reservation now looks up max output from litellm_params.model, as main does, so a pricing base_model never changes it. NO_MAX_TOKENS_TPM_RESERVATION_FRACTION sets the fraction for both the router check and the key and team TPM floor. A max_tokens that is not a number or string counts as no cap instead of raising inside the check.
* fix(router): refund a failed attempt's io token reservation in make_call
The proxy passes one logging object to every retry, and it runs failure
callbacks only for the first attempt, so the last attempt's ITPM and
OTPM reservation stayed held until the minute ended. Each attempt that
raises now gives its reservation back before the retry or fallback
* fix(router): count Anthropic cache writes once in ITPM reconcile
Native /v1/messages usage reports input_tokens without cache reads and writes, so the reconcile now maps it through AnthropicConfig.calculate_usage before subtracting cached tokens.
* fix(router): use a deployment max_tokens, charge late usage to its finish minute, and read the reservation fraction as a fraction
* fix(router): reserve with the deployment cap only through the param the request's API sends
* fix(router): never let a deployment's output cap raise the reservation above the default
* test(router): move the reservation fraction env var test to the integration suite
The unit test started a child interpreter, which the unit guide forbids. The integration test boots a real proxy with the env var set and checks admission against the held reservation.
* test(router): retry the reservation fraction scenario when the minute rolls over
The OTPM counter is keyed by minute, so a rollover between the held stream and the capped requests gave the capped request a fresh counter. The control request now leaves room for usage the first attempt left in the new minute.
* test(router): give the rollover retry its own deployment
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost): bill a provider reported usage.cost on every path unless the deployment is priced
* test(integration): a reported cost is attributed to output in the gateway parity cases
* fix(cost): skip a chat usage.cost that already holds litellm's stamped bill and Responses API objects
* fix(cost): record litellm's bill on the Responses terminal event so a later pricing pass does not read it as the upstream's charge
* fix(cost): mark usage.cost litellm wrote as its own bill and keep a breakdown for a reported cost
* fix(cost): keep the token breakdown when a header reported cost repeats an already priced bill
* fix(cost): read the stored breakdown total through the logging object's typed field
* refactor(cost): name the providers billed from their own usage fields for what they are
* fix(cost): keep the upstream's usage.cost beside litellm's bill and stamp a zero operator price on the stream frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop answering with the literal string "None" on the proxy-server, auth, and health routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* fix(proxy): stop answering with the literal string "None" on the spend and management routes
The error objects these routes return defaulted `type` and `param` to the
four-character string "None", so a client's error handler matched no known
OpenAI type and fell into its generic branch.
Route them through the shared openai_error_payload helpers instead, so `type`
comes from the carried type or the HTTP status and `param` serializes as JSON
null.
* test(proxy): pin the mid-stream failure frame and the DB outage body to a JSON null param
* fix(proxy): type upstream timeouts as timeout_error and update integration asserts for the null param
A 408 with no carried type now answers timeout_error instead of invalid_request_error, so a client branching on type keeps retrying it. Three integration suites still compared auth rejections against param="None".
* fix(proxy): keep a 408 on main's invalid_request_error type and pin the transcription file cap's 400
* fix(proxy): type a route failure by the status it answers with after the tag-routing and no-deployment remap
* fix(proxy): keep auth_error on proxy-raised 401/403 route failures and ban the string None as an error type or param
* fix(proxy): type a hook's 401 or 403 on completions, moderations and transcription by its status, as chat does
Only a status the proxy remapped into 401 or 403, such as a tag-routing denial, keeps auth_error.
* test(proxy): build the transcription file-cap test on a real router so it holds after the router refactor
* feat(dashscope): add image edits and text-to-speech
* fix(dashscope): bill reported speech characters and input images, follow the chat region for edits, keep the call timeout on the audio download
* fix(dashscope): bill deployment image prices as main does and route edits to the host speech uses
A deployment that sets its own input_cost_per_image, output_cost_per_image or input_cost_per_pixel bills through the default image calculator again, so an override is never read as a per-input-image price and pixel pricing on qwen-image-3.0-pro no longer raises.
Edits now resolve the caller's api_base or DASHSCOPE_API_BASE through the same API-root helper speech uses, on any host, with DASHSCOPE_API_BASE_IMAGE still winning. An image response whose usage is null or not an object keeps its images instead of raising.
* fix(images): bill a deployment's own prices on image edits through the proxy
The proxy builds the logging object before the router picks a deployment, so image_edit never handed the deployment's price fields to its cost calculation and billed the cost-map price instead. Pass the custom pricing params with the pre-call update, as the Responses API does.
* fix(images): bill per-pixel image prices at a DashScope W*H size
DashScope's mapped size reads 1328*1328, which the cost size reader rejected, so a deployment's input_cost_per_pixel billed every DashScope generation or edit at the 1024x1024 default.
* fix(dashscope): bill image prices by their meaning, report the rendered size, prefer the deployment edit base, fetch audio over https
* fix(dashscope): read the rendered image size from output_width and output_height
* test(images): bill a fal edit at its deployment's own image price
* fix(dashscope): keep the qwen-image-3.0 input-image fee out of the shared cost map row
Released proxies read the cost map from main and bill input_cost_per_image as the
flat price of each DashScope image, so the row's $0.003 input fee billed every
qwen-image-3.0 image there at a tenth of its $0.03 price. A deployment can still
set input_cost_per_image to bill DashScope's input images
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): bill streamed /v1/completions without stream_options like chat
* test(proxy): mark the text completion stream usage test variable Final
* fix(completions): drop Azure's empty prompt-filter frame when the proxy asked for stream usage
* test(spend): drop the module docstring from the text completion stream usage test
* fix(proxy): keep the internal stream usage strip flag out of the logged request body
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(openapi): build native extension for base counts
* test(openapi): exercise native import in base worktree
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Without drop_params, every copy of a tool from additional_tools input items
is forwarded as sent, so a provider that rejects duplicate tools returns its
own 400. Codex keeps the merge because the proxy turns drop_params on for it.
* test: add a manually dispatched integration trial on larger runners
Runs the providers, database and management integration groups on 16-core GitHub-hosted runners with Postgres and Redis as service containers, to measure wall time and how far each group scales with more workers
run_integration.sh now keeps a preset INTEGRATION_WORKERS, takes the service container names from the environment, and seeds test order from the GitHub run when there is no CircleCI one
* test: run the integration trial on push to this branch
* test: give integration trial workers a local redis-server
* test: run the providers trial on a 64-core runner with 32 workers
The 16-core run reached 53% of providers in 45 minutes with 16 workers, which points at CPU. Postgres gets max_connections=1000 and a 1 GB shm because one server now backs 32 worker proxies, where each CircleCI machine backs 4
* test: run every integration group in the trial on sized runners
* test: add a scan that refuses to upload integration results holding the license
* test: drop the member-name pass the archive byte scan already covers
* test: use a secret whose base64 and base64url encodings differ
* ci: run integration groups on run-ci pull requests and scheduled main runs
* ci: scope the integration license to a protected environment
* fix(otel): stop OTel v2 vendor presets from printing every span to stdout
With LITELLM_OTEL_V2 on and a vendor callback such as langfuse_otel or arize
holding operator credentials, the proxy printed every span as JSON to stdout,
prompts included, on the request path. Nobody set OTEL_EXPORTER, but the
console placeholder that OpenTelemetryV2Config folds in for an empty exporter
list stayed next to the vendor's own exporter.
The provider now skips that placeholder when the config has any other
exporter. A console exporter the operator names with OTEL_EXPORTER=console is
kept, and so is the placeholder when it is the only exporter.
* test(otel): type the console-placeholder tests and drop inherited OTEL_* env from the stdout check
* fix(otel): print the console default only for the otel callback
The proxy's background V2 logger (published when no callback built one, which is
what key-only and team-only Langfuse destinations run under) also kept the
unconfigured console exporter, so it printed every span with prompts to stdout.
The console default now prints only for a listed otel callback or an exporter the
operator named.
* fix(otel): keep the otel callback's console output on service-name routes
A key or team otel_service_name moves the call onto a provider the tenant
tracer cache builds, which never got the console choice, so the plain otel
callback stopped printing those calls. The no-operator-keys integration rig
also drops inherited Langfuse keys.
* feat(guardrails): scan sampled stream windows in the background (LIT-9338)
streaming_scan_in_background runs each sampled post_call scan as a task so
the stream keeps flowing; a block ends the stream before the next chunk.
* fix(guardrails): hold the stream end for the background verdict and cancel the scan on upstream errors (LIT-9338)
* fix(guardrails): wait for the cancelled background scan on upstream errors (LIT-9338)
* test(guardrails): cover /v1/responses, a failing scan peer and a mixed burst for background scans (LIT-9338)
* test(guardrails): build the burst calls without a nested comprehension (LIT-9338)
* fix(otel)!: keep a credential-less langfuse_otel on OTel v2 so key destinations get the Langfuse mapper
With LITELLM_OTEL_V2 on, langfuse_otel in callbacks and no operator
LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY, the preset raised at startup and
the callback fell back to the legacy logger. Key and team destinations were
then served by a generic v2 logger without the Langfuse mapper, so their
traces lost tags, user and cost, and the legacy logger added an orphaned
raw_gen_ai_request span.
The preset now always degrades to the credential-gated exporter when the
operator has no keys, the same way arize_preset does, so the startup logger
stays on v2, exports nowhere for traffic without a destination, and carries
the Langfuse mapper to every key and team destination.
* test(otel): type the langfuse tenant-only test helpers
* fix(otel): give key/team Langfuse destinations the Langfuse fields when langfuse_otel is not a callback
With langfuse_otel only in a key's or team's logging, the startup logger is the
generic one and its spans reached the key's Langfuse project without user,
session or tags. The emitter now adds the Langfuse mapper for a request with a
Langfuse destination, and every non-Langfuse exporter strips the Langfuse
request metadata, since any logger can now write it.
* fix(otel): stop a key's Langfuse logger from re-sending spans to the operator's collector
With langfuse_otel only in key or team logging, the proxy publishes the generic
V2 logger at startup and it already exports to the operator's OTLP collector.
The Langfuse logger a key's request builds kept the same base exporter, so the
collector got every model span twice. A logger built after the publish now keeps
only its preset's own exporters, as it already did beside a configured V2 logger,
and a later call that would rebuild one exporting nowhere reuses it.
* test(otel): assert the collector gets one copy instead of reading the key logger's exporters
* test(otel): count the collector's spans with one comprehension clause
* fix(otel): keep the legacy langfuse_otel logger and stand it down for requests a Langfuse destination carries
Reverts the preset degrade and the published-provider exporter trim, so every
request without a key or team Langfuse destination keeps main's legacy logger.
For a request with one, the V2 fan-out already delivers the whole trace with
the Langfuse fields, so the legacy logger emits nothing for it.
* test(otel): pin the legacy langfuse_otel paths a tenant's project still gets without a destination
* fix(otel): shed a destination's Langfuse request metadata before its other Langfuse fields
At its attribute limit a span sheds the Langfuse fields only a destination asked
for first, in the mapper's order, so the generation marker went before the
request metadata blob. Put the request metadata first, as a Langfuse-configured
logger already does.
* fix(otel): keep the legacy langfuse_otel span for a destination's call rejected before any provider
The OTel v2 fan-out records no call for a request the proxy rejects before a
provider (rate limit, pre-call hook), so standing the legacy logger down there
lost the user, session and tags main sent the tenant. Stand it down only for a
call the fan-out records, using the same check the v2 logger uses.
The tenant and collector integration tests now hold their exact counts for a
few seconds, so a late duplicate copy fails them.
* fix(otel): give a key's Langfuse destination the langfuse_environment the legacy logger wrote
The destination's resource carries langfuse.environment from the key's or team's langfuse_environment, else LANGFUSE_TRACING_ENVIRONMENT, so a key's project files its traces under the environment it chose instead of default. Only that destination's export carries it.
* fix(otel): keep the Langfuse fields written for a key's Langfuse destination off every other exporter
* fix(otel): give a key's Langfuse destination the prompt and response the legacy logger sent, only where it ran
* fix(otel): keep a key's Langfuse destination on the global capture setting when it sets none, with or without global Langfuse keys
* fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask
on_flagged block now returns 400 when Lakera flags only PII, instead of
masking it and forwarding the request. on_flagged mask keeps the old
behavior: mask PII-only detections and block anything else.
* test(guardrails): cover lakera_v2 on_flagged on every endpoint, hook and a Lakera outage
* test(guardrails): type the lakera_v2 on_flagged unit test helpers
* feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): integration cells for PANW include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): reject non-boolean include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(guardrails): update PANW tool-call comment for include_masked_response_on_block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): build PANW kwargs without mutation and clean up DB-created test guardrails
Restore main's single-expression kwargs in the PANW initializer. The two DB-backed integration tests now create their guardrail inside the scenario and delete it on exit.
* fix(guardrails): type raw PANW boolean setting before validation
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(vertex_ai): use HTTP/2 httpx client for search_api vector store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(httpx): exercise configured request path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(httpx): verify HTTP/2 request negotiation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(httpx): preserve HTTP/2 for sync proxy mounts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* build(deps): add h2 to proxy extra so vertex search HTTP/2 is on by default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http_handler): isolate ssl context for http2 clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(http): add an HTTP/2 opt-out and harden the Vertex AI Search HTTP/2 client
litellm.http2 is now unset by default, so http2: false or LITELLM_HTTP2=False
turns the provider's HTTP/2 default off. A caller's own SSLContext keeps the
search on HTTP/1.1, the HTTP/2 async client honors DISABLE_AIOHTTP_TRUST_ENV,
forced IPv4 transports get the CA bundle and client cert, and vector store
search retries once when the server drops the connection with a GOAWAY.
* fix(http): retry vector store search only on HTTP/2 GOAWAY and keep aiohttp pool and proxy behavior
Scope the one retry to a connection Google closes with an HTTP/2 GOAWAY and
log it. When HTTP/2 replaces the aiohttp transport, keep aiohttp's idle
keepalive and connection cap, and only then skip env proxies under
DISABLE_AIOHTTP_TRUST_ENV, so the global LITELLM_HTTP2 toggle keeps HTTPS_PROXY
* test(http): cover the vector store search GOAWAY retry in unit tests and drop duplicate SSL cache comments
* fix(http): open another HTTP/2 connection once 100 streams are in use
httpcore keeps every request to an origin on one HTTP/2 connection and caps it at 100 concurrent streams, so request 101 and later waited with no timeout. The handlers now spread requests over extra connections, opening one only when every open connection has 100 streams in flight. The force_ipv4 wire tests now cover ssl_verify=False and the client certificate.
* fix(vertex_ai): reuse one Vertex search OAuth token across searches
Every vector store search built a new search config, so the token cache it carries was thrown away and each search fetched a new Google OAuth token with a blocking call on the event loop. Under a burst of searches that blocking fetch held back every HTTP/2 response. The provider config is now shared, so searches with the same credentials reuse one token until it expires.
* fix(http): keep the HTTP/2 connection spread within the connection cap
* fix(http): keep sync vertex search on HTTP/1.1, since httpcore's sync HTTP/2 is not thread-safe
httpcore 1.0.9 hands two threads the same HTTP/2 stream id, so threaded sync
searches failed with protocol errors. Only the async client, which the proxy
uses, now defaults to HTTP/2, and the sync connection spread is removed.
* fix(http): split the connection cap across HTTP/2 lanes
Each lane had its own pool with the full connection cap, so an origin answering
HTTP/1.1, or several origins on one client, could exceed max_connections once
more than 100 requests were in flight. Lanes now open only while lanes times
100 streams stay under the cap, and each lane pool gets at most 100 of it.
* test(http2): scope wire-test peer logs to each fixture instance
* fix(http2): keep TE, force_ipv4 trust, evicted-client and opt-out behavior on the HTTP/2 path
- drop TE values other than trailers on https requests through the HTTP/2 spread, since h2 refuses them
- force_ipv4 transports keep httpx's default trust store (SSL_CERT_DIR) unless SSL is configured
- the evicted client closer asks the HTTP/2 spread whether a request is in flight
- LITELLM_HTTP2 set to 0, no or off, and a quoted "false" http2 setting, opt out of HTTP/2
- the wire test's held-search barrier waits on an event instead of polling
* fix(http2): count proxy-mounted transports when checking an evicted client for in-flight requests
* test(http2): move the HTTP/2 eviction cases that open a loopback server into the integration wire tests
* fix(http2): read an unquoted http2: 0 as off and free the stream slot of a response dropped without closing
* fix(http2): free the stream slot of a partly read response dropped without closing
* test(http2): check that the evicted client and its stream close after a partly read response is dropped
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): preserve reasoning fields on streaming cache hits
* fix(proxy): validate cached streaming delta fields
Signed-off-by: jesco-absolut <team@srswti.com>
* fix(caching): replay cached audio, images, reasoning and citations in stream order
A cached answer replayed as a stream now ends with its own empty finish chunk, so a chunk that
carries only audio or images keeps its payload and role. Reasoning fields lead on a chunk with no
text, and annotations arrive once on the last text slice, matching a live stream's order.
* fix(caching): replay a cached tool call after the text, like a live stream
Only the role, reasoning fields and provider fields lead a replayed answer. Tool calls now arrive on
their own chunk after the last text slice, and logprobs ride the first text chunk, so a cached
answer with reasoning and a tool call keeps a live stream's reasoning, text, tool call order.
* fix(caching): replay the reasoning before a cached tool call that has no text
A cached answer that thinks and then only calls a tool now replays its reasoning on its own chunk before the
tool call, the order a live stream sends, instead of one chunk carrying both.
* fix(caching): keep blank-only text when replaying a cached answer with reasoning
A cached answer that thinks, sends only whitespace as text, and then calls a tool now replays that whitespace
between the reasoning and the tool call, instead of dropping it.
---------
Signed-off-by: jesco-absolut <team@srswti.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): replace the pre-migration row check with a rolled-back dry run of the pending migrations
* feat(proxy): roll back each migration of the dry run on its own and baseline-check databases without a ledger
* test(proxy): cover the dry run CLI and PrismaManager wrapper without a database
* fix(proxy): refuse the no-ledger dry run for a custom migration folder like boot does, and probe the rolled-back indexes directly
* ci(codecov): stop carrying forward the roi-postgres and redis-compat flags
Their carried-forward sessions were measured against older proxy_cli.py revisions (633 and 689 line maps against 710 today) and mark docstring lines of the merged file as missed, which failed codecov/patch on this PR at 70% while every unit shard reports the lines hit
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: drain shutdown callbacks and persist pending tag spend
* fix: catch shutdown timeouts on Python 3.10
* fix: share shutdown callback budget with HTTP draining
* fix: type shutdown cancellation tasks and document its clock
* test: retain callbacks after bounded worker cancellation
* test(proxy): inspect the lifespan body after the SIGTERM wrapper split
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(utils): keep the success callback enqueue on its own task
The synchronous enqueue spawned the logging worker loop inside the awaited
call and ran callbacks that per-call loops used to drop, which hung the SDK
callback tests on the autouse worker flush. The shutdown worker flush covers
the queued callback on its own, so the one-tick task is restored
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback
The aggressive clear ran the rejected callback and marked it done against
the queue it was never put in, so flush() could return while a real
callback was still running and the shutdown drain cancelled it early. The
rejected callback is now re-enqueued once the clear made room, and only
runs uncounted when the queue is still full
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(logging_worker): build the aggressive-clear run set immutably
Final tuple instead of an appended list, with the re-enqueue in a
_requeue helper, and Final annotations on the overflow regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): track overflow callbacks through shutdown
* chore(logging): explain shared callback ownership
* test(logging): isolate deferred callback completion
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines
Gives OpenAITextCompletionHandler a process_output_streaming_response that
assembles the buffered TextCompletionResponse chunks per choice index at end
of stream, runs the guardrail over the assembled text, and writes text
rewrites back into the first text-carrying chunk of each choice. Tests and
live proof still pending.
* fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails
Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs.
* test(guardrails): cover streamed /v1/completions under post_call guardrails
* fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through
* fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter
* fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent
The content filter's streaming hook held back the end of a choice until its
finish_reason, which litellm sends only for the first choice of an n>1
stream, so choice 1 lost its last characters on text and chat streams. A
closing chunk now releases every unfinished choice when the stream ends.
A blocked text stream's closing chunk reported 0/0 usage when the upstream
sent none; it now carries usage only when the upstream reported it.
* fix(guardrails): send no usage on the content filter's end-of-stream chunk
* fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions
* fix(guardrails): stream Responses API events through the content filter as they arrive
* fix(guardrails): treat a null text completion choice index as choice zero when assembling streams
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(secret_managers): cover oidc/google api_key references resolved per request
Unit tests send a metadata server ID token for an admin oidc/google/<audience> api_key, reuse it within its
lifetime, refresh it before it expires, fetch it off the event loop, fail closed with a 401 that names only the
kind, and honor GCE_METADATA_HOST. The integration file scripts a metadata server and checks the bearer the
upstream receives for config, stored credential and DB credential sources, plus a metadata failure
* feat(secret_managers): resolve oidc/google api_key references per request
An admin-set oidc/google/<audience> api_key now resolves on each call to the GCE metadata server's ID token for that audience, cached until a minute before it expires. Async callers fetch on a cache miss in a worker thread, so the event loop never waits on the metadata server. A failed fetch answers 401 naming only the kind, and GCE_METADATA_HOST points the fetch at another metadata server.
* fix(oidc): bound the Google metadata fetch and answer 503 when it fails
A Google OIDC api_key fetch now waits at most GCE_METADATA_TIMEOUT seconds
(default 3), runs on its own thread pool instead of the loop's default
executor, and sends the token read from the cache at call start even if it
expires mid-call. A metadata timeout, network error, or 5xx answers a
retryable 503, while a 4xx or empty token stays a 401. The expiry and
empty-token warnings log the secret name on one line.
* test(oidc): expect 503 when the metadata server answers 5xx in the integration test
* fix(oidc): answer 503 on token-count routes during a metadata outage and fall back on a bad GCE_METADATA_TIMEOUT
* fix(secret_managers): keep get_secret's metadata wait, share concurrent google token fetches, refuse oidc/google on pass-through
A model's api_key fetch keeps the 3 second GCE_METADATA_TIMEOUT wait while get_secret callers keep the handler default. Any metadata transport error answers a retryable 503. Async cache misses for one reference share one fetch, and a call the reference may not go to is refused before any fetch. Pass-through routes answer 401 for an oidc/google credential instead of sending the reference text. The metadata client is closed after each fetch, and the unused sync resolver is removed.
* fix(utils): refuse an oidc/google api_key on pass-through calls before any fetch
A router-model or credential-name pass-through call with an oidc/google
reference now fails before the metadata fetch and sends nothing upstream,
since the pass-through config never sends the resolved key. The
single-flight test also waits on the fetch-start signal with a bounded
window instead of a fixed sleep.
* feat(proxy): reject chat routes on models declared non-chat via model_info.supported_endpoints
A model whose every deployment sets a non-chat model_info.mode and a supported_endpoints list with no chat endpoint now answers /v1/chat/completions, /v1/messages and /v1/responses with a 400 before any provider call. Models without both fields keep today's behavior.
* fix(proxy): let fallbacks answer and cover more chat routes in the non-chat model check
The check now steps aside when a request, key, team or router fallback would answer, skips such models in the rate-limit fallback loop, ignores mode and supported_endpoints copied from the cost map by old Admin UI saves, treats a blank mode as unset, and also covers /v1/completions, the Responses WebSocket, Gemini generateContent and /v1beta/interactions
* fix(proxy): only step aside for fallbacks that can answer chat, gate the Responses WebSocket after alias resolution, and compare declarations to the loaded catalog
* fix(proxy): gate the Responses WebSocket with the router after aliases resolve, and ignore client-set internal fallback flags
* fix(proxy): walk fallback groups breadth first and release reservations on a rejected WebSocket
Each fallback group is looked up once however often the request repeats it, and a WebSocket closed by the endpoint gate runs the failure hook so rate-limit reservations are released
* fix(proxy): walk fallback layers without recursion and gate the compact Responses route
The fallback walk now folds one layer at a time, so the recursion check passes, and /v1/responses/compact on a model declared non-chat gets the same 400 as the other chat-shaped routes
* fix(proxy): keep chat on rows whose cost map entry serves chat
A row that carries the cost map key of an entry that serves chat counts as declaring nothing, so a catalog edit that adds a chat path never turns old Admin UI copies into 400s
* fix(proxy): check chat sent to a deployment id or backend model name
* fix(proxy): check a model the read-through loads from the database
* fix(proxy): step aside for fallbacks the check cannot follow and wildcards that serve a backend name
A fallback target with no deployments of its own, a client routing stamp, and a wildcard that does not match the requested backend name no longer turn a working request into a 400
* fix(proxy): ignore a client model_group that is not a string when building rate-limit headers
A client metadata.model_group sent as an int or a list crashed the error handler with a 500 once the embedding-only check rejected the call before routing. The header helper now treats a non-string stamp as no model group. Drops the pattern clause in the backend-name lookup, since the group lookup already returns wildcard matches. Adds the third audit round's routing-stamp cells.
* fix(proxy): keep a non-string client model_group out of spend rows
* fix(proxy): validate standard fallbacks before the typed fallback-group lookup
* feat(router): opt in to non-chat deployment filtering
* fix(router): skip unresolved local rate-limit fallbacks
* fix(router): ignore caller-provided endpoint selector values
* fix(router): normalize declared chat endpoint prefixes
* test(router): cover preflight error-value behavior
* test(router): derive replay metadata defaults from model info
* fix(router): preserve fallback aliases and check synchronous text calls
* test(routing): rewrite the WebSocket model with a pre-call callback, since main no longer runs guardrails at socket admission
* test(routing): keep the endpoint gate's owned proxies on the worker's database
* fix(router): keep the endpoint gate on model_info patches that omit it and on same-group stream retries
A partial PATCH of model_info reset a stored enforce_supported_endpoints true back to false, because the
field defaulted to False and the update merges model_dump(exclude_none=True). The field now defaults to None
so an omitted flag keeps the stored value
The same-group retry of a stream dropped before content called the provider without the endpoint selection
context, so it could land on an enforced embedding-only deployment of the same group. It now runs inside
endpoint_selection like every other router attempt
* fix(router): probe rate-limit fallbacks through get_routable_deployments so the router surface does not grow
Main's router fence lets code outside the router use only listed Router members and
router_utils modules. The proxy's fallback probe now calls the listed
get_routable_deployments with the request's call type, and the gate's 400 moves to
litellm.exceptions as EndpointNotSupportedByModelError. It stays separate from main's
UnsupportedEndpointError, which the router retries on the group's other deployments,
while the gate refuses the whole group
* test(responses): cover the WebSocket error frame for a model that refuses the endpoint
* fix(proxy): let the rate-limit fallback probe try a fallback this worker has not loaded yet
The probe now skips a fallback only when the endpoint check removed every deployment it has, so a model added on another worker still reaches route_request and its database lookup
* fix(proxy): keep key router settings off the Responses WebSocket, read unrooted or single-string endpoint lists, and add status to the refusal frame
* fix(router): warn when enforce_supported_endpoints has no usable list and match chat endpoints regardless of case, trailing slash, or Gemini path
* test(router): type the load-warning test's caplog and router
* test(routing): match the endpoint gate's integration tests to base fallback behavior on unloaded fallbacks and the Responses WebSocket
---------
Co-authored-by: Mateo <mateo@Mateos-MBP-2.localdomain>
* fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts
* refactor(anthropic): hold the chat completions document change for a later commit
* fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai
* fix(snowflake): turn file parts into document blocks for Cortex Claude
* fix(prompt_security): sanitize chat file parts that carry file_data
* fix: answer 400 or drop PDFs for models that cannot read them
Models whose model_info or cost map entry sets supports_pdf_input to
false, and Ollama, now get the PDF dropped under drop_params and a 400
from litellm without it. GigaChat uploads file parts, SAP keeps sending
them as image_url data URLs, azure_ai passes dict content through, and
annotations other than url_citation move to provider_specific_fields
instead of failing the response.
* fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys
Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and
litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider
annotations to provider_specific_fields, and split them the same way when a stream is rebuilt.
* fix(vertex_ai): send PDF file parts to Vertex partner models as image_url
Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages
reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as
unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL.
* fix(cost-map): leave native DeepSeek PDF support unset
/v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there
and would only change chat completions, which this PR does not otherwise touch.
* fix(main): drop a user turn that held only a PDF under drop_params
Dropping the PDF from a PDF-only user turn left an empty content list, which
Foundry and OpenAI-shaped providers reject with a 400, so drop_params still
failed for that request. The emptied user turn is now removed.
* fix: read OCI PDFs, gate deployment base models, and stop empty drops
OCI generic chat sends file parts as image data URLs again, so OCI Gemini
keeps reading PDFs from /v1/messages and chat file parts.
The PDF gate now reads model_info.base_model, drops PDFs under drop_params
for azure_ai deployments that never declared PDF support (the old flattening
behavior), and answers 400 when dropping the PDF leaves no user message.
The cost map marks the Vertex partner text models that a live probe showed
answering without reading the PDF.
* test(integration): cover the PDF input gate, messages PDF file parts and provider annotations
Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed
* test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells
* fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload
* fix(main): PDF check reads the deployment's top-level base_model
* fix(main): keep roles alternating after drop_params removes a PDF-only turn
When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other
* test: type the PDF and annotation tests the PR added
* test: type the GigaChat, SAP and Snowflake PDF tests
* fix(fallback): read stored router_settings from the database on fallback writes
POST /fallback and DELETE /fallback/{model} read the current rules through the
config cache. When Redis refuses DEL, the eviction before that read fails and
Redis keeps serving the old copy, so each new rule dropped the earlier ones
The write path now reads the router_settings row straight from the writer
database and stores only that row plus the changed key, so config-file router
settings no longer get copied into the database row
* fix(fallback): edit the config file's fallback list when it owns the key
A delete of a rule only the config file holds answers the config-owned 400 again instead of 404, and restating a rule the file already has answers 200 without copying the file's list into the database row. Adds a writer-read regression test and an integration test against a Redis whose ACL refuses DEL
* fix(fallback): re-read the config file's fallbacks ownership on every fallback write
* test(fallback): run the real config reload in the fallback write tests
* test(fallback): key the refused-DEL Redis double by cache key
* fix(fallback): keep the config-owned 400 for an unset env reference and replace a non-object stored row
* fix(fallback): replace a stored router_settings row that holds a non-JSON string
* fix(fallback): compare a restated config-file rule against its resolved env value
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): strip beta-gated body fields when a leg drops the anthropic-beta
When a provider leg drops an anthropic-beta value (the chat handler's allowlist
filter, /v1/messages' filtered headers, Bedrock Converse's additionalModelRequestFields),
also drop the body fields that beta gates (thinking.block_binding, thinking.display=updates,
mid-conversation system clear_at, output_config, tool_addition/tool_removal blocks) so the
request degrades on that leg instead of failing with a provider 400. Copy-on-write so a
fallback leg still sends the full request, and each strip is logged at WARNING.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove a leg that drops a beta sends neither the beta nor clear_at
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): strip beta-gated fields against the final body anthropic_beta list
Converse counts the body anthropic_beta list as active, Invoke strips after
its auto-injected betas are known and injects thinking-binding-controls for
thinking.block_binding, test helpers are fully typed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): strip beta-gated system fields before an older model's system turn becomes a user turn
Models without supports_mid_conversation_system send no mid-conversation beta,
so tool_addition and tool_removal blocks are dropped (and logged) before the
turn is converted instead of riding along as user content. The Invoke
output_config regression test reads the real cost map instead of patching
_bedrock_model_supports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): pin the integration proxy to the bundled beta allowlist and type the helper locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): annotate every new test local with Final and type the fixtures and returns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): strip beta-gated body fields only under drop_params
Without drop_params the request now goes out as sent and the provider
answers its own 400, like any other unsupported param. inline-tools-2026-09-15
also unlocks tool_addition and tool_removal blocks, so a leg that carries it
keeps them.
* fix(anthropic): strip beta-gated fields on Foundry chat and Bedrock chat Invoke under drop_params
Foundry chat and Bedrock chat Invoke now drop beta-gated body fields the same way the other legs do when drop_params is on and the request does not send the gating beta. The mid-conversation system normalizer validates only on the conversion path, and the strip WARNING names the beta the caller can send to keep the field.
* fix(anthropic): send the thinking betas for block_binding and display updates, strip Databricks chat thinking
Every Claude leg now adds the thinking-binding-controls beta when thinking carries block_binding and the display-updates beta for display updates, so those fields go through instead of a 400. Databricks chat rejects both fields with or without the beta, so it drops them under drop_params. A system turn the strip empties stays when it still carries output_config, and the drop warning names both causes.
* fix(anthropic): send the thinking betas on every leg and keep fields where the beta is unknown
Vertex messages and Claude Platform chat now add the thinking betas when thinking carries
block_binding or display updates. Converse sends only the caller's header betas when it has them.
A provider with no row in the beta config forwards beta-gated fields instead of dropping them,
an emptied trailing system turn that would leave an assistant prefill is sent as is, and the
prompt cache baseline body matches the body that goes out. The drop WARNING now names the beta
that unlocks each field
* fix(anthropic): filter Converse body betas, keep Foundry auto betas, read the bundled beta config for row checks
* fix(anthropic): validate messages only when a beta-gated drop runs
A non-object messages entry on a Claude model without mid-conversation
system support raised a raw pydantic ValidationError before the
beta-config and drop_params checks. The strip now passes the messages
through untouched unless it drops something, so the provider's 400
reaches SDK callers as BadRequestError again
* fix(anthropic): keep system blocks with a non-string type instead of raising
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(caching): audit #44542 response cache model group key
* fix(caching): key the response cache by the router's metadata bucket only
* fix(caching): read the router's model group before the caller's metadata
* fix(caching): read caching groups only beside the router's model group
A caller's own metadata.caching_groups on /v1/responses and /v1/messages could
still pick the cache key, so a string group let one model group fill another's
entry. The key now reads caching_groups only from the metadata that holds the
router's model_group.
* test(caching): type the planted caching groups parameter
* fix(caching): read the model group from the metadata that carries the call's deployment
A router call can carry a litellm_metadata it did not write, from the auto router's classifier embedding or an SDK caller, while the router writes its own group into metadata. The cache now picks the metadata source whose model_info matches the call's deployment, then falls back to the previous order.
* fix(caching): keep main's model group lookup for calls outside the router
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): stream a guardrail block on /v1/messages as Anthropic events
* fix(proxy): keep post-call guardrails off the streamed messages block
The streamed block is sent as-is, like the Responses endpoint's block, so post-call guardrails and streaming hooks never scan the proxy's own block message. Tests cover a post-call rail on the stream and the call id on the JSON block
* test(proxy): expect Anthropic events from a streamed messages guardrail block
The legacy guardrail suite pinned the old single bare data: line, which is the
malformed block this branch replaces. The test now checks the six Anthropic
stream events and the block text, so it fails if the bare line comes back
* test(proxy): block messages through a real pre-call guardrail in endpoint tests
The block tests replaced ProxyBaseLLMRequestProcessing.base_process_llm_request on the class to raise the block. They now send a real request through the endpoint with a pre-call guardrail that raises the passthrough block, and check that the x-litellm-call-id header carries the request's call id
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(logging): mark mirrored silent_model calls with is_silent_experiment in spend logs and logging payloads
* fix(router): keep the mirror flag when litellm_metadata is set and strip a client-sent is_silent_experiment
Spend rows read router metadata from litellm_metadata when a call carries it, so the router now sets the flag there too. The proxy drops a client-sent is_silent_experiment with the other router-owned keys, so real traffic cannot be recorded as mirrored.
* fix(logging): mark mirrored calls with a router-only key so clients cannot set is_silent_experiment
Spend rows and the standard logging payload now derive is_silent_experiment from _litellm_silent_experiment, which only the router writes and the proxy and pass-through routes strip from client metadata. A caller's own is_silent_experiment key stays untouched, so it keeps its current effect on mirroring and on provider metadata, and unmirrored calls log false in both places.
* test(langfuse): expect is_silent_experiment false on every logged generation
* fix(logging): mark mirrored calls with a router-only enum value so no client route can set is_silent_experiment
The router stamps SilentExperimentMarker.MIRRORED, which client JSON can never produce, so batches, Responses WebSocket frames and pass-through need no strip. Langfuse layers the flag after its requester-metadata dedupe.
* fix(logging): set is_silent_experiment only on mirrored calls so real calls keep their payloads
Real calls no longer gain an is_silent_experiment key in spend rows or the standard logging payload, so OTEL, S3, GCS and every other logger receive what they did before. The Langfuse regression test now runs the real prompt helper.
* fix(logging): keep the mirror marker out of callback metadata and read the spend-row flag from the logging payload
Logging now strips the router's mirror marker from the request metadata buckets before any callback or vendor sees it, without touching the router's own dicts, so retries keep it. Spend rows read is_silent_experiment from the standard logging payload, which survives the collector's JSON round trip, so collector-mode rows keep the flag too.
* fix(logging): mark mirrored calls through a context variable so router usage counts them once
* fix(router): leave callback-started calls unmarked and stop client flags from turning mirroring off
Logging handlers now run outside the mirrored-call context, so an LLM call a success or failure callback starts while logging a mirrored call is not flagged as mirrored. The guard against mirroring a mirror reads the same context variable instead of request metadata, so a client-sent is_silent_experiment no longer skips the mirrors. The unused module-level get_standard_logging_metadata stops copying resolved keys from raw metadata.
* fix(router): keep main's client flag mirroring skip and the other logging metadata keys
A client's is_silent_experiment: true skips mirroring again as on main, the module-level get_standard_logging_metadata drops only the flag, and the silent experiment tests wait on a logged-call signal instead of polling a deadline
* fix(logging): take the mirror flag in Prometheus labels and Datadog cost tags only from the router
* test: type the new parametrized tests and keep the recorder from changing its argument
* fix(datadog): keep client request tags from setting the silent experiment cost tag
A request tag is_silent_experiment:true set the Datadog cost tag on a direct call.
Only the router flag sets it now. Also covers the sync failure handler's
mirrored-call reset with a failure function callback test.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): let a Claude Code gateway user choose the session team
The gateway's browser verification page now lists the developer's teams, and the device code grant mints the session for the picked team. A team the developer is not on is refused with a 400, and a form without a team keeps the first-team default.
* fix(proxy): refuse a stale gateway team pick and require an explicit choice
* fix(gateway): list the browser-picked team first so an older replica mid-deploy issues it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): keep every startup line JSON when json_logs is on
Boot-time messages the proxy printed straight to stdout (banner, feedback
box, model list, cache and callback notices, policy table, migration
notices, CLI launch notes) and the first records of uvicorn- or
granian-spawned workers came out as plain text while json_logs was on,
breaking log pipelines that parse every line as JSON.
- print_startup_message() prints unchanged in plain mode and, with
json_logs on, emits one INFO record on the proxy logger (ANSI
stripped) that skips the logger and handler level checks like the
print it replaces; show_banner() routes to it in JSON mode only
- the CLI exports JSON_LOGS=true when the config sets json_logs so
spawned workers log JSON from their first record
- the --enforce_prisma_migration_check deprecation notice prints after
the config is read so a config-only json_logs covers it
Resolves LIT-1624
* fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize()
Since #46803 the `lite` shell calls show_banner() against an uninitialized
litellm package, and reading litellm.json_logs inside it initialized the whole
SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR
removed on small VMs). show_banner() is back to its main version; the proxy's
initialize() now emits the JSON banner record itself and calls show_banner()
only in plain mode. The banner unit tests move to the initialize() call site.
* fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger
uvicorn applies --log_config after proxy_cli has switched JSON logging on and before
initialize() switches it on again through load_config(). A logging config that names
"LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where
initialize() emits the banner and the feedback box, so at one worker both went nowhere;
two-worker children import litellm after uvicorn's logging setup and were not affected.
print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON
handler uses, when the proxy logger has no handler of its own.
Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the
banner and the feedback box as plaintext, e2303a3925 prints neither, this head prints both as
one JSON record each. New unit test fails on e2303a3925; new integration cell
test_json_logs_with_operator_log_config_keeps_every_line_json passes at one and two workers.
* fix(proxy): keep a long startup listing whole in its JSON record
print_startup_message ran its record through the proxy logger's filters, so
in JSON mode the required-migration SQL list and the policy list were
middle-truncated at MAX_STRING_LENGTH_STDOUT_LOG (4096 by default) with the
litellm_truncated marker, where the prints they replace showed them whole.
That cap bounds echoed request payloads; a startup message is operator text,
so the record now gets the uncapped processing of a DEBUG line: secret
redaction and base64 collapse still apply, the text cap does not.
Red on 41f92b745a, green here:
- tests/unit/test_logging.py: a listing longer than the cap comes out whole;
a credential in a startup message is still redacted
- tests/integration/configuration/test_json_logs_startup.py: a 60-policy
config boots with json_logs on and the policy listing record (9762 chars)
carries every policy with no truncation marker (4096 chars, marker, on
the unfixed tree)
* fix(proxy): keep the --enforce_prisma_migration_check notice on --health and --test
This PR moved the deprecation notice after the config is read, so that a
json_logs set only in the config applies to it; --health and --test return
before that point and stopped printing it, a visible change main did not
have. The notice is now also emitted before those two returns, through the
same helper, so a deployment running `litellm --health
--enforce_prisma_migration_check` still learns the flag is dead, as on main.
A normal server start emits it once, after the config, as in the previous
commit.
Red on 2d7a2d6844, green here: tests/unit/proxy/test_proxy_cli.py asserts
the notice on --health and on --test.
* fix(proxy): keep the JSON startup records on stdout under an operator handler on the proxy logger
An operator --log_config that gives "LiteLLM Proxy" a handler of its own, a NullHandler
that silences it or a plain WARNING-level handler on stderr, is in force in the window
where the banner, the feedback box and the cache notice are emitted: uvicorn applies it
after the CLI process switched JSON on and before initialize() re-installs the JSON
handler through load_config(). 41f92b745a sent the startup records through the proxy
logger's handlers and fell back to a JSON stdout handler only when there were none, so
under such a config the records landed in the operator's handler, as plaintext on stderr,
or vanished, where the prints they replace reached stdout whatever the logging config
said.
print_startup_message() now writes every startup record through the proxy's own JSON
handler on stdout, filters included, and never through the proxy logger's handlers; the
operator's handler gets nothing from a startup message, as it got nothing from the print.
The normal JSON handler is the same class, formatter and filters, so the records a
pipeline reads are unchanged when no operator config is in play.
Tests: tests/unit/test_logging.py gets the two handler shapes (red on f6f5957d16: the
record went to stderr as plaintext or nowhere); tests/integration/configuration/
test_json_logs_startup.py gets a silencing operator --log_config cell at 1 and 2 workers
(red on f6f5957d16: banner, feedback box and cache notice missing; green here).
* fix(proxy): print the --enforce_prisma_migration_check notice when the config fails to load
f6f5957d16 printed the deprecation notice after the config is read, so a json_logs set
only in the config makes it a JSON record. A boot whose config file is missing or does
not parse then exited before that point and never printed it, where main printed it
before the error (10-20 gauntlet finding).
The notice is printed where main printed it, unless a config file can still switch
json_logs on: then it is held until the config is read and printed right after the
json_logs switch, and a click close callback prints it on the way out if the boot fails
first. functools.cache keeps it to one print either way.
Tests: tests/unit/proxy/test_proxy_cli.py covers a failed config read (red on
75043aed83), a plain config printing it once and a config-only json_logs making it one
JSON record; tests/integration/configuration/test_json_logs_startup.py adds the JSON cell
and a missing and an unparseable config cell (red on 75043aed83, green here), through
refused_boot_log, which now takes extra CLI arguments.
* feat(proxy): add suppress_startup_output to turn off startup prints
* fix(proxy): print startup messages once and without color codes off a terminal
* test(policy-engine): check the policy table keeps colors on a terminal and drops them on a pipe
* fix(proxy): keep startup warnings when suppress_startup_output is on and apply it from CONFIG_FILE_PATH and before other settings
Warnings that name something the operator must fix now print through print_startup_warning, which
skips the suppress gate. The config key is read with the CLI's json_logs handling for --config and
CONFIG_FILE_PATH, applied before the rest of litellm_settings in load_config, and can only turn the
setting on. The Prometheus config panel obeys the setting, and startup prints use the plain log
format's color rule.
* fix(proxy): read CONFIG_FILE_PATH in the CLI only when workers would load it
Workers load CONFIG_FILE_PATH only when it names an existing YAML file, so the CLI's early read of
json_logs and suppress_startup_output skips any other value instead of failing a boot that used to start.
* fix(proxy): read only suppress_startup_output from CONFIG_FILE_PATH in the CLI, without secrets
The CLI now reads CONFIG_FILE_PATH as a plain file with its includes, skips it when a config bucket is set,
and applies only suppress_startup_output, also when --config is passed. json_logs there stays with the
workers as before. Fatal boot errors drop their color codes off a terminal like the startup prints.
* fix(logging): emit startup warnings as WARNING records in JSON mode
* fix(proxy): keep one JSON record per search tool in the startup listing
* fix(proxy): keep booting with an ASCII stdout or an empty litellm_settings
Startup messages now go through click.echo, which writes UTF-8 when stdout is ASCII, as the banner did
before. An empty litellm_settings section reads as no settings in the CLI, as the workers already read it
* test(proxy): boot the CLI tests on the real ProxyConfig config readers
* test(proxy): cover the startup warnings that print with startup output suppressed
* test(proxy): boot the deprecated resolver flag test on the real ProxyConfig
* test(proxy): drop the startup output contract's narrating docstrings
* test(proxy): cover the pgbouncer start failure message off a terminal
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(github_copilot): treat a malformed api-key.json like a missing one
* fix(github_copilot): read the key and the endpoints of api-key.json as separate strict views
* fix(github_copilot): ignore a non-UTF-8 api-key.json like any other malformed file
Read the cached key file as bytes so a UTF-16 file, which Windows
PowerShell 5.1 writes with >, fails JSON validation instead of raising
the decoder's error, and the request gets the refresh or the login hint
* fix(github_copilot): decode UTF-16 key files and refresh over a malformed endpoints field
* fix(github_copilot): decode a UTF-16 access-token file and refuse an undecodable one like a missing one
* fix(github_copilot): use the refreshed key when api-key.json cannot be written
* fix(github_copilot): keep a refreshed key that cannot be saved in memory
* fix(github_copilot): serve a kept key and its endpoint together after the key file is repaired
A key held in memory is now checked before api-key.json, so the token and the endpoint come from the same source, and the map holds one entry per key file with the access token beside it
* test(github_copilot): type the authenticator test fixture and the tests this branch adds
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): use collection endpoint for custom context cache
* fix(vertex_ai): build the cachedContents collection URL under a custom api_base
* fix(vertex_ai): keep a gateway prefix that contains /publishers/
* test(integration): expect the cachedContents collection URL on a custom Gemini api_base
* fix(vertex_ai): cut cache collection URLs at the base's location, graft a mount prefix
* fix(vertex_ai): cache under a publisher model api_base in that model's location
A Vertex AI api_base that names an endpoint or no model keeps its old cache URL, so
no cache is created that the chat call cannot use. The cache body takes its project and
location from the api_base model, matching the collection URL.
* fix(vertex_ai): cache for the api_base's model and skip caching on a Gemini v1 base
A Vertex api_base naming a publisher model now gives the cache its model as well as its project and location, so the cache matches the model generateContent calls. A Gemini api_base ending in /v1 skips context caching, since the v1 API has no cachedContents and rejects the cachedContent field.
* fix(vertex_ai): check the cache token minimum against the api_base's model
The cache is created for the model a publisher-model api_base names, so the minimum prompt size now comes from that model too. A prompt below its minimum is sent uncached instead of failing at cache create.
* fix(vertex_ai): use the stricter cache minimum and send a Gemini base's 404 uncached
---------
Co-authored-by: paulo037 <paulo.h.carneiro.18@gmail.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: stabilize cleanup, lockout, and post-kill timing tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bound the wedged Redis burst p95 by the socket timeout the test sets
The burst p95 under a fixed 0.75s cap tracked how fast the box ran the 30
concurrent writes, so a loaded runner failed it at unchanged code. A handler
that waits on the wedged publish takes at least the coordination Redis socket
timeout, so the test now sets that timeout and asserts the p95 stays under it.
* test: pin the lockout clock and bound the wedged Redis burst by the proxy's 2s wait cap
The lockout tests now set the login throttle's clock instead of reading the real one, so Retry-After is pinned at 77 and a round-down fails on every run
The wedged Redis burst p95 is now bounded by the 2s cap the proxy puts on its own Redis waits, so a burst wait capped under the 5s socket timeout still fails the test
The fresh post-kill clients keep the harness's 2s keep-alive expiry, here and in the two tests #47080 changed
* test: tell ceil from round in the lockout tests and bound the burst by the imported eviction cap
The refused attempt now runs with 76.4s left on the 77s block, where ceil gives 77 but round and int give 76, so
both lockout tests fail if Retry-After stops rounding up.
The burst bound imports _BULK_EVICTION_WAIT_SECONDS instead of copying its 2s value, so lowering the cap moves
the bound with it.
* test: bound the wedged Redis burst by the shorter of the eviction cap and the socket timeout
Any wait the proxy puts on the stopped Redis lasts at least the shorter of the two, so the bound now follows
whichever one a later change lowers.
* test: drive the tool index cleanup test with the manual clock and keep the post-kill block as on main
The post-kill reset is already fixed on main by making kill() wait for the
worker's last thread, so the fresh-client and keep-alive edits go back to
main's version. The tool index cleanup test now advances the injected clock
instead of timing the real one, and the lockout clock helper gets a type hint.
* test: advance the spend log cleanup tests' manual clock through its own method
* test: freeze the throttle clock in the login_utils lockout tests and drive the run budget tests with the manual clock
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop a malformed prompt_cache_breakpoint under drop_params on the direct /v1/responses path
* refactor(responses): reuse the shared prompt_cache_breakpoint_for_wire helper in the chat bridge
* fix(responses): read a non-flag deployment drop_params as unset in the marker walk, like the param mapper
* fix(responses): drop every prompt_cache_breakpoint under drop_params when the model has no breakpoints
Also drop a marker set on the message item itself, sanitize before the
cache_control conversion so a client typo no longer erases that breakpoint,
and skip items that carry no marker so unmarked history costs no extra walk
* fix(responses): keep a valid prompt_cache_breakpoint unless the model is known to reject it
A deployment alias the cost map does not know (an Azure deployment name, a fine-tune, a
litellm_proxy alias) kept its valid markers before this PR and must keep them under
drop_params. base_model is read from litellm_params first, then model_info, as chat
completions does.
* fix(responses): drop a valid marker on known non-breakpoint models and keep the tool schema gate unchanged
* fix(responses): judge breakpoint support by model name so cost-map aliases keep valid markers
* fix(responses): keep the caller's request stripped of cache_control as before when a marker is copied
* fix(responses): keep valid prompt cache markers in place and let cache_control fill a dropped one
Under drop_params the marker step now hands back the caller's own objects when every marker is
already wire-safe, so a request with valid markers is no longer copied and walked a second time.
A deployment that turns cache_control into markers keeps the caller's own markers too, whatever
its base_model says, so one request never gets two answers about breakpoint support.
On /v1/messages a malformed raw marker next to cache_control is dropped before the conversion,
so the block gets the cache hint cache_control alone would give, as on /v1/responses.
* test(responses): keep a null marker and the original key order on a model that reads breakpoints
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): scope the signoz, otel conversation id, and series cap proxy fixtures to their module
Session scope booted one 2-worker proxy per xdist worker and kept it until the
worker's session ended, so a CircleCI node holding two of these files piled up
proxies until the VM stopped responding. Module scope lets the harness group
each file's rig tests onto one worker and tears the proxies down after them.
* docs(tests): keep proxy fixtures module scoped in integration tests
* docs(tests): say how the harness groups proxy fixtures in the integration AGENTS.md rule
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): resume background response streams with stream and starting_after
Retrieving a response with stream=true and starting_after=N now returns the
replayed event stream from OpenAI and Azure instead of one JSON object, on the
SDK (get_responses / aget_responses) and on the proxy's GET /v1/responses/{id}
Providers that cannot replay a stored stream drop the params under drop_params
and raise UnsupportedParamsError without it. Malformed values answer 400 with
the param named. A replay that reaches its terminal event settles the
background response's bill once and passes the provider's usage through
without estimating one. An empty replay (cursor at the end) ends normally
* fix(responses): encrypt replayed stream ids and resume Azure AI Foundry streams
A replayed background stream now gets the same response id encryption as a
created one, so the key that made the response can reuse the id. Azure AI
Foundry deployments resume streams like Azure ones, and the chat-bridged
Responses iterator closes its wrapped stream instead of raising.
* fix(responses): drop malformed resume params under drop_params and keep JSON answers to a streamed retrieve
A bad stream flag drops to a plain retrieve and a bad cursor to a full replay when drop_params is on, and a cursor without stream is ignored even when malformed. A streamed retrieve that the upstream answers with JSON returns the response object instead of an empty stream. get_responses forwards only the include keys from extra_query.
* fix(responses): give proxy stream resumes the deployment stream_timeout and document retrieve query params
* fix(responses): keep deployment stream off retrieves and reject oversized cursors
* fix(responses): keep router default stream off plain retrieves
* test(responses): black box coverage for resume refusals, drop_params, stream timeout and billing
* test(responses): replay a real stream in the billing test and type the fake wrapper chunks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): replay the original Responses item order, phase, and boundaries on the next chat turn
* fix(responses): keep tool-call-only stream content empty and drop unreachable replay guards
* fix(responses): reject an assistant phase OpenAI does not define unless drop_params is set
* fix(responses): keep other providers' reasoning and stream output unchanged and fall back on records that do not fit
* fix(responses): label a structured output turn with its returned message's phase and keep lone surrogates from failing the checksum
* fix(responses): leave a turn without a phase when its content message cannot be read
* test(responses): add black box tests for chat turn history replay
* test(responses): expect the replay record on bridged chat replies
* test(responses): expect the replay record in bridged chat translation cases
* fix(streaming): keep the served service tier on the finish chunk the stream adds
* test(responses): read the expected chat message without JSON validation so the replay matcher stays
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): cover Codex tool lists repeated across turns on the Bedrock chat bridge and Mantle
* fix(responses): keep each Codex tool once when its tool list repeats across turns
* test(responses): type the new tool list dedupe unit tests
* test(proxy): cover management and health routes labelling rejections as auth failures
* fix(proxy): name management and health route errors by their status instead of auth_error
* fix(proxy): keep auth_error for 401 and 403 and answer database outages with 503 in route catch-alls
/key/list and /key/aliases now share the catch-all helper too, so they stop
adding the Authentication Error prefix and typing a 400 as internal_server_error
* fix(proxy): answer 503 when the database drops mid-query and prisma fails to name the error
A real outage surfaces from prisma's query engine as a builtin AttributeError, which the
prisma-module check skipped, so the route catch-alls still answered 500 or 400
* fix(proxy): report wrapped and stalled database outages as 503 and label upstream failures by their own status
* fix(proxy): answer the upstream status on the catch-all, keep auth_error on tag denials, report model insert outages as 503
* fix(proxy): answer 503 for database failovers and label provider auth errors like chat
A database that refuses connections while starting up, shutting down, in
recovery, or read-only after a failover now answers 503 no_db_connection
instead of 400. A provider's 401 or 403 keeps the type /chat/completions
gives it, while a 401 or 403 the proxy raises keeps auth_error. An
exception with no message is named by its type.
* fix(proxy): scope failover 503s to management route errors and keep a re-raised error's cause
* test(proxy): expect the OpenAI error shape from management route refusals
Team-not-found 404s now answer invalid_request_error, and every handler-raised refusal answers param null instead of the string None.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(responses): drop the null created_by OpenAI rejects when output items are replayed
* fix(responses): drop a null created_by from replayed input items too
A conversation saved from an older proxy still carries created_by: null on its compaction, apply_patch and shell items, so OpenAI keeps rejecting the replay. Drop the null from input items on the HTTP request transform and the WebSocket response.create rewrite, keeping a created_by that is not null.
* fix(responses): drop a null created_by from counted input items too
* test(responses): cover saved reasoning nulls and re-compaction of a saved conversation
* fix(responses): drop a set created_by from input under drop_params
* fix(types): type the Responses output serializer's parameters
* fix(responses): drop a set created_by on reasoning items and Perplexity presets under drop_params
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(responses): audit #47445
Integration cells for billing every round of a streamed MCP auto-execute on one spend row, on a native Responses upstream and the chat bridge, across tool rounds, a rejected or broken follow-up, a post-call guardrail block, a client hang-up, the response cache, and the next bridged turn
* test(responses): check a broken follow-up bills its own nonzero usage
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46997
Add request-path cells for the global proxy spend counter: a key with no user spends toward litellm max_budget, and two proxies sharing Redis add every request to one counter that matches the proxy budget row
* test(proxy): wait for the recorded budget spend and shared counter before checking them
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): preserve every grounding web source
* test(gemini): cover all grounding sources per passage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Zabir Raihan <zabirraihan@icloud.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds integration cells for a different key reading the response, message redaction, a rotated, deleted, or team-deleted creator, and a creator whose Langfuse keys resolve to nothing
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(router): add deployment rpd and tpd daily limits to enforce_model_rate_limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): route rate limit cache increments through typed helpers
* chore(ui): regenerate schema.d.ts for deployment rpd and tpd fields
* test(types): add rpd and tpd to the owned litellm_params inventory
* fix(router): run daily rpd/tpd checks after the minute checks and increment rpm and rpd in one cache trip
* fix(router): flatten the async rpm reject branch for the ruff strict gate
* fix(router): hand rpm and rpd back on any rate limit reject
A request the rpm, rpd or tpd check rejects no longer keeps its request counts, so retries against a deployment whose rpd is spent cannot fill its rpm and block the in-group fallover with the minute 429.
* test(router): a minute-rejected request no longer keeps its rpm count
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(routing): audit #46601
Adds integration cells for the falsy budget 400, the empty and none-only declarations under drop_params, Vertex AI Claude, Databricks Claude, and the Bedrock native output retry
* test(routing): expect the Anthropic default max_tokens on Bedrock Invoke chat
Reads the default from litellm instead of pinning 4096, which #48502 raised to 128000
* test(routing): assert whole sent bodies in the Vertex AI and Bedrock native output cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(daily-activity): read guardrail spend per key from the daily guardrail usage table
Adds DailyActivityRepository.guardrail_spend, which sums LiteLLM_DailyGuardrailUsageUnits.cost by date, guardrail and key. A guardrail row counts only when its key has a spend row on that day under the same scope filters as the rest of the daily activity response, so user, key and date filters apply to guardrail cost too
* feat(daily-activity): add a per-guardrail spend breakdown to /user/daily/activity/aggregated
Each day now has breakdown.guardrails[<guardrail_id>] with metrics.spend and an api_key_breakdown carrying each key's spend and metadata. Guardrail entries only report spend, since that is the one metric stored per key. The breakdown is on the user endpoint only, is empty when a model filter is set because guardrail cost is not stored per model, and lists only keys within api_key_limit while the guardrail total keeps every key. spend and budgets are unchanged
Resolves LIT-9785
* feat(daily-activity): add the guardrail breakdown to team and organization activity, with guardrail names
/team/daily/activity/aggregated and /organization/daily/activity/aggregated now return breakdown.guardrails the same way the user endpoint does, since each key belongs to one team and one organization. Each guardrail entry also carries metadata.guardrail_name, read from LiteLLM_GuardrailsTable for guardrails created through the UI or API and falling back to the breakdown key for config.yaml guardrails, whose key is already their name
* fix(daily-activity): keep daily activity up when the guardrail breakdown query fails, and treat an empty model filter as none
A database error from the guardrail spend query now logs a warning and returns the response without guardrails, where it used to fail the whole aggregated response with a 500. An empty model= no longer drops the guardrail breakdown, matching how the rest of the endpoint already ignores it. The daily activity golden files and fixtures gain the empty guardrails key every breakdown now carries
* refactor(daily-activity): fold guardrail breakdown in a worker thread and drop docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(daily-activity): report untracked guardrail units next to spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
