* test(nvidia_riva): audit #41021
Adds an owned Riva ASR gRPC server and provider integration cells for the riva_offline request-body ban, the offline and streaming paths, the per-deployment caller opt-in, and the 413 and 408 error mappings
* test(nvidia_riva): check word timing requests and audio samples in the Riva double
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(router): cover silent_model shadow fan-out and streaming end to end
Proxy cells check that each silent_model target gets one request in the
caller's stream mode, that the caller sees only the primary, and that every
shadow lands its own spend row under its own model group. SDK cells check the
sync Router.completion thread path streams each shadow and fires its success
callback.
* test(router): flatten the spend row lookup with itertools.chain
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47358
Add integration cells for pass-through bodies and settings-endpoint mappings across workers
* test(proxy): wait for every worker to hold the saved mapping before probing
* test(proxy): compare passthrough bodies to the client body and probe each worker on its own connection
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Pin the queue route contracts #47498 changed: the priority header on streamed answers, the 400 for a request with no priority to queue at, a provider bad request keeping its own status, and a deployment-id request served ahead of the queue
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Covers the json_schema envelope edge cases #47478 fixed on the Anthropic json tool path: a key listed only in
`required`, a tool-named wrapper on a stream, non-ASCII and lone-surrogate answers, a stream cut before the tool
block closes (proxy and sync SDK, Anthropic and Bedrock Invoke), and a string `output_format` with and without
`drop_params` on Anthropic and Vertex chat and /v1/messages. The scripted Anthropic peer now refuses a non-object
`output_format` the way Anthropic does
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(anthropic, bedrock): audit #47489 thinking replay across Claude routes and Bedrock Converse
* test(bedrock): compare the whole replayed Converse turn in the merged-message and cache point cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(files): audit #40621
Pin the raw cloud storage file id gate on /v1/files content and delete against a running proxy: twice-encoded gs:// ids stay blocked for non-admin keys, proxy admin keys read s3:// objects, and ordinary provider ids keep working
* test(files): check the full SigV4 signature on the admin raw s3 read
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #46815
Add integration tests for every count route #46815 changed, and count a Claude
subscription caller's tokens with their forwarded token when the deployment has
no configured key, which #46815 turned into a 401
* test(proxy): accept the forwarded subscription token in the count endpoint mocks
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A DeepSeek V4 stream that ends without a usage block now has a spend-row cell proving its prompt_tokens are counted with DeepSeek's tokenizer, the one path #47353's own cells did not reach
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit #47418
* test(proxy): hold the stream open past its first chunk and require cost headers in the audit cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds an integration cell proving a failed /v1/responses request counts its input from the newest compaction item on its failure spend row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover the max_parallel_requests slot a streamed /v1/messages MCP auto-execute keeps
A streamed /v1/messages request that auto-executes MCP tools replays its answer
as a stream after its model rounds already logged success. The stream path frees
the request's max_parallel_requests slot only on a client disconnect, so after
message_stop the slot stays taken and the key's next request answers 429
The test streams such a replay to the end and expects the slot to be free. It
fails until the fix lands
* fix(proxy): free the max_parallel_requests slot when a replayed /v1/messages MCP stream completes
A streamed /v1/messages request that auto-executes MCP tools runs its model
rounds without streaming and replays the final answer as a stream. The first
round claims the request's only success log, and its limiter callback keeps the
slot because the tool call reused it afterwards. The stream path released the
slot only on a client disconnect, so after message_stop the key stayed at its
max_parallel_requests limit and its next request answered 429
Stream cleanup now releases the stashed slot when the stream completed and its
logging object already claimed the success log, which only happens when the
stream replays a response that was logged before it started. A real stream
still releases through its own success callback
* test(mcp): free the key's only parallel slot after a streamed auto-execute ends
Run a streamed MCP auto-execute on a key with max_parallel_requests 1 on
every LLM surface, then send the next request on that key and expect 200.
On main both /v1/messages surfaces answer 429 here, while /v1/responses and
/v1/chat/completions already pass
* test(proxy): keep only the MCP stream regression tests, since #48628 already frees the slot at stream end
* test(proxy): require the MCP stream regression test's response to stream and reach its end event
* test(gemini): audit #47542
Proxy integration cells for the Gemini and Vertex default output budget: the env budget reaches Google on chat and responses, turns on long decode, yields to the caller's own limit, caps per deployment, skips resumes, and a camelCase two-candidate request is never continued
* test(gemini): annotate the budget wire module's pytestmark as Final
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Checks that a SageMaker stream failing after its first chunks reaches /v1/chat/completions, /v1/messages and /v1/responses callers as an error with no success end event, and that each writes a failed spend log row
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Check that cancelling a LocalSandbox run stops the grandchild process it started, using a real child process
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(langfuse_otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): name trace metadata identity keys after their request metadata fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): type metadata helper and drop null fields so they cannot clobber caller trace metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): annotate new request metadata tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): build rig config and wire batches immutably
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): skip empty trace identity values so teamless requests cannot erase attribution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): drain wire batches per poll instead of accumulating
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): cover null stripping and empty trace identities
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): set redaction flag directly instead of monkeypatching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): add integration audit matrix for langfuse metadata keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): tighten audit chaos and unauthenticated cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): restore contracts.json formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): close SDK clients, pin worker identity, and normalize response id matching
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): gate the slow sink reply instead of sleeping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): emit request metadata and identity under langfuse.observation.metadata and langfuse.trace.metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel v2): satisfy type discipline gate for metadata export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop prohibited covers markers from langfuse otel integration tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): put the mutable-ok suppression on the dict construction line the type-discipline gate reports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse otel): share request metadata export across callbacks
* fix(langfuse otel): drop prompt variables, MCP tool I/O and vector store queries from exported request metadata
* fix(langfuse_otel): keep span export alive when nested request metadata has non-string keys
The content strip validated prompt_management_metadata, mcp_tool_call_metadata
and vector_store_request_metadata items as str-keyed mappings, so an SDK
caller passing one with a non-string key raised inside
exported_request_metadata: OTEL v2 dropped the LLM span for every vendor and
v1 langfuse_otel lost the request metadata and every attribute after it.
Validate those nested mappings with object keys instead. Also hoists the
langfuse_otel imports.
* fix(langfuse_otel): send the request metadata blob to Langfuse destinations only
- A requester_metadata that is not a mapping no longer raises. The v2 LLM span
(every vendor) and the v1 Langfuse attributes survive it again.
- Prompt-quoting routing_decision fields stay out of langfuse.observation.metadata.
- An exporter or key/team destination for any other backend drops
langfuse.observation.metadata and langfuse.trace.metadata.user_api_key_*, as on
main. This covers a collector folded into the Langfuse logger.
- The v2 Langfuse key-set pin in the arize family test gains the two new keys.
* fix(langfuse_otel): export only caller identity under the Langfuse metadata keys
- The Langfuse metadata keys carried the whole request metadata. On OTel v2 that
sent request headers, caller metadata, user email, user agent and auth metadata
to a Langfuse destination that never received them on main; on v1 it doubled
each span's metadata payload, so big caller metadata doubled the export size
- Both paths now export only the key alias, user, end user, team, team alias and
org. On v2 that is the team plus the identity among the operator's
`baggage_metadata_keys`, which v2 spans already carry as `litellm.metadata.*`.
Everything else stays in the bare `metadata` attribute, as on main
- `output_messages` returns a list again, and the arize v2 b3 key pin is back to
main's set
* fix(langfuse_otel): drop the caller identity first when a span hits its attribute limit
v1 wrote the identity attributes after the OpenInference ones, so a span near the
128-attribute limit (about 33+ tools) evicted openinference.span.kind, metadata
and llm.model_name instead. The identity now goes first on a fresh span, so the
SDK evicts it before anything else, and a caller's active span that already
carries attributes does not get it.
v2 sheds the Langfuse identity keys before any indexed message attribute when a
span is over budget, so a span keeps every attribute it carried without them.
* fix(langfuse_otel): keep other spans and dropped counts as main exports them
- v1 writes the caller identity only into the room the span's attribute
limit leaves next to the error attributes a failed call adds, so the
exported dropped-attribute count matches main
- v2 strips the Langfuse request metadata only from LiteLLM's own spans;
application, instrumentation and server spans keep every key they set
- a stripped v2 span keeps the dropped attribute, event and link counts
- a config that lists the Langfuse mapper without the langfuse_otel
exporter also keeps the request metadata away from other exporters
* fix(langfuse_otel): keep the caller identity off the caller's active span
v1 writes LiteLLM's attributes onto the caller's active span when one is
current instead of starting its own span. That span goes to the caller's
own exporter, so the identity keys reached a non-Langfuse collector and
could take attribute slots the caller fills later. The identity keys now
go only on the span LiteLLM starts, whatever the caller's span carries.
* fix(langfuse otel): keep the identity keys on the OTEL v2 Langfuse mapper only
The v1 langfuse_otel callback goes back to main: it sends no
langfuse.observation.metadata or langfuse.trace.metadata.user_api_key_*
keys. The OTEL v2 Langfuse mapper keeps them. The integration cells now
run every endpoint, client, redaction, failure and chaos case on v2.
* test(langfuse otel): count only accepted batches in the sink outage cell
The wire also records the batches the sink answered 503, so an exporter
retry made a span look exported twice. The chaos cells now also check the
spans that landed carry the caller identity.
* test(langfuse otel): split stacked span comprehensions and drop constant comments
* test(langfuse otel): split the metadata integration module into focused files
* fix(langfuse otel): send the org id in trace metadata too
The trace metadata keys now reuse IDENTITY_METADATA_FIELDS, so a key in an org gets langfuse.trace.metadata.user_api_key_org_id next to the generation's org id.
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tavily): return raw_content, score, favicon and the response envelope from Tavily search
Tavily search dropped every field except title, url and content, so
include_raw_content, include_answer, include_images and include_favicon
reached Tavily but their output never reached the caller. Each result now
carries raw_content, score and favicon when Tavily returned them, and the
top-level fields (answer, images, request_id, usage, ...) come back as
provider_metadata, same as the Exa fix in #47423
* test(tavily): move the Tavily search wire test's sample data to module constants, like the Exa test
* feat(passthrough): let upstreams report the model name via x-litellm-upstream-model
A pass-through upstream that prices its own requests already reports cost and
tokens in x-litellm-response-cost / x-litellm-total-tokens. Product endpoints
that take no `model` in the body still landed under `unknown`, so their spend
could not be broken down.
When the upstream sends x-litellm-upstream-model, record its value as the
request's model on success and failure rows, overriding any body `model`. The
name is a label only: it never marks the request as upstream-priced and is not
used to look up a price. A blank or missing header changes nothing.
Refs LIT-9910
* fix(passthrough): keep the upstream-reported model name bounded
The upstream model name becomes a spend row, a daily rollup key and a metric
label. Use only the first element of a repeated header (RFC 9110 field
combining, which is how httpx joins it) and cut the name to
MAX_SPEND_LOG_MODEL_NAME_LENGTH, the limit the spend log already applies to
model names.
Refs LIT-9910
* test(passthrough): cover x-litellm-upstream-model through spend rows and rollups
Drives a config-defined pass-through against a scripted upstream and reads
back the spend row, the daily team rollup and /team/daily/activity. Covers the
header winning over a body model on buffered and streamed responses, failure
rows, a model-only header keeping cost_per_request, the no-header and blank
header cases, and repeated or long header values.
Refs LIT-9910
* test(passthrough): annotate new test locals with Final
Refs LIT-9910
* test(integration): update stale expectations to intended provider behavior
Each expectation moved because a merged PR changed behavior on purpose.
- chatgpt string input is wrapped as one user message (#41522)
- Gemini 2.5 tool media sits beside function_response (#48428)
- sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645)
- a body base_model is stripped without allow_client_pricing_override (#47639)
- ai_decide answers without usage return usage null (#48261)
- azure_ai forwards mapped client anthropic-beta values (#48033)
- the OpenCode scripted upstream mints a fresh id per request
* test(e2e-ui): type the guardrail name before deleting it
GuardrailDeleteModal requires the exact name before Delete enables since
#45893. All three guardrail delete flows now go through one helper.
* test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body
OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces,
so the gateway's traceresponse now ends in -03. The producer copies the server
span's own trace flags, so the tests now accept any two-digit flags with the
sampled bit set instead of pinning -01
#48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an
object" on purpose, so the [1] body case is renamed non_object_body and
expects that 400
* test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test
The outage test waited for "Exception while exporting Span batch" in the proxy
log, which is OpenTelemetry's wording and no longer appears with 1.45. The test
keeps asserting what the gateway guarantees, callers unaffected during the
outage, a live gateway, and every span after recovery delivered once with no
outage span duplicated
* test(guardrails): count the Responses disconnect spend rows #47611 added
Since #47611 a streamed /v1/responses disconnect writes its own spend row, so
the concurrent disconnect test saw 20 rows where it counted 10 by model group.
It now reads each row's call type and expects ten acompletion rows with the
outage verdict and ten aresponses rows with success. /v1/messages disconnects
still write no row on main, so they are left out of the count rather than
pinned as absent
The responses-httpx case of the single disconnect test no longer needs its
LIT-8603 skip and passes
* test(observability): size K4's timeout from the harness boot budget
K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to
58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of
30 runs hit pytest-timeout. The cap and the restart gate wait now come from
readiness_seconds and graceful_stop_seconds, like other multi-boot tests
* fix(guardrails): redact secrets in tool traffic and Anthropic system
* fix(guardrails): bound tool payload scans and stop nested tool_result recursion
Scan each tool value beside its key instead of an indented dump, read
tool_result content one level deep, keep non-finite tool arguments as
raw text, and give read-only guardrails one fragment per tool payload
* test(guardrails): cover read-only guardrail consumers of tool and system text
* fix(guardrails): keep tool argument parsing, scanning and tests within bounds
Treat tool arguments that decode to a lone surrogate as raw text, decode
JSON-escaped detections back to the leaf value, quote unkeyed leaves
without repeating a parent key per list item, drop cycle tracking that
parsed JSON cannot reach, and use the CI license instead of a fake
license server
* fix(guardrails): scan list items beside a capped parent key
List items keep their parent key in the scanned JSON line, capped to its
last 64 characters, so a credential key over a list is still detected
while a long key over many items no longer multiplies the scan text
* fix(guardrails): cap the scanned key after JSON escaping
A key of control characters escaped to six times its length after the
cap, so every list item's scan line grew with it
* fix(guardrails): collapse whitespace in the scanned key context
Space-heavy keys made every inherited list item scan several times slower
than a short key
* fix(guardrails): build each key's scan context once per payload
Repeating the collapse, escape and cut for every list item made a long key
over many items cost key length times item count
* fix(guardrails): cover Responses tool call items and instructions
Responses carries the same tool call arguments and system prompt as chat
and messages under function_call.arguments, custom_tool_call.input and
instructions, so secrets there reached the provider unscanned
* perf(guardrails): skip rescanning text hide-secrets already found clean
Agents resend their whole conversation on every turn, so each turn
rescanned every earlier tool result. Each guardrail now keeps a bounded
set of digests of scan inputs that held no secret and skips those; text
with a secret is always rescanned and no secret material is kept
* fix(guardrails): size the clean-scan cache for many sessions and replace longest secrets first
4096 digests held about 20 long agent sessions per worker, so interleaved
sessions evicted each other and every resend rescanned its history;
65536 digests (about 10 MB) hold a few hundred. Tool payload secrets are
now replaced longest first after JSON unescaping, so a shorter overlapping
match cannot leave part of a secret behind. The key-context test checks
the work it bounds instead of wall-clock time, which CI could not meet
* fix(guardrails): keep unmasked tool argument text and redact keyword matches cut inside an escape
* fix(guardrails): write a tool visitor's whole payload when it changes more than string values
* fix(guardrails): read message text before tool and system text so a vendor error on the new text cannot skip it
* fix(guardrails): moderate tool and system text in its own call so a long payload cannot hide a flagged message
* fix(guardrails): skip the empty moderation call when only tool or system text is present
* fix(guardrails): scan long multi-line tool strings as their own lines so the scan stays linear
* fix(guardrails): keep the key beside every line of a long multi-line tool string
* fix(guardrails): confine tool-traffic coverage to hide-secrets
Restore iter_message_text to its main behavior, so banned keywords, Azure
Content Safety, IBM detectors and the OpenAI and Google moderation hooks
read exactly what they read on main. Only walk_user_text, which
hide-secrets alone calls, still reaches system, instructions and tool
payloads.
* test(guardrails): stop the tool-traffic integration tests from flaking on a busy host
Use the shared per-case timeout for the rig client, and open a new
connection per call in the worker-kill burst so a pooled keep-alive
socket to the killed worker is never reused.
* test(guardrails): drop the wall-clock check from the deep tool arguments unit test
The integration test test_deeply_nested_tool_arguments_do_not_stall_the_worker
already covers the timing; the unit test keeps the redaction check.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): abort the request with a 499 when the client disconnects during the body upload
Co-authored-by: Sinfony8838 <Sinfony8838@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): half-uploaded body on chat and vector store routes never reaches the provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): close the auth-time OTel server span before aborting a half-uploaded body
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Sinfony8838 <Sinfony8838@users.noreply.github.com>
* fix(router): keep a lone deployment warm under router allowed_fails unless the policy names the exception
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): give cooldown fixtures a sibling deployment so allowed_fails=0 still forces a cooldown
* fix(router): keep benching a lone deployment that has a fallback group or an empty deployment policy
* fix(router): accept dictionary fallback targets in the lone deployment check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep every startup line JSON when json_logs is on
Boot-time messages the proxy printed straight to stdout (banner, feedback
box, model list, cache and callback notices, policy table, migration
notices, CLI launch notes) and the first records of uvicorn- or
granian-spawned workers came out as plain text while json_logs was on,
breaking log pipelines that parse every line as JSON.
- print_startup_message() prints unchanged in plain mode and, with
json_logs on, emits one INFO record on the proxy logger (ANSI
stripped) that skips the logger and handler level checks like the
print it replaces; show_banner() routes to it in JSON mode only
- the CLI exports JSON_LOGS=true when the config sets json_logs so
spawned workers log JSON from their first record
- the --enforce_prisma_migration_check deprecation notice prints after
the config is read so a config-only json_logs covers it
Resolves LIT-1624
* fix(proxy): keep show_banner() SDK-free, pick the JSON banner in initialize()
Since #46803 the `lite` shell calls show_banner() against an uninitialized
litellm package, and reading litellm.json_logs inside it initialized the whole
SDK at the banner (0.14 s to 0.83 s for a bare `lite` here, the seconds that PR
removed on small VMs). show_banner() is back to its main version; the proxy's
initialize() now emits the JSON banner record itself and calls show_banner()
only in plain mode. The banner unit tests move to the initialize() call site.
* fix(proxy): keep the JSON banner when an operator --log_config names the proxy logger
uvicorn applies --log_config after proxy_cli has switched JSON logging on and before
initialize() switches it on again through load_config(). A logging config that names
"LiteLLM Proxy" removes that logger's handlers in the window between the two, which is where
initialize() emits the banner and the feedback box, so at one worker both went nowhere;
two-worker children import litellm after uvicorn's logging setup and were not affected.
print_startup_message() now falls back to a one-off JSON handler, on the stream the JSON
handler uses, when the proxy logger has no handler of its own.
Live, json_logs on plus a log_config naming the proxy logger, one worker: main prints the
banner and the feedback box as plaintext, e2303a3925 prints neither, this head prints both as
one JSON record each. New unit test fails on e2303a3925; new integration cell
test_json_logs_with_operator_log_config_keeps_every_line_json passes at one and two workers.
* fix(proxy): keep a long startup listing whole in its JSON record
print_startup_message ran its record through the proxy logger's filters, so
in JSON mode the required-migration SQL list and the policy list were
middle-truncated at MAX_STRING_LENGTH_STDOUT_LOG (4096 by default) with the
litellm_truncated marker, where the prints they replace showed them whole.
That cap bounds echoed request payloads; a startup message is operator text,
so the record now gets the uncapped processing of a DEBUG line: secret
redaction and base64 collapse still apply, the text cap does not.
Red on 41f92b745a, green here:
- tests/unit/test_logging.py: a listing longer than the cap comes out whole;
a credential in a startup message is still redacted
- tests/integration/configuration/test_json_logs_startup.py: a 60-policy
config boots with json_logs on and the policy listing record (9762 chars)
carries every policy with no truncation marker (4096 chars, marker, on
the unfixed tree)
* fix(proxy): keep the --enforce_prisma_migration_check notice on --health and --test
This PR moved the deprecation notice after the config is read, so that a
json_logs set only in the config applies to it; --health and --test return
before that point and stopped printing it, a visible change main did not
have. The notice is now also emitted before those two returns, through the
same helper, so a deployment running `litellm --health
--enforce_prisma_migration_check` still learns the flag is dead, as on main.
A normal server start emits it once, after the config, as in the previous
commit.
Red on 2d7a2d6844, green here: tests/unit/proxy/test_proxy_cli.py asserts
the notice on --health and on --test.
* fix(proxy): keep the JSON startup records on stdout under an operator handler on the proxy logger
An operator --log_config that gives "LiteLLM Proxy" a handler of its own, a NullHandler
that silences it or a plain WARNING-level handler on stderr, is in force in the window
where the banner, the feedback box and the cache notice are emitted: uvicorn applies it
after the CLI process switched JSON on and before initialize() re-installs the JSON
handler through load_config(). 41f92b745a sent the startup records through the proxy
logger's handlers and fell back to a JSON stdout handler only when there were none, so
under such a config the records landed in the operator's handler, as plaintext on stderr,
or vanished, where the prints they replace reached stdout whatever the logging config
said.
print_startup_message() now writes every startup record through the proxy's own JSON
handler on stdout, filters included, and never through the proxy logger's handlers; the
operator's handler gets nothing from a startup message, as it got nothing from the print.
The normal JSON handler is the same class, formatter and filters, so the records a
pipeline reads are unchanged when no operator config is in play.
Tests: tests/unit/test_logging.py gets the two handler shapes (red on f6f5957d16: the
record went to stderr as plaintext or nowhere); tests/integration/configuration/
test_json_logs_startup.py gets a silencing operator --log_config cell at 1 and 2 workers
(red on f6f5957d16: banner, feedback box and cache notice missing; green here).
* fix(proxy): print the --enforce_prisma_migration_check notice when the config fails to load
f6f5957d16 printed the deprecation notice after the config is read, so a json_logs set
only in the config makes it a JSON record. A boot whose config file is missing or does
not parse then exited before that point and never printed it, where main printed it
before the error (10-20 gauntlet finding).
The notice is printed where main printed it, unless a config file can still switch
json_logs on: then it is held until the config is read and printed right after the
json_logs switch, and a click close callback prints it on the way out if the boot fails
first. functools.cache keeps it to one print either way.
Tests: tests/unit/proxy/test_proxy_cli.py covers a failed config read (red on
75043aed83), a plain config printing it once and a config-only json_logs making it one
JSON record; tests/integration/configuration/test_json_logs_startup.py adds the JSON cell
and a missing and an unparseable config cell (red on 75043aed83, green here), through
refused_boot_log, which now takes extra CLI arguments.
* fix(proxy): keep startup records as printed instead of running secret redaction
In JSON mode the startup messages ran secret redaction, which the prints they replace never
had. Its sk- key pattern has no left boundary, so a model named risk-assessment-gpt4o was
logged as riREDACTED in the model list (10-21 gauntlet finding). The records now skip
redaction like the prints did; the processed record still counts as scrubbed, so the
handler's secret filter and the formatter leave it alone. The stdout cap stays off and base64
collapse stays on, as before.
Tests: tests/unit/test_logging.py checks the model name comes out whole (red on 7d17f126e5),
replacing the test that asserted startup records were redacted;
tests/integration/configuration/test_json_logs_startup.py adds a boot with a
risk-assessment-<id> model whose model list record carries the name (red on 7d17f126e5).
* refactor(proxy): meet the AGENTS.md rules Greptile flagged on the startup-message change
The policy listing builds its rows as tuples in two helpers instead of appending to a list;
plain output is unchanged (same bytes as before across 3000 random policy and attachment
configs). The explanatory comments added with this change are gone, and the docstring of
print_startup_message now says the record is whole and unredacted.
Tests: the migration-diff JSON test no longer runs the fake prisma CLI; it checks the
skipped-diff notice in-process, and the fixture extension it needed is reverted. The
json_logs export and deprecation-notice CLI tests run the real proxy module and a real
config file instead of a mocked ProxyConfig and proxy module, and the --health/--test
notice tests no longer patch our helpers. Each still fails when the export, the hold or
call_on_close is removed. New test parameters are typed.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(complexity_router): Codex compaction requests hold the user_turn decision
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): treat Codex compaction traffic as user_turn continuation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): validate Codex turn metadata with frozen models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(complexity_router): flatten body system text through _message_text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(complexity_router): drop unrelated reformatting from the mapped test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(complexity_router): skip pin escalation on Codex compaction turns
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
An OpenAI or Azure SDK client litellm built and evicted from the client
cache is usually garbage collected inside the closer's 900 second grace
window, so the closer's weak reference dies and the client's aiohttp
session is never closed. LiteLLMAiohttpTransport.__del__ now hands an
owned, still-open ClientSession to _close_recycled_session, which already
covers the current-loop, foreign-loop and closed-loop cases.
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: BangerSoul <bangersoul@users.noreply.github.com>
* feat(proxy): refuse migrations that existing rows would make fail, before the first one runs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(migrations): start the serving-proxy preflight case from a database without the api_key index
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type the preflight count rows without casts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): ruff format the db preflight flag
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy-extras): keep the extras files at their existing formatting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy-extras): type the ledger query and the reader pid so the type gate holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the preflight report type where the extras package may be absent
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): suppress the optional-extras unknown on the preflight report variables
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): suppress the optional-extras unknown on the preflight import
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): return a typed preflight report when the extras package is unknown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): sort the preflight import block
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the preflight report binding Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): size the whole btree entry in the row check and name the flag for what it does
The index check now lays out each row's entry the way index_form_tuple does (tuple header, null bitmap, every indexed column at its alignment, rounded to 8) and compares it with BTMaxItemSize for the server's block_size, so a key that passes octet_length > 2704 but still fails the (api_key, startTime) index is caught. --db_preflight becomes --check_migrations_against_existing_data, db_preflight.py becomes migration_data_checks.py and the timeout env var LITELLM_MIGRATION_DATA_CHECK_TIMEOUT
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): price native guardrailConfig trace usage into spend and x-litellm-response-cost
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bill native Bedrock guardrail trace usage on chat, responses and pass-through
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(streaming): record the Bedrock guardrail trace of a streamed metadata chunk on the logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): apply guardrailConfig on /v1/messages and price its trace, record the stream trace on both builder paths, redact the native guardrail_response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): validate the guardrail trace with one pydantic model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): record the native guardrail entry when litellm_params metadata is None
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): import the renamed messages invoke transformation module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): skip native guardrail billing on cache hits and redact the trace with message logging off
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): walk the guardrail trace iteratively when redacting it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): record the native guardrail trace where the Bedrock response is parsed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): type the native guardrail trace fixtures and locals
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): drop the experimental label from the MCP server and client paths
Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims
pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): derive trace-level name, input, output and tags on the generation span
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): keep every reasoning summary in the derived observation output
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): leave joined traces untouched by derived trace fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): restore preset cache key fallback and type the derived-field helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): keep trace tags off joined traces entirely
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): tolerate out-of-band upstream requests and add sink outage chaos cell
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): build the langfuse rig config without mutating the loaded yaml
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(langfuse_otel): flatten output items with itertools.chain to satisfy LIT014 budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langfuse_otel): type the _RecordingSpan and _emitted unit helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(langfuse_otel): keep joined-trace tags, mask trace io, leave v2 trace io to the root
Joined traces (existing_trace_id) keep the caller tags they carried before,
and only the derived trace fields are skipped for them. mask_input and
mask_output now redact the derived trace input and output, matching the
vanilla Langfuse callback. The v2 mapper derives the trace name only,
because the root observation already carries the trace input and output
there. Uses the public preset cache key helper, moves imports to module
top, restores the original formatting of the unit test file, and adds
integration cells for the v2 llm_only scope and SDK sync/async calls.
* fix(langfuse_otel): split string caller tags the way Langfuse reads them
A caller tags value given as a comma list or a JSON array string (body
metadata.tags or the langfuse_tags header) used to reach Langfuse raw,
where ingestion split it into separate tags. Merging it with the derived
tags kept it as one tag, so parse it the same way before the merge
* test(otel): expect langfuse.trace.name on the v2 langfuse_otel generation span
* fix(langfuse_otel): read string mask flags the way the classic callback does
The langfuse_mask_input / langfuse_mask_output headers arrive as strings,
so "false" redacted the derived trace io. Parse them with str_to_bool and
fall back to truthiness, as the classic langfuse callback does
* fix(langfuse_otel): derive trace fields only on traces litellm starts
A litellm span that joins a trace an app or caller started (inbound
traceparent, the app's active span, a caller-provided parent span) no
longer gets a derived trace name, input, output or tags, like a request
joining an existing_trace_id. Langfuse applies the later-starting span's
trace update, so the derived values overwrote the app's own trace name
and io.
OTel v2 derives the name on the request root only when it has no parent,
and the generation span keeps main's caller-only trace name.
* test(langfuse_otel): keep main's OTel v2 generation key set in the arize family cell
The OTel v2 generation span no longer carries a derived langfuse.trace.name,
so the pinned key set goes back to main's.
* fix(langfuse_otel): name the OTel v2 request root once, from the request's own call
The v2 Langfuse logger re-derived langfuse.trace.name on the request root
for every LLM call in the request. A post_call llm_as_a_judge guardrail's
judge call replaced the caller's trace_name on the root with
litellm-acompletion, and with another v2 callback (otel, arize_phoenix)
listed before langfuse_otel that vendor's root span gained
langfuse.trace.name on every request.
The derived name now fills only an unnamed root this logger's provider
started, and never comes from a call litellm makes for its own features.
A caller trace_name still names the root on every call, as on main.
* fix(langfuse_otel): keep the derived trace fields from pushing other attributes off a full span
The derived trace name, tags, input and output were written after the OpenInference attributes with no room
check, so a span near the SDK's attribute limit (about 34 tools at the default 128) evicted keys main
exported, such as openinference.span.kind, metadata and llm.model_name, and raised the dropped count.
They are now written last and only into the room the limit leaves next to the error attributes a failed
call adds, so a full span exports what it would without them.
* test(langfuse_otel): give the full-span cells' derived request a caller trace id, as the joined one's traceparent does
* chore(langfuse_otel): drop the docstrings and comments the house rule disallows and type the new tests' parameters
* test(langfuse_otel): set the span attribute limit the full-span tests run at and keep Final locals out of the app-trace loop
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages
Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): keep Final variables out of loops in the rename shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): match the warning text in the rename shim test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): mark the alias loader spec restore for the type discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): format the renamed package shim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(anthropic): keep the rebind-ok marker on the spec restore line
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(anthropic): carry the new package search path on alias specs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(anthropic): declare the shim test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): count provider-side 401s in LLM failure and deployment metrics
The invalid virtual key skip no longer applies to an error the LLM provider
returned while the deployment was called with its own credential, so a 401
for a bad deployment credential increments litellm_llm_api_failed_requests_metric,
litellm_deployment_failure_responses and litellm_deployment_total_requests and
marks litellm_deployment_state as a partial outage again. Proxy-gate rejections
and a failing credential that belongs to the caller (a rejected per-user
credential, an api_key sent with the request, a per-user GitHub Copilot
session) keep the skip
* fix(prometheus): keep a caller's own Anthropic OAuth token failure out of the deployment metrics
A provider 401 on an Anthropic OAuth token the caller forwarded through the proxy is the caller's credential
failing, so it stays out of the deployment failure and outage series like a caller-supplied api_key. Adds
integration cells for an Anthropic deployment's own key, the forwarded caller OAuth token and a direct SDK
call that brings its own key
* test(prometheus): pin how a caller's own key or Authorization header on a deployment is counted
An SDK Router call that passes its own api_key is counted against the one-off deployment the router builds for that key, so the operator deployment keeps only its own failure. A provider 401 on an Authorization header the caller sends through the proxy counts against the deployment, as on v1.80.12
* fix(prometheus): keep caller-sent provider credentials and token-exchange 401s out of deployment metrics
A caller can send its own provider credential in the request body, such as
azure_ad_token, azure_scope, the Azure Entra app fields, watsonx token,
gigachat_access_token, OCI keys, AWS keys or Vertex credentials, at the top
level or inside extra_body. A provider 401 on that credential is the caller's
failure, so it stays out of the shared deployment's failure, request,
LLM-failed and state series, as on main. A token-exchange deployment trades the
caller's identity token on every call, so its 401s stay out as well, matching
the router cooldown.
* test(prometheus): Final locals, precise helper types and one for per comprehension in the new test helpers
* test(prometheus): pin that a provider 401 at a caller-sent api_base counts against its one-off deployment
* fix(prometheus): keep a provider 401 on caller-sent headers out of deployment metrics
A caller can override the deployment's credential with headers it sends in the
request body: headers or provider_specific_header, or an Authorization, api-key,
x-api-key, x-goog-api-key or Ocp-Apim-Subscription-Key entry in extra_headers.
A provider 401 on such a request is the caller's failure, so it stays out of the
shared deployment's failure, request, LLM-failed and state series, as on main.
* fix(prometheus): skip a provider 401 only when the caller's headers carry a credential
The proxy now records headers, extra_headers or provider_specific_header in
credential_fields only when they name a provider credential header
(Authorization, api-key, x-api-key, x-goog-api-key, Ocp-Apim-Subscription-Key),
the same way api_key is recorded. A caller that sends a plain header such as a
trace id still has a provider 401 on the deployment's own key counted against
the deployment.
* fix(prometheus): stamp credential_fields after the proxy forwards the caller's headers
A provider credential header the proxy forwards from the caller's HTTP request
(forward_llm_provider_auth_headers + forward_client_headers_to_llm_api) lands in
data["headers"] after proxy_server_request is built, so the stamp missed it and a
401 on that caller credential was counted against the deployment. Compute the
stamp after the header forwarding steps and pin both forwarded cases.
* fix(prometheus): skip a provider 401 when the caller sends any header or an OpenAI organization
The proxy now records headers, extra_headers or provider_specific_header in
credential_fields whenever the caller sends or the proxy forwards any header
there, and records organization when the caller sends one in the body or as a
forwarded OpenAI-Organization header. A forwarded header can be a gateway key
such as x-portkey-api-key or an OpenAI-Project id, and a wrong organization
gets a 401 from OpenAI, so these 401s stay out of the deployment metrics as on
main instead of marking a healthy deployment as a partial outage.
* fix(prometheus): ignore the user information the proxy adds when stamping caller headers
With add_user_information_to_llm_headers on and header forwarding enabled, the
proxy adds x-litellm-user_api_key_* headers to the forwarded headers. Those
carry the proxy's own key and user details, not the caller's, so they no longer
make a request look caller-scoped, and a provider 401 on the deployment's own
key on such a request is counted.
* fix(prometheus): skip a 401 raised before the provider is called or at the caller's own endpoint
A deployment with no key of its own fails with LiteLLM's local "Missing
Anthropic API Key" 401 before any provider call, and a caller who sends
aws_bedrock_runtime_endpoint (or another provider endpoint field) can get a
401 from their own gateway. Both were counted against the shared deployment
and marked it as a partial outage, where main skipped every 401.
Count a 401 only once the provider was called (api_call_start_time is set),
and treat caller-sent provider endpoint fields like caller credential fields.
* fix(prometheus): count the deployment's own 401 on requests that carry only anthropic-version or anthropic-beta
The Anthropic SDK sends anthropic-version on every request, and the proxy copies
anthropic-version and anthropic-beta into provider_specific_header for Anthropic,
Bedrock and Vertex. Both were treated as caller headers, so a 401 on an Anthropic
deployment's own key from any Anthropic SDK client stayed out of the deployment
metrics.
These two headers only pick the API version and beta features. Anthropic answers
a bad anthropic-version or an unknown anthropic-beta with a 400, never a 401, so
they cannot turn the deployment's own key into a 401 and no longer mark the
request as carrying a caller header.
* test(prometheus): read and write the owned proxy's own database in the expired-key and MCP cells
Since #47213 an owned proxy boots on a private database, so the expired-key update and the MCP guardrail row read now go through the gateway's database_url instead of the worker's.
* fix(prometheus): keep credential_fields as main stamps it and give Prometheus its own caller_credential_fields
credential_fields tells the Rust OCR bridge which stripped body fields the
caller sent. Repurposing it for the Prometheus caller-credential check changed
its value (organization and extra_headers added, anthropic-version-only
headers dropped). Restore main's stamp unchanged and stamp the Prometheus
classification under caller_credential_fields, which only Prometheus reads.
* fix(prometheus): skip a provider 401 on a stored credential the caller picks by litellm_credential_name
A request body litellm_credential_name swaps the deployment's stored
credential for one the caller chose, so a 401 on it is the caller's, not the
deployment's. Treat it like the other caller credential fields so it no longer
marks a healthy shared deployment as a partial outage.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(team): keep the team reset schedule when a temp budget increase is set on a member inheriting the team default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect the temp-only member row to carry the team default cap
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): gate key budget_duration and budget_id updates like max_budget
A key owner or team member could shorten budget_duration or swap budget_id on
/key/update without the budget authorization max_budget gets, so the reset job
cleared spend on a capped key or a bigger tier lifted its cap
Fixes#46916
* test(proxy): cover tier keys and self-serve budget policies for key budget updates
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(router): stop concurrent deployment deletes from crashing deployment lookups
delete_deployment and upsert_deployment shrink model_list before they repair
the index maps, so a lookup on another thread in that window read a stale
index: IndexError, or the neighbouring deployment and its credentials. An add
racing a delete could corrupt the index for good.
One RLock now covers model_list and the three index maps. Delete, upsert, add
and the set_model_list reset change them under it, and every lookup that turns
an index into a deployment reads under it.
Fixes#45242
* test(router): release the held delete when the read reaches the index lock
The race tests released the held write after a fixed 0.2 s, so a reader the runner
stalled past it read the repaired list and passed on unfixed code. The read now runs
on its own thread and the write stays held until the read returns or reaches the
router's index lock; the 10 s bound only fails a broken synchronization.
The integration test gains the audit cells: chat, streaming, Responses and
Anthropic messages calls (sync and async, by model name and team public name)
during a held delete, the same for upsert, add and set_model_list, a request for
the deployment being deleted, a provider error, a double delete, and thread and
async bursts during upsert churn.
* test(router): cover the early return of a delete and the team filter of get_model_ids
delete_deployment now returns None from inside the index lock, and get_model_ids by
name reads its deployments through _deployments_named; pin both: deleting an unknown
id returns None and leaves the list alone, and exclude_team_models drops only the
team deployment.
* fix(router): read deployments by name under the index lock outside the Router
The response polling provider check, the complexity router's adaptive pool and the
savings baseline turned the name index into list positions themselves, so a concurrent
delete could still hand them a stale slot. They now read through Router.deployments_named,
which takes the deployment index lock.
* fix(router): type the locked deployment lookups as DeploymentTypedDict
deployments_named and _team_deployments_named returned tuple[Any, ...]. They now return
DeploymentTypedDict rows, which is what model_list holds. The adaptive pool setup in the
Router and the complexity router drop their branches for pydantic Deployment rows, which
these lookups never return, and should_include_deployment accepts the typed row.
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text
The pass-through guardrail translation json-dumps the request body as the
scan text when no request_fields are configured. The unified guardrail hook
injects litellm_metadata into that body first, so the dump carried the
caller's key metadata. Exclude it alongside metadata and litellm_logging_obj;
vendors still receive it through request_data.
* test(guardrails): type the pass-through scan-text tests per AGENTS.md
* fix(otel): decide per-team Arize sampling from the trace id so every pod agrees
* test(otel): cover per-team sampling across pods, workers, endpoints and restarts
* fix(otel): remember only a failed tree's sampling verdict so a later failed call answers to the error rate
A successful tree's verdict follows from the trace id alone, so remembering it only made a later failed call of the same trace follow the success rate on the pod that served both, while other pods judged it at the error rate
* test(otel): flush every worker before the restart cell kills it and cover a failed call after a dropped one
* fix(otel): decide team sampling per request, not per trace
Calls sharing a caller's trace id are separate requests, keyed by their root span.
Each is decided by the trace hash at the rate for its own outcome, and only its own
late spans follow its verdict, so no pod applies an earlier call's verdict to a later
call of the trace
* fix(otel): forget a span's request root when it ends, not when newer spans start
The root of each open span was kept in a first-in-first-out table of 16384 entries,
so a span two levels below the root that stayed open while the worker started more
spans than that lost its entry and ended under its parent's id. Its request then
never closed and was held from every sampled destination until shutdown.
Each open span now keeps its entry until it ends, and a span starting under a parent
that has already ended reads the root from the parent span itself
* fix(otel): decide a request as its spans end once it falls out of the open span table
A request whose span was pushed out of the table of open spans, because the worker
held more than 16384 open at once, ended that span under another key and was held
from every sampled destination until shutdown. Pushing a span out now drops its
request's open count too, so the request is decided as each of its spans ends, the
way one pushed out of the open request table already is
* test(otel): annotate the team sampling test locals as Final
* test(otel): build the caller's remote parent in one helper instead of rebinding it per trace
* fix(otel): keep each open span's request until it ends, bounded by the open requests
The span table had its own bound of 16384, so a worker holding more spans open than
that pushed out the oldest, and a failure in such a span no longer reached its
request. Each request now holds the ids of its open spans, and a span's entry in the
lookup goes when the span ends or when its request is pushed out of the open request
table, the same bound main already applies. A failure anywhere in a tracked request
reaches its verdict however many spans are open
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(streaming): stop re-raising and re-logging a failed stream on every next()
A stream that raised keeps its provider iterator alive, so a consumer that iterates past the error re-detected, re-raised and re-logged the same failure on every call, flooding failure callbacks for one request. Latch the failure and re-raise it without logging again.
* fix(streaming): drop the retained traceback before re-raising a failed stream
* test(streaming): count failures through a real logging callback and require a clean stop on later reads
* test(streaming): require the stuck stream to close early and release its socket
The provider stream now records a GeneratorExit and caps its chunks, so draining it or detecting the loop late fails. Two wire cases check a sync and an async stuck stream release their TCP session.
* test(streaming): avoid failure recorder name collision
---------
Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
