* test(integration): cover audio endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close audio SDK clients after each test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): avoid list mutation in audio tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): keep exact upstream call check in model resolution test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover audio translations, bracket spellings, streaming and scoped alias keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin audio metadata parse sites, exact cost and speech bug pins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop inline comment on audio wire assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen audio wire and authorization contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover agents_a2a endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin agent daily spend to the logged request day and assert cleanups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect agent daily activity newest day first
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover A2A 1.0 task aliases, agent id headers, declared access groups and per-token spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert full A2A stream, header, alias, task and spend contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert forwarded a2a-version header on A2A sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): release A2A stream gate only after a full SSE frame
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin A2A trace id, stream headers, denied alias and task methods
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin A2A card protocol version
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the whole accepted push config response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover model_management endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the exact model access refusal after a rename
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for allowlist convergence in the rename test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give the rename test lint suppression its reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close reviewer gaps on model management contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): share the blocked serving assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen authorization and configuration regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): converge spend readbacks, ignore model discovery, cover slash credential names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the dashboard credential edit body round-trip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): count Admin UI page views and tab clicks through the proxy
* fix(ui): share one telemetry settings save across both tabs so a second toggle waits for the first
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): let every dashboard role reach the UI event routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): fail with the denial detail when a dashboard role cannot reach UI event routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): document POST /telemetry/ui_events with a description and a 200 response schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep the test-quality reason on the line pytest's checker reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): pick the telemetry destination on the Telemetry page and warn when none is chosen
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): send the destination along with groups in the telemetry settings hook test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): add a top-level telemetry switch and gate turning telemetry on behind an enterprise modal
The page can only turn telemetry off. Turning it on from the switch, a group
or a first destination opens a modal: enterprise proxies are told to contact
their LiteLLM enterprise representative, others that telemetry is an
enterprise-only feature. Group switches and the destination stay visible but
disabled while telemetry is off.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): keep reports in a local table with a persistent instance id and an admin export route
* feat(telemetry): key deployment hashes from a random secret stored in the proxy DB
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): retry a locally stored window under the same id so it is never counted twice
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(telemetry): state the overlap and dedupe contract for paging stored reports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate dashboard API types for the telemetry reports paging note
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): require an explicit LITELLM_TELEMETRY_DESTINATION or saved destination before recording
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate dashboard API types for the explicit telemetry destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): register the attempt logger and create its hash secret only once a sink is built
A proxy with a database that never opts in now carries no telemetry callback
and writes no deployment hash secret to LiteLLM_Config.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): record proxy requests and provider attempts into the telemetry sink
* fix(telemetry): validate env settings, fix the window and only record the request's own provider calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): bind the call id from a typed local
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record a finished request into the sink current when it finishes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record every failed provider attempt, including retries and fallbacks on a shared logging object
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): report timeouts and connection failures as their own provider status classes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): record failed attempts from the per-deployment failure hook instead of a new core logging hook
Reverts the async_log_attempt_failure_event hook added to CustomLogger and
litellm_logging in 3421b38685. The router already calls
async_post_call_failure_deployment_hook once per failed attempt, retries and
fallbacks included, so every failed fallback still gets its own row without
touching the shared logging object.
* fix(telemetry): register the attempt logger only once a telemetry group is turned on
A proxy that never opts in now carries no telemetry callback. Once registered
the logger stays, returning early while every group is off.
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(telemetry): add AggregatingSink, fixed-bucket histograms, Report and HttpExporter
* fix(telemetry): add records to rows in place, guard concurrent flushes, cap restored rows and count bad histogram values
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): give the shared row helper its lint reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(telemetry): retry the same statuses as the proxy's should_retry (408, 409, 429, 5xx)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(telemetry): cover populated block_types and header_keys in report JSON
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): skip custom metadata labels a metric already carries
A custom metadata label whose sanitized name matches one of a metric's
built-in labels was appended a second time, so every .labels() call on
that metric raised ValueError. With custom_prometheus_metadata_labels:
[project_alias], the per-project per-model rate limit gauges raised on
every request from a project key with per-model limits, and the rest of
the success event (latency, deployment success, total requests) was
skipped.
* test(prometheus): cover shared custom labels on every endpoint, failure, metadata and outage burst
* test(prometheus): drain the shared upstream at the start of each integration test
* test(prometheus): assert the forwarded route, model and stream per endpoint cell
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(telemetry): add litellm.telemetry records, consent groups and sink protocol
* docs(telemetry): reword TelemetryGroup docstring and drop module docstring
* fix(telemetry): add timeout and connection_error status classes, check relative imports in the package boundary test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude
Vertex AI Claude deployments that emulate response_format through a forced
tool sometimes nest the schema object under an extra key such as
`parameters` or `input`, and that envelope reached message.content with a
200. The converter now unwraps exactly one nested object that matches the
schema, leaving a payload that is itself schema-shaped alone.
Native structured-output rows sent the deprecated top-level `output_format`,
which Vertex rejects with a 400. The Vertex request now carries it as
`output_config.format`, with an explicit caller value winning.
* fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers
* test(anthropic): expect the json tool description through a shared constant
* fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys
* fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key
An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name
wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a
streamed answer whose first key is a schema property named name or tool_name waits for its second
key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided
past 4 KB is held to the end
* test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema
* fix(anthropic): unwrap json tool envelopes by structure instead of schema validation
The unwrap now reads only the schema's top-level property names. A lone
undeclared key holding an object, alone or beside a name equal to the json
tool's own name, is the envelope, and the unwrap repeats while the inner
object shares no key with the schema. jsonschema and referencing are no
longer imported on the request path. The stream holds only an answer that
opens with the tool's name or with an undeclared key holding an object, and a
schema without top-level properties streams as it arrives.
* fix(anthropic): walk json tool envelopes iteratively
* fix(anthropic): keep json tool arguments too deep to decode instead of raising
* fix(anthropic): unwrap json tool envelopes beside undeclared string labels
* fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent
A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter.
* test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost_calculator): price an explicit base_model from its own row when the deployment provider row has no rates
A chatgpt/ deployment with model_info.base_model recorded zero spend because the
base model was looked up under the chatgpt provider, which resolves to the
deliberately unpriced chatgpt/ row. When the deployment provider's row for the
base model declares no rates, price it under the provider of the base model's
own row instead. Deployments whose provider row declares rates are unchanged.
(cherry picked from commit 42d9a8693e)
* test(integration): bill a chatgpt base_model deployment from the priced row on a live proxy
Co-authored-by: sla55er <stiliyanmarkov@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: sla55er <stiliyanmarkov@gmail.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6
A cache_control mark on a system, developer, user or tool message sent to /v1/chat/completions now reaches an OpenAI deployment that reads prompt_cache_breakpoint (gpt-5.6+ on api.openai.com, or one opted in through prompt_cache_options) as a prompt_cache_breakpoint on the same content part, instead of being stripped. Marks on assistant turns and tool definitions are still dropped, and deployments that do not read breakpoints keep today's behavior.
* test(openai): rewrite chat prompt_cache_breakpoint wire test as sent-vs-received literals
Each test now shows the request the caller sends and the exact body OpenAI receives as plain JSON, with one helper that returns the upstream body. Drops the per-test marker plumbing and the spend-log check, which did not test the cache mark translation.
* test(openai): cover a marked chat message with null content on the breakpoint path
A system, developer, user or tool message that carries cache_control but whose content is neither a string nor a list of parts is handed to the existing strip untouched instead of raising inside the transform. The new case pins that the request still goes out with the mark stripped and no breakpoint added.
* refactor(openai): reuse the cache control hook's breakpoint placement on chat completions
The chat transform had its own copy of how a message-level cache_control lands on a part for OpenAI's prompt_cache_breakpoint. It now calls one hook helper that the injection path also uses, so which roles and part types can carry a breakpoint is decided in one place. A message-level mark on list content now lands on the last part OpenAI accepts a breakpoint on, matching injection.
* test(e2e): cover cache_control reaching gpt-5.6 explicit prompt caching on chat completions
Four live cases through the proxy: a marked system part, a marked string system message, a marked user part and a marked tool result. Each must write the cache on the first call and read it on the second.
* fix(openai): parse content parts instead of a banned TypeGuard when renaming cache marks
The strict ruff gate bans typing.TypeGuard as unverified narrowing. The part rename now validates an accepted part with pydantic before handing it to the shared rename helper.
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors
Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy
modules with one pydantic validation at the point the payload enters the
module, so every value read afterwards carries a real type.
reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright,
ruff-strict or type-discipline rule is higher repo-wide or in any single
file, and no cast, ignore or suppression marker was added.
* chore(typing): replace 47 Any annotations that every caller already satisfies with object
Each annotation here is a parameter or return type whose value is only
stored, compared, logged or forwarded, so object describes it exactly and
no function body changes. A site was kept only if no basedpyright,
ruff-strict or type-discipline rule rose in any file that references it.
reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655.
* chore(typing): type upstream responses and loose annotations across providers, proxy and integrations
Second pass of the Any reduction. Decoded upstream JSON is validated once at
the provider boundary and read through typed views afterwards, annotations
that every caller already satisfies move from Any to a concrete type, and the
nine search transformations validate each result item on its own so error
text no longer carries the item index.
No cast, no type: ignore, no suppression marker and no budget file edit
* ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard
* fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6
Native /v1/responses used to strip every Anthropic cache_control mark, so a
gpt-5.6 deployment in explicit mode never wrote a cache. Marks on system,
developer and user parts and on function_call_output now become
prompt_cache_breakpoint on the same block when the deployment reads
breakpoints, and are stripped exactly as before everywhere else
* fix(openai): read str providers and base_url in the prompt cache breakpoint gate
Responses configs generated from a provider JSON file, and openai_like
itself, name their provider as a plain str, so .value crashed every native
Responses request through them. The gate also only read api_base, while
responses() keeps a caller's base_url as an extra field on the params, so an
injected mark on a custom host was rewritten to a breakpoint
* test(openai): cover the pass-through branches of the prompt cache breakpoint swap
A part type OpenAI does not cache keeps no breakpoint and an item mark skips
it, and an item or content of a shape OpenAI will reject reaches the strip
untouched instead of failing inside the proxy
* fix(prompt_caching): keep a client-written prompt_cache_breakpoint when a cache_control mark lands on the same block
A JSON-RPC tools/call on /{server_name}/mcp (or /mcp with x-mcp-servers naming one server) that uses the upstream's bare tool name answered "Tool not found" on every proxy process that had not yet listed that server, because the per-process tool-name map stays empty for a server whose upstream only lists tools with the caller's own credentials, and mcp SDK 2 no longer lists before an uncached call. call_mcp_tool now hands the single scoped server to execute_mcp_tool as requested_server_id, so the existing REST contract lists the server with the caller's headers and routes the bare name to it, and a bare name shared by two servers goes to the one the path names
Co-authored-by: aayushbaluni <73417844+aayushbaluni@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast team cache invalidation on team writes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover cross-instance team cache invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): source Home What's new from day 0 support blog posts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): Home cards read card_title and card_summary from blog tags with provider logo
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files
Replace sleeps that raced background logging, cache writes, refreshes and
thread work with either awaiting that work directly (logging worker flush,
the batch logger's flush_queue, the tasks the call spawned, thread joins) or
an Event the test's own fake sets when it runs. Each wait keeps a 5s hang
guard that never decides pass or fail. Tests that need a fake clock, real
sockets, or measure latency are left unchanged, and no production code moves
* test(unit): await the tasks each completion spawns instead of yielding once
The logging tests yielded once with asyncio.sleep(0) so the task that
enqueues the success log could run before the worker flush. That relied
on the enqueue happening before the task's first await. Snapshot the
running tasks around each acompletion call and await the ones it spawned,
so the guarantee holds however many steps sit before the enqueue
* fix(logging): redact nested callback input and provider response fields
* fix(logging): keep shared request body for per-callback redaction and drop hidden provider fields
The deprecated per-callback message_logging=False path ran perform_redaction on
the shared model_call_details, which now also redacted the nested
proxy_server_request body. The spend log Request column and
previous_response_id replay read that body, so one opted-out callback blanked
them for everyone. That path now keeps the shared body and hidden params, and
redact_streaming_responses_for_custom_logger hands the opted-out callback its
own redacted litellm_params.
Redacted litellm_params also clear provider_specific_fields copied into
metadata.hidden_params (Anthropic thinking text), in litellm_params and in the
nested request body, and the nested body redacts system and instructions.
* fix(logging): keep replayed instructions and router usage marker under redaction
The shared request body keeps system and instructions again, so spend logs
store them and previous_response_id follow-ups replay the real instructions.
Per-callback copies still redact them.
Provider-specific fields in metadata.hidden_params are cleared in place, so
the metadata dict keeps its identity and the router's counted-usage marker
stays visible to deployment_callback_on_success (no double tpm/rpm count).
* fix(logging): never copy shared metadata dicts during redaction
The shared redaction pass now leaves litellm_params metadata dicts as they
are, so a hidden_params write from the other success handler between the
in-place clear and the copy can no longer move the router usage marker to a
stale dict. The key-presence predicate change is reverted as unneeded.
* refactor(logging): redact nested request body and provider fields with the existing message rules
Run perform_redaction on a copy of proxy_server_request.body, the same call the
spend log already makes, and redact reasoning inside provider_specific_fields the
way the message's own reasoning_content and thinking_blocks are redacted, instead
of clearing every provider field. Opaque fields such as Gemini thought_signatures
stay, and the metadata dicts and streaming objects are handled as on main.
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): return provider-matching 4xx for the LIT-8062 status code follow-ups
- OCR multipart errors are 400 invalid_request_error on the file param
- a missing model is a 400 on chat, responses and messages even with a wildcard deployment
- responses max_output_tokens <= 0 reaches the provider instead of being raised to 16
- daily activity rejects end_date before start_date with a 400
- vector store file attributes are sent in full so the provider enforces its key limit
- router-wide defaults no longer satisfy a param the Router takes positionally, or a model the router does not serve
- a JSON null user_config is treated as absent
* chore: keep the type gates at their merge-base counts
* fix(proxy): keep fixed-target wildcards serving a missing model and close router-default gaps
- a missing model is rejected only when the wildcard would forward it into the target
- moderation only counts router defaults for models get_model_list lists
- user_config defaults and deployment params never satisfy a positionally bound param
* test(integration): pin the fixed-target wildcard path and no provider call on positional-param 400s
* chore: validate pattern-matched deployments instead of reading untyped dicts
* fix(proxy): keep router defaults counting for a2a agent models
* test: drop the signature-walk test and type the new test helpers
* test(integration): audit matrix for the status-code follow-ups
Drives every mode the fix branches on through a real proxy, Postgres, Redis and
the scripted upstream: positional params against router, deployment and
user_config defaults; moderation listed vs unlisted; unknown and a2a models;
every user_config shape on chat, responses and messages, streaming and SDK
clients; forwarding vs fixed-target wildcards for missing, null and empty
models; Responses max_output_tokens edges; OCR parse errors; vector store file
attributes; every daily activity route; and a provider outage mid burst plus a
killed worker on a two-worker proxy. Product gaps found are skipped as BUG.
* style: ruff format the audit test files
* test: name the auth pre-check as the source of the JSON array 500
* test: build the audit helpers without mutation and share one sentinel
* test: build every audit request body as one literal
* fix(proxy): keep router defaults and model-less dispatch for requests a fallback chain serves
* fix(proxy): count request, key and team fallbacks when an unlisted model inherits router defaults
* fix(proxy): skip key and team fallbacks on the client-credential dispatch path
* fix(proxy): keep router defaults when the router's own * fallback serves an unlisted model
* fix(proxy): count client-style fallbacks, forward an empty model, report invalid usage chat dates
- The router-default pre-check now treats client-style request, key and team fallbacks
([{"model": ...}]) as serving an unlisted model, as the router does
- The forwarding-wildcard gate only rejects a missing or null model; an empty model
reaches the provider as before (vLLM serves its loaded model for it)
- The /usage/ai/chat tools hand the model the date-range error instead of an empty
dataset or a generic fetch error
* test(usage): type the invalid-date tool test
* fix(proxy): keep rejecting an empty model behind a forwarding wildcard
Key and team model checks skip an empty model, so forwarding it let a key
restricted to another model reach a server's default model (vLLM serves
its loaded model for ""). Pin that with a restricted-key integration cell.
* fix(usage): report the query layer's date 400 to the usage chat model
Checking dates before the fetch refused formats the global usage tool's query
accepts (2026/10/01 returned spend on main). Catch the daily activity query's
own 400 instead, so each tool rejects exactly what it rejected before and only
the message changes.
* fix(proxy): restore three model-less and unlisted-model paths main served
An empty model is looked up as sent, so next to a forwarding */* wildcard it
still reaches the fixed * deployment it matches. A null model with a prompt_id
is left to the prompt manager, which may pick the model. A Claude Code
subagent turn on a session-bound auto/complexity router counts as served, so
the router default max_tokens still applies.
* fix(proxy): only skip the model check for a prompt_id when a prompt manager is registered
Without a callback prompt manager nothing can pick the model, so a null model
with a prompt_id behind a forwarding wildcard gets the same 400 as any other
null model.
* fix(proxy): leave a model-less Claude Code subagent turn to its session router
A subagent turn on a session bound to an auto, complexity, adaptive or
quality router is answered by that router whatever model it names, so a
null or missing model behind a forwarding wildcard is no longer rejected.
An empty model stays rejected.
* fix(proxy): skip the wildcard model check only for a bound Claude Code session
Read the session's router binding, as the router does, instead of only
checking that the request could resume one, so a subagent turn without a
model on an unbound session is still rejected before a forwarding
wildcard sends it upstream.
* refactor(proxy): stop predicting routing in the proxy pre-checks
- drop the wildcard missing-model gate: a model-less request behind a forwarding wildcard goes to the router and gets the provider's own status, as on main
- drop the "model the router serves" check: router defaults count for any named model again, except moderation, where the Router merges them only for listed models
- keep the positional-param, moderation and null user_config fixes
* test(ocr): cover the form, size, empty body, invalid JSON and file-document 400s
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
test_cold_concurrent_schema_validation_accepts_valid_arguments pins the
process to a single CPU core and fans out 8 concurrent calls through
anyio.to_process's subprocess worker pool, each needing to cold-spawn a
fresh interpreter and import jsonschema. Under a slow or contended CI
runner that routinely exceeds the production 30-second budget before any
actual validation work runs, failing the test on a genuine timeout rather
than a validation bug.
_tool_argument_validation_error now takes an optional timeout_seconds,
defaulting to the existing constant, so production behavior is
unchanged. The test passes a timeout sized for its own single-core,
cold-start stress scenario, matching the margin the companion
integration test (test_schema_validation_stops_expensive_work_and_recovers)
already grants a similar concurrency=8 case.
Each test below passed or failed on the same commit depending on test order
or port timing in the CircleCI integration workflow
- readiness probe only polls a port the launched proxy owns, so a stolen
port no longer sends GET /health/integration-readiness to another test's
wire server
- team update tests seed fresh member users, since a proxy admin relocation
adds team members to the destination org and leaked org_b membership into
the shared INTERNAL_USER
- realtime whisper tests wait until every proxy worker serves only the new
gpt-realtime-whisper deployment, via a per-worker deployment id route
- configured fallbacks test runs on its own proxy and database, because
/config/update persists router settings and cannot unset fallbacks, which
left [] and changed the no-fallback error text other tests assert
- otel excluded services tests each get a private Redis so a sibling proxy
cannot warm the budget registry cache the tenant span depends on
- Langfuse audit upstream uses unique response ids instead of len(targets),
which could repeat under concurrency and drop SpendLogs rows
* test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): bring back known_bug and carry the xai grok-4.7 stream breakdown bug as a strict xfail case
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover streamed 0.3 replies through the real client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): accept new kwargs in create_a2a_client fake; add protocol-version integration tests
The create_a2a_client fake in tests/agent_tests/test_a2a_agent.py listed a
fixed signature, so asend_message's new protocol_version= kwarg raised
TypeError in test_a2a_non_streaming and test_a2a_streaming. Accept and
ignore extra kwargs with **_, matching the new _capture fake in test_main.py.
Add tests/integration/compatibility cells for litellm_params.a2a_protocol_version
against a real 2-worker proxy with Postgres and Redis and scripted A2A peers
modeled on langgraph-api 0.15.4: pinned send/stream on all three routes, aliases,
the 1.0 opt-out, unchanged genuine v1 agents, PATCH taking effect on both
workers, persistence via /v1/agents, the localhost retry, invalid values,
upstream errors, the dialect hint, and the completion bridge stripping the key.
Merge base 5f969982fc: 33 red, 7 green. Head ba709d3de5: 40/40 green twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(focus): add ternary destination for FOCUS export
Add a `ternary` FOCUS export destination (plus a `ternary` callback) so a LiteLLM proxy can push FOCUS-formatted spend to Ternary with `callbacks: ["ternary"]` -- no separate export job
- FocusTernaryDestination: a pure sink modeled on the sibling vantage/mavvrik/gcs destinations; POSTs the FOCUS CSV to the Ternary cost-ingestion endpoint, chunking an oversized backfill (csv-module repartition, quoted-newline safe; oversized single row raises rather than silently dropping) with a stable X-Ternary-Upload-Id + 0-based index/total so the receiver stages-then-swaps. connection_id is validated + URL-encoded; base_url must be HTTPS (loopback exempt for local); the upload has a bounded timeout; it aborts on the first failed chunk so nothing partial lands
- TernaryLogger: presets the FOCUS pipeline (CSV) and enriches the FOCUS Tags JSON with the per-request token counts the shared transformer drops (Ternary weights allocation by tokens); the enrichment is confined to this path and only adds keys. Cadence is daily (default) or interval (validation loops, an explicit interval-seconds required); the export window is day-aligned and drops days before the window start so a whole-day-replace receiver never truncates
- Registration mirrors vantage/mavvrik: ternary callback literal, logging dispatch, scheduler job, and CustomLoggerRegistry entry -- no shared-code changes
- Tests: mocked unit tests only (no live calls), 100% line coverage of both new modules
* fix(focus): warn instead of silently dropping token tags on mismatch
_merge_token_tags returned the frame unchanged when the transformed data
had no `Tags` column or its row count disagreed with the source, so every
token count was dropped with no signal -- the receiver then lands NULLs
that read as "this gateway reports no tokens". Emit a warning on each
degenerate path (mirroring _drop_days_before), and cover both with caplog
tests asserting the warning and the unchanged return.
* feat(focus): upload one UTC day per Ternary signed-URL PUT
Replace the multipart POST with chunking by a per-day signed-URL upload,
matching the Mavvrik destination's pattern. The logger partitions the raw
usage rows on `date` (dropping unparseable dates with a warning), then
transforms, enriches and delivers each day separately; one day's failure
is logged and the remaining days still upload, with a single error raised
at the end. The destination requests a signed URL for the day, validates it
is an HTTPS GCS URL, and PUTs the CSV with exactly the signed headers.
Removes the 30 MiB / 100k-row chunking, the X-Ternary-* upload headers and
the .partN filenames; a re-sent day overwrites its object, so retries stay
idempotent without them.
* fix(focus): re-send the last two whole days so late-flushed spend lands
LiteLLM writes DailyUserSpend in batches, so spend dated yesterday can be
flushed after the 00:05 UTC daily export. That export only covered the
previous UTC midnight onward, and the next day's export moved past it, so
the tail of each day was never sent. Start the window at UTC midnight two
days back: the following run re-sends yesterday whole, overwriting its
per-day object with the complete day.
* fix(focus): lazy-load the Ternary engine, fetch whole days, keep ternary out of the generic factory
Every proxy imported polars at boot through the logger registry, so a proxy without polars failed to start even with no FOCUS callback. The polars-backed engine now lives in its own module and loads on the first export
A non-UTC Postgres session timezone shifted the naive updated_at filter, so the oldest day in each run lost rows and Ternary's whole-day replace made the loss permanent. The Ternary read now starts one day early with no upper bound and keeps rows by their own UTC date
callbacks: ["focus"] with FOCUS_PROVIDER=ternary uploaded a parquet blob of one interval over a whole-day CSV file. The generic factory no longer builds a Ternary destination and the Ternary engine injects its own
* fix(focus): keep only the days a Ternary export window covers
Dropping the updated_at upper bound let a bounded export_usage_data window upload every day through today. Rows are now kept by their UTC date between the window's first day and the day before its end instant
* refactor(focus): leave the shared FOCUS export engine as on main
The Ternary engine sets the base engine's fields itself and passes its own
destination, so focus/export_engine.py and its test match main again.
---------
Co-authored-by: Aaron Sproul <aaron@ternary.app>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(guardrails): add Wingback guardrail integration
Mirrored from #38685 at d73b66544d (net diff against its merge base, applied on main)
* fix(guardrails): dispatch Wingback through the unified guardrail
The proxy only runs a unified guardrail when apply_guardrail is defined on
the guardrail's own class, so WingbackGuardrail now binds the parent's
decorated method in its class body. Chat, messages and responses pre, post
and during call checks reach Wingback again, and each check logs once.
Also types the initializer and config model for the lint gates and uses
monkeypatch.setenv in the unit test
* chore(guardrails): drop the Wingback dispatch comment
The unit and integration hook tests already fail when the apply_guardrail binding is removed, so the comment is redundant under the repo comment rule
---------
Co-authored-by: bsk-wb <bharath@wingback.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(cost): add BridgeParityTestCase with a gpt-6-sol /v1/messages to Responses pair
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): rename BridgeParityTestCase to BridgeCostParityTestCase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): collect BRIDGE_COST_PARITY only from modules that define one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep the ARN regional-row regression tests to the touched lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
