* test(integration): cover mcp_management endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): require server_not_found for pending and rejected submissions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin per-worker submission reloads, toolset and alias-collision spellings, raw key overrides
Assert the approving and rejecting worker immediately and only wait on the peer, add an outsider key and exact upstream calls, list by toolset_name, cover a config alias shadowing a DB server_name, send key_hash as a raw sk- key, and add BUG skips for API disallowed_tools and overrides on keys or teams without object_permission
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin XFF trusted-peer, load balancer chains and sibling refusal body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): enforce output_policy trust chain, JSON-RPC XFF views, MCP tool-policy BUG pins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): force a registry reload during the pending window, pin the admin default listing and the whole tools/call body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover realtime endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): reject duplicate multipart parts and cover the transcription_sessions alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the type ignore and check the aliased transcription response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin realtime multipart part types, replayed transcription session, undeclared session fields and budget refusal path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move the refusal path note next to the expected message
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert realtime refusal status before parsing and pin the replayed transcription session whole
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): sort the realtime webrtc test imports
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pick the transcription mint and token with one Final each
* test(integration): pin realtime transcription mint, replay, and redeem contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(integration): ruff format the realtime webrtc test
* test(integration): drop the redundant comment on the realtime SDK calls test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): type the realtime client secret refusals table
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(realtime): tighten endpoint contract assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover videos_containers endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): end the first container page on an ownerless id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert container upload parts, SDK video extend, bare video ids, cost from map
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): catch a stale container list cache, cover video edit JSON, bracketed video refs and object input_reference
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin container optional fields, video client extras and the /videos alias on the wire
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin multipart character id decoding on video create as a BUG skip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover chat_completions endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin per-call prompts and both n=2 chat choices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover mixed path-model auth, string guardrail rejection and typed text-completion forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the prompt-spellings cache workaround and cover Azure SDK text completions on the deployments alias
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover audio endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close audio SDK clients after each test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): avoid list mutation in audio tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): keep exact upstream call check in model resolution test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover audio translations, bracket spellings, streaming and scoped alias keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin audio metadata parse sites, exact cost and speech bug pins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop inline comment on audio wire assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen audio wire and authorization contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover agents_a2a endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin agent daily spend to the logged request day and assert cleanups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect agent daily activity newest day first
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover A2A 1.0 task aliases, agent id headers, declared access groups and per-token spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert full A2A stream, header, alias, task and spend contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert forwarded a2a-version header on A2A sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): release A2A stream gate only after a full SSE frame
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin A2A trace id, stream headers, denied alias and task methods
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin A2A card protocol version
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the whole accepted push config response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover model_management endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the exact model access refusal after a rename
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for allowlist convergence in the rename test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give the rename test lint suppression its reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): close reviewer gaps on model management contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): share the blocked serving assertions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strengthen authorization and configuration regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): converge spend readbacks, ignore model discovery, cover slash credential names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin the dashboard credential edit body round-trip
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(prometheus): skip custom metadata labels a metric already carries
A custom metadata label whose sanitized name matches one of a metric's
built-in labels was appended a second time, so every .labels() call on
that metric raised ValueError. With custom_prometheus_metadata_labels:
[project_alias], the per-project per-model rate limit gauges raised on
every request from a project key with per-model limits, and the rest of
the success event (latency, deployment success, total requests) was
skipped.
* test(prometheus): cover shared custom labels on every endpoint, failure, metadata and outage burst
* test(prometheus): drain the shared upstream at the start of each integration test
* test(prometheus): assert the forwarded route, model and stream per endpoint cell
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(vertex_ai): unwrap tool-emulated json_schema envelopes and send output_config.format for Claude
Vertex AI Claude deployments that emulate response_format through a forced
tool sometimes nest the schema object under an extra key such as
`parameters` or `input`, and that envelope reached message.content with a
200. The converter now unwraps exactly one nested object that matches the
schema, leaving a payload that is itself schema-shaped alone.
Native structured-output rows sent the deprecated top-level `output_format`,
which Vertex rejects with a 400. The Vertex request now carries it as
`output_config.format`, with an explicit caller value winning.
* fix(anthropic): unwrap tool-emulated json_schema envelopes on streams and tool-name wrappers
* test(anthropic): expect the json tool description through a shared constant
* fix(anthropic): drop the json tool description that makes sonnet-5 omit required keys
* fix(anthropic): keep schema-accepted answers whole and decide held streams at the second key
An answer the json_schema accepts is returned as is before any envelope rule runs, the tool-name
wrapper is unwrapped only when the schema rejects it, a malformed schema turns the unwrap off, a
streamed answer whose first key is a schema property named name or tool_name waits for its second
key before streaming, the held fragments go through a StringIO, and a prefix that stays undecided
past 4 KB is held to the end
* test(anthropic): keep the stream family cells to providers whose stream iterator gets the schema
* fix(anthropic): unwrap json tool envelopes by structure instead of schema validation
The unwrap now reads only the schema's top-level property names. A lone
undeclared key holding an object, alone or beside a name equal to the json
tool's own name, is the envelope, and the unwrap repeats while the inner
object shares no key with the schema. jsonschema and referencing are no
longer imported on the request path. The stream holds only an answer that
opens with the tool's name or with an undeclared key holding an object, and a
schema without top-level properties streams as it arrives.
* fix(anthropic): walk json tool envelopes iteratively
* fix(anthropic): keep json tool arguments too deep to decode instead of raising
* fix(anthropic): unwrap json tool envelopes beside undeclared string labels
* fix(anthropic): keep json answers whole on schemas that allow extra keys and stream held text as sent
A schema that allows undeclared keys and requires none now keeps an undeclared wrapper key on both the stream and non-stream paths. When no envelope is found, the held stream text is now sent exactly as the model wrote it. On Vertex /v1/messages, output_format now wins over output_config.format when both are set, which matches the Anthropic adapter.
* test(anthropic): cover label envelopes, dynamic-key schemas, as-written held streams and output_format precedence
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(cost_calculator): price an explicit base_model from its own row when the deployment provider row has no rates
A chatgpt/ deployment with model_info.base_model recorded zero spend because the
base model was looked up under the chatgpt provider, which resolves to the
deliberately unpriced chatgpt/ row. When the deployment provider's row for the
base model declares no rates, price it under the provider of the base model's
own row instead. Deployments whose provider row declares rates are unchanged.
(cherry picked from commit 42d9a8693e)
* test(integration): bill a chatgpt base_model deployment from the priced row on a live proxy
Co-authored-by: sla55er <stiliyanmarkov@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: sla55er <stiliyanmarkov@gmail.com>
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openai): carry cache_control to prompt_cache_breakpoint on chat completions to gpt-5.6
A cache_control mark on a system, developer, user or tool message sent to /v1/chat/completions now reaches an OpenAI deployment that reads prompt_cache_breakpoint (gpt-5.6+ on api.openai.com, or one opted in through prompt_cache_options) as a prompt_cache_breakpoint on the same content part, instead of being stripped. Marks on assistant turns and tool definitions are still dropped, and deployments that do not read breakpoints keep today's behavior.
* test(openai): rewrite chat prompt_cache_breakpoint wire test as sent-vs-received literals
Each test now shows the request the caller sends and the exact body OpenAI receives as plain JSON, with one helper that returns the upstream body. Drops the per-test marker plumbing and the spend-log check, which did not test the cache mark translation.
* test(openai): cover a marked chat message with null content on the breakpoint path
A system, developer, user or tool message that carries cache_control but whose content is neither a string nor a list of parts is handed to the existing strip untouched instead of raising inside the transform. The new case pins that the request still goes out with the mark stripped and no breakpoint added.
* refactor(openai): reuse the cache control hook's breakpoint placement on chat completions
The chat transform had its own copy of how a message-level cache_control lands on a part for OpenAI's prompt_cache_breakpoint. It now calls one hook helper that the injection path also uses, so which roles and part types can carry a breakpoint is decided in one place. A message-level mark on list content now lands on the last part OpenAI accepts a breakpoint on, matching injection.
* test(e2e): cover cache_control reaching gpt-5.6 explicit prompt caching on chat completions
Four live cases through the proxy: a marked system part, a marked string system message, a marked user part and a marked tool result. Each must write the cache on the first call and read it on the second.
* fix(openai): parse content parts instead of a banned TypeGuard when renaming cache marks
The strict ruff gate bans typing.TypeGuard as unverified narrowing. The part rename now validates an accepted part with pydantic before handing it to the shared rename helper.
* fix(openai): carry cache_control to prompt_cache_breakpoint on /v1/responses to gpt-5.6
Native /v1/responses used to strip every Anthropic cache_control mark, so a
gpt-5.6 deployment in explicit mode never wrote a cache. Marks on system,
developer and user parts and on function_call_output now become
prompt_cache_breakpoint on the same block when the deployment reads
breakpoints, and are stripped exactly as before everywhere else
* fix(openai): read str providers and base_url in the prompt cache breakpoint gate
Responses configs generated from a provider JSON file, and openai_like
itself, name their provider as a plain str, so .value crashed every native
Responses request through them. The gate also only read api_base, while
responses() keeps a caller's base_url as an extra field on the params, so an
injected mark on a custom host was rewritten to a breakpoint
* test(openai): cover the pass-through branches of the prompt cache breakpoint swap
A part type OpenAI does not cache keeps no breakpoint and an item mark skips
it, and an item or content of a shape OpenAI will reject reaches the strip
untouched instead of failing inside the proxy
* fix(prompt_caching): keep a client-written prompt_cache_breakpoint when a cache_control mark lands on the same block
* fix(proxy): broadcast team cache invalidation on team writes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover cross-instance team cache invalidation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): source Home What's new from day 0 support blog posts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): Home cards read card_title and card_summary from blog tags with provider logo
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): redact nested callback input and provider response fields
* fix(logging): keep shared request body for per-callback redaction and drop hidden provider fields
The deprecated per-callback message_logging=False path ran perform_redaction on
the shared model_call_details, which now also redacted the nested
proxy_server_request body. The spend log Request column and
previous_response_id replay read that body, so one opted-out callback blanked
them for everyone. That path now keeps the shared body and hidden params, and
redact_streaming_responses_for_custom_logger hands the opted-out callback its
own redacted litellm_params.
Redacted litellm_params also clear provider_specific_fields copied into
metadata.hidden_params (Anthropic thinking text), in litellm_params and in the
nested request body, and the nested body redacts system and instructions.
* fix(logging): keep replayed instructions and router usage marker under redaction
The shared request body keeps system and instructions again, so spend logs
store them and previous_response_id follow-ups replay the real instructions.
Per-callback copies still redact them.
Provider-specific fields in metadata.hidden_params are cleared in place, so
the metadata dict keeps its identity and the router's counted-usage marker
stays visible to deployment_callback_on_success (no double tpm/rpm count).
* fix(logging): never copy shared metadata dicts during redaction
The shared redaction pass now leaves litellm_params metadata dicts as they
are, so a hidden_params write from the other success handler between the
in-place clear and the copy can no longer move the router usage marker to a
stale dict. The key-presence predicate change is reverted as unneeded.
* refactor(logging): redact nested request body and provider fields with the existing message rules
Run perform_redaction on a copy of proxy_server_request.body, the same call the
spend log already makes, and redact reasoning inside provider_specific_fields the
way the message's own reasoning_content and thinking_blocks are redacted, instead
of clearing every provider field. Opaque fields such as Gemini thought_signatures
stay, and the metadata dicts and streaming objects are handled as on main.
---------
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(proxy): return provider-matching 4xx for the LIT-8062 status code follow-ups
- OCR multipart errors are 400 invalid_request_error on the file param
- a missing model is a 400 on chat, responses and messages even with a wildcard deployment
- responses max_output_tokens <= 0 reaches the provider instead of being raised to 16
- daily activity rejects end_date before start_date with a 400
- vector store file attributes are sent in full so the provider enforces its key limit
- router-wide defaults no longer satisfy a param the Router takes positionally, or a model the router does not serve
- a JSON null user_config is treated as absent
* chore: keep the type gates at their merge-base counts
* fix(proxy): keep fixed-target wildcards serving a missing model and close router-default gaps
- a missing model is rejected only when the wildcard would forward it into the target
- moderation only counts router defaults for models get_model_list lists
- user_config defaults and deployment params never satisfy a positionally bound param
* test(integration): pin the fixed-target wildcard path and no provider call on positional-param 400s
* chore: validate pattern-matched deployments instead of reading untyped dicts
* fix(proxy): keep router defaults counting for a2a agent models
* test: drop the signature-walk test and type the new test helpers
* test(integration): audit matrix for the status-code follow-ups
Drives every mode the fix branches on through a real proxy, Postgres, Redis and
the scripted upstream: positional params against router, deployment and
user_config defaults; moderation listed vs unlisted; unknown and a2a models;
every user_config shape on chat, responses and messages, streaming and SDK
clients; forwarding vs fixed-target wildcards for missing, null and empty
models; Responses max_output_tokens edges; OCR parse errors; vector store file
attributes; every daily activity route; and a provider outage mid burst plus a
killed worker on a two-worker proxy. Product gaps found are skipped as BUG.
* style: ruff format the audit test files
* test: name the auth pre-check as the source of the JSON array 500
* test: build the audit helpers without mutation and share one sentinel
* test: build every audit request body as one literal
* fix(proxy): keep router defaults and model-less dispatch for requests a fallback chain serves
* fix(proxy): count request, key and team fallbacks when an unlisted model inherits router defaults
* fix(proxy): skip key and team fallbacks on the client-credential dispatch path
* fix(proxy): keep router defaults when the router's own * fallback serves an unlisted model
* fix(proxy): count client-style fallbacks, forward an empty model, report invalid usage chat dates
- The router-default pre-check now treats client-style request, key and team fallbacks
([{"model": ...}]) as serving an unlisted model, as the router does
- The forwarding-wildcard gate only rejects a missing or null model; an empty model
reaches the provider as before (vLLM serves its loaded model for it)
- The /usage/ai/chat tools hand the model the date-range error instead of an empty
dataset or a generic fetch error
* test(usage): type the invalid-date tool test
* fix(proxy): keep rejecting an empty model behind a forwarding wildcard
Key and team model checks skip an empty model, so forwarding it let a key
restricted to another model reach a server's default model (vLLM serves
its loaded model for ""). Pin that with a restricted-key integration cell.
* fix(usage): report the query layer's date 400 to the usage chat model
Checking dates before the fetch refused formats the global usage tool's query
accepts (2026/10/01 returned spend on main). Catch the daily activity query's
own 400 instead, so each tool rejects exactly what it rejected before and only
the message changes.
* fix(proxy): restore three model-less and unlisted-model paths main served
An empty model is looked up as sent, so next to a forwarding */* wildcard it
still reaches the fixed * deployment it matches. A null model with a prompt_id
is left to the prompt manager, which may pick the model. A Claude Code
subagent turn on a session-bound auto/complexity router counts as served, so
the router default max_tokens still applies.
* fix(proxy): only skip the model check for a prompt_id when a prompt manager is registered
Without a callback prompt manager nothing can pick the model, so a null model
with a prompt_id behind a forwarding wildcard gets the same 400 as any other
null model.
* fix(proxy): leave a model-less Claude Code subagent turn to its session router
A subagent turn on a session bound to an auto, complexity, adaptive or
quality router is answered by that router whatever model it names, so a
null or missing model behind a forwarding wildcard is no longer rejected.
An empty model stays rejected.
* fix(proxy): skip the wildcard model check only for a bound Claude Code session
Read the session's router binding, as the router does, instead of only
checking that the request could resume one, so a subagent turn without a
model on an unbound session is still rejected before a forwarding
wildcard sends it upstream.
* refactor(proxy): stop predicting routing in the proxy pre-checks
- drop the wildcard missing-model gate: a model-less request behind a forwarding wildcard goes to the router and gets the provider's own status, as on main
- drop the "model the router serves" check: router defaults count for any named model again, except moderation, where the Router merges them only for listed models
- keep the positional-param, moderation and null user_config fixes
* test(ocr): cover the form, size, empty body, invalid JSON and file-document 400s
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
Each test below passed or failed on the same commit depending on test order
or port timing in the CircleCI integration workflow
- readiness probe only polls a port the launched proxy owns, so a stolen
port no longer sends GET /health/integration-readiness to another test's
wire server
- team update tests seed fresh member users, since a proxy admin relocation
adds team members to the destination org and leaked org_b membership into
the shared INTERNAL_USER
- realtime whisper tests wait until every proxy worker serves only the new
gpt-realtime-whisper deployment, via a per-worker deployment id route
- configured fallbacks test runs on its own proxy and database, because
/config/update persists router settings and cannot unset fallbacks, which
left [] and changed the no-fallback error text other tests assert
- otel excluded services tests each get a private Redis so a sibling proxy
cannot warm the budget registry cache the tenant span depends on
- Langfuse audit upstream uses unique response ids instead of len(targets),
which could repeat under concurrency and drop SpendLogs rows
* test(cost): add xai grok-4.7, Bedrock runtime chat completions and Bedrock Mantle cost cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): bring back known_bug and carry the xai grok-4.7 stream breakdown bug as a strict xfail case
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(a2a): add per-agent a2a_protocol_version override for langgraph-api 0.15 cards
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): cover streamed 0.3 replies through the real client
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(a2a): accept new kwargs in create_a2a_client fake; add protocol-version integration tests
The create_a2a_client fake in tests/agent_tests/test_a2a_agent.py listed a
fixed signature, so asend_message's new protocol_version= kwarg raised
TypeError in test_a2a_non_streaming and test_a2a_streaming. Accept and
ignore extra kwargs with **_, matching the new _capture fake in test_main.py.
Add tests/integration/compatibility cells for litellm_params.a2a_protocol_version
against a real 2-worker proxy with Postgres and Redis and scripted A2A peers
modeled on langgraph-api 0.15.4: pinned send/stream on all three routes, aliases,
the 1.0 opt-out, unchanged genuine v1 agents, PATCH taking effect on both
workers, persistence via /v1/agents, the localhost retry, invalid values,
upstream errors, the dialect hint, and the completion bridge stripping the key.
Merge base 5f969982fc: 33 red, 7 green. Head ba709d3de5: 40/40 green twice.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(focus): add ternary destination for FOCUS export
Add a `ternary` FOCUS export destination (plus a `ternary` callback) so a LiteLLM proxy can push FOCUS-formatted spend to Ternary with `callbacks: ["ternary"]` -- no separate export job
- FocusTernaryDestination: a pure sink modeled on the sibling vantage/mavvrik/gcs destinations; POSTs the FOCUS CSV to the Ternary cost-ingestion endpoint, chunking an oversized backfill (csv-module repartition, quoted-newline safe; oversized single row raises rather than silently dropping) with a stable X-Ternary-Upload-Id + 0-based index/total so the receiver stages-then-swaps. connection_id is validated + URL-encoded; base_url must be HTTPS (loopback exempt for local); the upload has a bounded timeout; it aborts on the first failed chunk so nothing partial lands
- TernaryLogger: presets the FOCUS pipeline (CSV) and enriches the FOCUS Tags JSON with the per-request token counts the shared transformer drops (Ternary weights allocation by tokens); the enrichment is confined to this path and only adds keys. Cadence is daily (default) or interval (validation loops, an explicit interval-seconds required); the export window is day-aligned and drops days before the window start so a whole-day-replace receiver never truncates
- Registration mirrors vantage/mavvrik: ternary callback literal, logging dispatch, scheduler job, and CustomLoggerRegistry entry -- no shared-code changes
- Tests: mocked unit tests only (no live calls), 100% line coverage of both new modules
* fix(focus): warn instead of silently dropping token tags on mismatch
_merge_token_tags returned the frame unchanged when the transformed data
had no `Tags` column or its row count disagreed with the source, so every
token count was dropped with no signal -- the receiver then lands NULLs
that read as "this gateway reports no tokens". Emit a warning on each
degenerate path (mirroring _drop_days_before), and cover both with caplog
tests asserting the warning and the unchanged return.
* feat(focus): upload one UTC day per Ternary signed-URL PUT
Replace the multipart POST with chunking by a per-day signed-URL upload,
matching the Mavvrik destination's pattern. The logger partitions the raw
usage rows on `date` (dropping unparseable dates with a warning), then
transforms, enriches and delivers each day separately; one day's failure
is logged and the remaining days still upload, with a single error raised
at the end. The destination requests a signed URL for the day, validates it
is an HTTPS GCS URL, and PUTs the CSV with exactly the signed headers.
Removes the 30 MiB / 100k-row chunking, the X-Ternary-* upload headers and
the .partN filenames; a re-sent day overwrites its object, so retries stay
idempotent without them.
* fix(focus): re-send the last two whole days so late-flushed spend lands
LiteLLM writes DailyUserSpend in batches, so spend dated yesterday can be
flushed after the 00:05 UTC daily export. That export only covered the
previous UTC midnight onward, and the next day's export moved past it, so
the tail of each day was never sent. Start the window at UTC midnight two
days back: the following run re-sends yesterday whole, overwriting its
per-day object with the complete day.
* fix(focus): lazy-load the Ternary engine, fetch whole days, keep ternary out of the generic factory
Every proxy imported polars at boot through the logger registry, so a proxy without polars failed to start even with no FOCUS callback. The polars-backed engine now lives in its own module and loads on the first export
A non-UTC Postgres session timezone shifted the naive updated_at filter, so the oldest day in each run lost rows and Ternary's whole-day replace made the loss permanent. The Ternary read now starts one day early with no upper bound and keeps rows by their own UTC date
callbacks: ["focus"] with FOCUS_PROVIDER=ternary uploaded a parquet blob of one interval over a whole-day CSV file. The generic factory no longer builds a Ternary destination and the Ternary engine injects its own
* fix(focus): keep only the days a Ternary export window covers
Dropping the updated_at upper bound let a bounded export_usage_data window upload every day through today. Rows are now kept by their UTC date between the window's first day and the day before its end instant
* refactor(focus): leave the shared FOCUS export engine as on main
The Ternary engine sets the base engine's fields itself and passes its own
destination, so focus/export_engine.py and its test match main again.
---------
Co-authored-by: Aaron Sproul <aaron@ternary.app>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(guardrails): add Wingback guardrail integration
Mirrored from #38685 at d73b66544d (net diff against its merge base, applied on main)
* fix(guardrails): dispatch Wingback through the unified guardrail
The proxy only runs a unified guardrail when apply_guardrail is defined on
the guardrail's own class, so WingbackGuardrail now binds the parent's
decorated method in its class body. Chat, messages and responses pre, post
and during call checks reach Wingback again, and each check logs once.
Also types the initializer and config model for the lint gates and uses
monkeypatch.setenv in the unit test
* chore(guardrails): drop the Wingback dispatch comment
The unit and integration hook tests already fail when the apply_guardrail binding is removed, so the comment is redundant under the repo comment rule
---------
Co-authored-by: bsk-wb <bharath@wingback.ai>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(cost): add BridgeParityTestCase with a gpt-6-sol /v1/messages to Responses pair
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): rename BridgeParityTestCase to BridgeCostParityTestCase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): collect BRIDGE_COST_PARITY only from modules that define one
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): bill the regional row for inference-profile ARNs that embed a cross-region prefix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(bedrock): keep the ARN regional-row regression tests to the touched lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): apply the anthropic fast multiplier to cache cost breakdown
Anthropic fast mode billed totals at provider_specific_entry.fast while the
spend log cost_breakdown kept cache_read_cost and cache_creation_cost at 1x.
Move the speed lookup into a shared helper, multiply it into the billed token
rates used by get_token_type_cost_breakdown, and have the Anthropic calculator
read the same helper so totals and breakdown cannot drift apart.
Production change carried from #44074.
Co-authored-by: kerry <kerry@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost_calculator): narrow the provider entry instead of defaulting to a dict literal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* test(responses): pin bridged commentary plus final answer as one chat choice
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): pin reasoning, two-message and two-tool-call bridge shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): number bridged chat tool calls 0, 1 instead of all 0
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): fold bridged Responses output into one chat choice
A Chat Completions request bridged to /responses got one choice per
Responses message item, so a reply shaped [message, message,
function_call] put the tool call on choices[1] and clients reading
only choices[0] never saw it. Fold every message text, annotation,
reasoning item and tool call into choices[0]
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): drop mutable annotations and the extra cast in the choice fold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): remove a cast-ok marker that suppressed nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): join folded message texts with a newline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(responses): join folded message texts with no separator so stream and non-stream match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: ruff format otel excluded_services changes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: use raw regex for excluded_services match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): reject non-string destination callback vars
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): harden destination callback var validation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): compare parsed values in callback var conflict check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(sagemaker): inference components and OpenAI-compatible embeddings
Send model_id on a SageMaker embedding deployment as the invoke call's
InferenceComponentName, sync and async, and keep it out of the request body.
Add the sagemaker/openai/<endpoint> route, which sends the OpenAI embeddings
request shape ({"input": [...]} plus extra_body fields) and reads
data[].embedding and usage from the response for containers that serve the
OpenAI embeddings API. Plain sagemaker/<endpoint> deployments keep the
Hugging Face shape
* refactor(sagemaker): type validate_environment messages and cover base64 embeddings
* fix(sagemaker): name model_id in inference component errors and reject a non-string model_id on embeddings
* fix(sagemaker): fall back to the error text when SageMaker sends a null message
A ClientError whose Error.Message is null (SageMaker's InternalFailure shape) reached the
inference component hint as None and crashed with a TypeError; the message now falls back
to the error's own text on the chat and embedding paths
* test(sagemaker): audit matrix for inference-component embeddings
* fix(sagemaker): read the error status and text through one shape-checked helper
botocore's ConnectionClosedError carries response=None, so the except blocks that
read e.response.get(...) crashed with an AttributeError and the proxy answered 400
"Invalid request format" for a dropped connection. client_error_details validates
the botocore error shape with pydantic and falls back to 500 plus the error's own
text for any other shape (a missing response, an httpx Response, a plain exception).
* fix(sagemaker): drop the inference component hint from chat and derive the SDK wire expectation from the request
The chat except blocks never see SageMaker's text: HTTPHandler.post raises from
raise_for_status before the handler's own SagemakerError line, so the hint check ran
on httpx's generic "Client error '400 Bad Request' for url" line and could not match on
any chat path. The hint now names litellm.embedding only and the chat blocks keep the
status and text they had, without the dead check. The two OpenAI SDK wire tests pinned
the SDK's default encoding_format; they now expect whatever the SDK sent.
* refactor(sagemaker): meet main's per-rule lint and type ceilings
Main started gating every lint rule on its merge-base count and added LIT015
(frozen pydantic models) after this branch was cut, so the merge with main
pushed LIT001, LIT011, LIT015, BLE001 and reportArgumentType past their
ceilings. The response models are frozen with tuple fields, the OpenAI
embedding response is built with model_copy instead of writing into the
caller's object, the invoke_endpoint arguments are a TypedDict, and
optional_params and extra_body are validated into Mapping[str, object] before
they are read
* refactor(sagemaker): pick the embedding config in one place and stop pinning upstream error text
The openai/, Voyage and Cohere checks lived in both get_model_config and
get_supported_openai_params, so they now share one lookup. Tests compare error
messages against the error's own text, or against botocore's and json's own
wording, instead of copying that wording, and assert whole messages where the
text is ours
* fix(caching): key direct calls on model_id so inference components don't share cached embeddings
* test(sagemaker): name the fake-boto3 SDK test class for what it covers
* test(sagemaker): kill and stop chaos workers through the owned-proxy harness
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
A caller hitting /v1/messages with a cache_control mark and an OpenAI
gpt-5.6 or newer deployment behind it lost the mark when the request was
rebuilt as a Responses request, so OpenAI never cached at the boundary the
caller chose. The bridge now rewrites each marked system, user and
tool_result block to prompt_cache_breakpoint on the same block when the
deployment reads breakpoints, keeps the system prompt as a developer
message so its mark survives, and still strips the mark for deployments
that do not read it
The hook's gate becomes public so the bridge and the hook share one
answer for which deployments read breakpoints, and the one-block swap
lives next to the existing breakpoint helper in prompt_templates
* feat(providers): add CoralBricks as a native provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): harden cost map consistency tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): type cost helpers, read cost maps from disk, cover streaming spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): drop Final on loop-rebound names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): add the provider audit cells, the Add Model browser cell and the README row
* test(coralbricks): derive UI spend from model info, type browser rows, always release held upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): release held upstream before the wire server joins its threads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): number forwarded chats with itertools.count instead of a mutable list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(coralbricks): drop retired glm-5.3-flash-fp4 and test against deepseek-v4.1-flash-fast-fp4
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(coralbricks): pin retries, await the respawned worker, read the UI rate from the catalog
The retry cells now set num_retries on the deployment, since the CI rig posts
router_settings.num_retries 0 after readiness and the router default of 2 no
longer applies there. The SIGKILL chaos cell waits for the respawned worker's
startup before the owned proxy is stopped and derives its timeout from the
harness budget. The Add Model spec waits for the catalog the form loads, reads
the expected rate from it, and finds the deployment through /v1/model/info,
which does not paginate
* fix(coralbricks): price the current model names and the new DeepSeek input rate
CoralBricks renamed glm-5.3-fp4 to glm-5.3-fast and deepseek-v4.1-flash-fast-fp4
to deepseek-v4.1-flash-fast (the fp4 names stay as deprecated aliases), and cut
DeepSeek V4.1 Flash input from $0.30 to $0.01 per 1M on 2026-10-08. Rates read
from https://www.coralbricks.ai/api/public/models on 2026-10-10; a live cache
write billed 6014 tokens at 9e-08 each, so the cache-write rate stays input
plus cacheWritePerM.
The test config builders now build new dicts instead of editing the parsed
YAML, and the cost-map test locals carry Final.
* fix(coralbricks): keep pricing for the deprecated fp4 slugs CoralBricks still serves
* fix(coralbricks): list the catalog rows for wildcard deployments and derive the test's input limit from the row
* fix(coralbricks): price the deprecated glm-5.3 and deepseek-v4.1-flash slugs
---------
Co-authored-by: Mubashir Osmani <mubashir@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): wait for SIGSTOP to take effect before handing back a suspended proxy process
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ci): resume a process whose SIGSTOP never lands before failing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(dd_span_tagger): emit litellm_user_email span tag for JWT-authenticated requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(dd_span_tagger): use dotted litellm.user_email tag for consistency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(model_hub): surface model_info.description in model group info and Model Hub UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(router): aggregate model group description without in-place mutation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(health): skip background health check DB writes when the latest-row read fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock_mantle): map the OpenAI-shaped context overflow envelope too
Mantle reports context overflow in two envelopes. #37862 covered the
structured `validation_error` one carrying token counts
("prompt tokens (N) exceed model maximum (M)"). Mantle also returns the
OpenAI-shaped body:
{"error":{"code":"context_length_exceeded","message":"Your input
exceeds the context window of this model. ...","param":"input",
"type":"invalid_request_error"}}
That passes the existing `invalid_request_error` gate but matches neither
the token-count pattern nor `is_error_str_context_window_exceeded`'s
substring list, so it fell through to a generic BadRequestError. Clients
such as Claude Code only reactive-compact on a recognised overflow
phrase, so an overflowing request surfaced as an ordinary error with no
recovery.
Return the same normalised "prompt is too long: ..." message for this
envelope when it carries no counts, keeping the numeric wording when the
provider supplies them.
* fix(bedrock_mantle): map the streamed context overflow and keep its 400 on /v1/messages streams
* fix(anthropic_adapter): keep only the context overflow's 400 on streamed /v1/messages so other provider 4xx still fall back
* test(anthropic_adapter): wrap a parametrize line over the 120 char limit
* test(integration): audit Bedrock Mantle context overflow mapping across routes, clients, and fallbacks
* test(integration): move the streamed /v1/messages overflow spend row into a BUG-skipped test
* test(integration): keep every chaos cell in the burst and skip only the missing spend row
* fix(bedrock_mantle): answer 400 on an in-stream context overflow
The Mantle overflow mapper built its ContextWindowExceededError on the
original exception's response, so an overflow that Mantle reports inside
a 200 stream (the chat-mode token-count envelope, which the OpenAI chunk
parser wraps as a 500) kept that 500 and the streamed /v1/messages route
answered HTTP 500 api_error instead of 400 invalid_request_error. The
mapper now builds the error on a 400 response that keeps the original
headers and request, and the mapping test covers an in-stream 500 and 503.
The integration suites pin the streamed /v1/messages overflow to the
pre-content 400 route under a retry budget and to the message_start plus
invalid_request_error event without one, drop the chaos carve-out that
expected no spend row, and give the fallbacks file a timeout mark sized to
the proxy's graceful stop.
* fix(bedrock_mantle): keep the provider's 400 body and match the overflow code only as error.code
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
