* fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream
* test(websearch_interception): type the conversion helpers and drive the replay test through mock_response
* test(websearch_interception): pin the replay test's values as Final and record follow-ups immutably
* fix(proxy): strip every agentic-loop control key from client bodies, the web search keys included
* fix(interception): move stream_options off forced non-stream calls for code interpreter and headroom, and keep adapters from replaying nested calls
* test: annotate new interception test locals with Final
* fix(responses): drop params a deployment hook removed on sync Responses calls
* test: drive the sync Responses stream_options regression through a mock transport and real Azure config
* test: type the agentic loop recorder and capture the sent request on an object
* test(integration): cover interception stream_options on chat, responses, messages and the SDK
* test(integration): assert the follow-up error text and release a held burst on failure
* test(integration): pin the follow-up failure contract and wait for the replacement worker before teardown
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(realtime): only inject response.create when a transcript guardrail suppressed VAD auto-response
The proxy sent `response.create` to the backend after every
`conversation.item.input_audio_transcription.completed`, whether or not a
`realtime_input_transcription` guardrail had disabled the backend's server-VAD
auto-response. Without such a guardrail the backend already created the turn's
response, so the injected one was a duplicate and the backend answered every
user turn with `conversation_already_has_active_response`.
Gate the re-trigger on the same predicate as the suppress side
(`_has_audio_transcription_guardrails`), via `_proxy_drives_turns()`, in both
the raw-backend and the provider_config paths. Guardrails still run on every
transcript; only the injection is gated. Transcription-only sessions keep
skipping the injection as before.
Flip the unit test that asserted the old behaviour and add a provider-path
regression test.
Fixes#31726
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(realtime): assert guardrail-gated response.create on the provider_config path
Mirror of test_realtime_guardrail_allows_clean_transcript for backends that go through
provider_config. With a realtime_input_transcription guardrail the proxy disabled the
backend's auto-response, so a clean transcript must trigger exactly one response.create.
Locks in the other half of the suppress/re-trigger pair introduced for #31726.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* test(realtime): type the passthrough provider_config helper's frame argument
* test(realtime): expect no proxy response.create on an unguarded voice session in the integration suite
* test(realtime): audit cells for client-owned turns without a transcript guardrail
* test(realtime): wait for the killed worker to exit, not for its parent to reap it
* test(realtime): pin each worker-kill session's outcome to the worker that accepted it
* test(realtime): drive the provider_config path through a concrete passthrough realtime config
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility
* fix: preserve HTTPX compatibility across OpenAI SDK versions
* refactor: trim OpenAI SDK compatibility patch
* refactor: simplify SDK client defaults
* test: cover SDK compatibility across provider routes
* fix: use shared names for SDK transport helpers
* fix: address SDK compatibility review feedback
* test(openai): send requests through the SDK API factory clients
* fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2
* ci(base-sdk): count httpx2 as base-only when a base dependency brings it in
OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure
* test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively
* test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10
---------
Co-authored-by: Marcus Wood <marcuswood@openai.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): switch /v1/decisions to the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(decisions): build the usage the OpenAI response schema now requires
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lens): ask signal questions through the OpenAI Decisions schema
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): refuse safety_identifier for System One providers unless drop_params
* feat(decisions): refuse safety_identifier on providers that cannot take it unless drop_params drops it
* test(decisions): cover hosted_vllm in the safety_identifier and per-provider wire tests
* fix(decisions): check a System One request's safety_identifier against the provider too
* fix(decisions): drop a non-string safety_identifier under drop_params
A malformed safety_identifier now follows the drop_params convention in both body shapes: it answers 400 without drop_params and is dropped before the provider call with it. A string identifier on OpenAI stays on the wire either way.
* fix(decisions): let /v1/decisions drop a non-string safety_identifier under drop_params
The route checked the whole body before routing, so a non-string
safety_identifier answered 400 even when the deployment or the body set
drop_params. The route now leaves that field to the Decisions call, which
knows every drop_params source
* refactor(decisions): return the unsupported safety_identifier as a value and raise it in _prepare_call
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(gemini): map OpenAI voice names to Gemini prebuilt TTS voices
* test(gemini): type the voice mapping test helpers
* fix(gemini): leave nova out of the voice mapping since Gemini accepts it as given
* test(gemini): add integration cells for the OpenAI voice mapping on Gemini and Vertex TTS
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(databricks): keep #/$defs refs in json_schema response_format for non-Claude models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): type the json_schema response_format override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): integration cells for json_schema $defs refs across endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): split stacked loop in streaming json_schema cell
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(databricks): mark BaseConfig dict contract on json_schema override
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(databricks): cover json_schema refs on every endpoint, edge refs, sad paths and an outage burst
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): cover tool_reference tool results on bedrock converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep tool_reference results as text on converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format tool reference parser
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover bedrock converse tool_result of only tool_reference blocks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): tag converse tool_reference case with e2e metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): avoid unchecked cast in converse tool_reference mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep tool_reference results as text on responses paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): tighten tool_reference mapping on responses paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(completion-extras): expect tool_reference names in responses input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(bedrock): drop responses converter changes, keep converse tool_reference fix only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(proxy): keep the mapped status on assistants and threads route errors
* test(proxy): type and annotate the assistants and threads route error tests
* fix(proxy): redact internal details from assistants and threads route error messages
* test(proxy): integration cells for assistants route error mapping
* test(proxy): drop malformed JSON cases that never reach the assistants route handler
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(azure_ai): support Microsoft-Decision-1 on the decisions API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): strip project path and API suffix together in decisions base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(agents): ban semicolons and colons in human-facing text
Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "."
* docs(agents): rewrite semicolons in nested AGENTS.md files
* docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines
* docs(agents): point at the PR template path that exists
* docs(agents): restore the generated Next.js block in the dashboard AGENTS.md
* feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider
* fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/
* fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor
* fix(ui): name every OSS classifier provider in the classifier radio copy
* feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry
* fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry
* fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router
A /model/new for an auto router lands on one replica. A request routed to
that auto router on a sibling replica missed it, the model read-through
loaded only the auto router's own row, and the pre-routing hook then picked
a tier the sibling had not loaded, so the caller got a 400 "no healthy
deployments for <tier>" until the next periodic reload. An auto router miss
now runs the full add_deployment reconcile, tiers included. Found by the
audit's two-worker rig on this PR's routing cells; CircleCI's one-worker
proxy never opens the window.
* test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier
Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing
cells for the complexity router's databricks classifier on all three
endpoints with the Postgres rows read back, management cells for the
classifier's spend writes and for the evaluation-mode health check of a
Databricks serving endpoint.
* feat(policy-engine): stream detect-only post_call pipeline steps live when buffering is off
* fix(policy-engine): scan live pipeline streams a provider error cuts short and discard rewrites per step
A live detect-only pipeline now scans the chunks the client received when the provider fails mid-stream, then re-raises the error. Each step's rewrite is discarded right after the step, so later steps scan the text the client actually received.
* fix(proxy): keep a guardrail verdict recorded after a stream failure on the failure spend row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* chore(github_copilot): sync model catalog with Copilot's served models
* test(github_copilot): assert catalog invariants instead of pinning vendor facts
* fix(github_copilot): keep thinking capabilities on served Claude catalog rows
* fix(github_copilot): keep supports_reasoning on catalog rows that shadow the family fallback
* fix(model-catalog): preserve merged Copilot Haiku metadata
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on
* fix(bedrock): sign the listing query RFC 3986 style and raise BedrockError on a failed listing
* test(utils): keep test_utils.py at main's formatting, only the Bedrock discovery test is new
* fix(bedrock): chain BedrockModelInfo's constructor so the passthrough config keeps its event parser
* fix(bedrock): list discovered ids without the provider prefix so partial wildcards filter
get_valid_models returns bare vendor ids for every other provider and for the static bedrock catalog, and the proxy's wildcard expansion adds the provider prefix itself. The lister prefixed its ids, so a partial wildcard like bedrock/anthropic.* never matched the proxy's filter and listed uncallable bedrock/anthropic.bedrock/<id> entries
* test(bedrock): integration cells for wildcard discovery through the proxy and the SDK
* fix(proxy): keep a partial wildcard a filter when its deployment repeats the prefix
The wildcard expansion guessed filter-or-alias by whether any provider id carried the prefix. A deployment such as model_name bedrock/anthropic.* over model bedrock/anthropic.* can only ever route names that share the prefix, so when the account has no on-demand anthropic.* id (every Anthropic model behind an inference profile) the guess fell into the alias branch and listed bedrock/anthropic.<every id>, none of them callable. A deployment whose model repeats the suffix now always filters, and lists nothing when nothing matches
* fix(proxy): prefix every discovered id under a custom wildcard prefix that starts a vendor id
* test(bedrock): Final and read-only annotations in the wildcard discovery tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): route Grok 4.7 tools with reasoning to native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): mint unique tool call ids on native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send a blank text block for Converse tool results with no supported content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): default Grok on Bedrock to native Chat Completions and leave Converse unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): gate tool call id minting to Grok on native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): check the Grok tool call id gate at both call sites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): remint positional tool call ids on native Chat Completions for every model
AWS returned call_0, call_1 for GPT 5.6 as well as Grok on 2026-10-09 (and unique ids for GPT 5.6 earlier the same day), so the id format is not fixed per model. Drop the Grok-only gate and remint any call_<digits> id on the route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop stop sequences for Grok instead of forwarding them to a 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(teams): let proxy admins grant team admins the right to raise their team budget
Adds a raise_max_budget entry to team_admin_editable_team_fields. With max_budget alone a team admin can still only keep or lower the team budget. Adding raise_max_budget lets them raise it, capped by the organization's max_budget when the team belongs to one, while removing the budget stays with proxy admins. The entry is rejected without max_budget, /team/info reports may_raise_max_budget to the caller, and the Admin UI nests the new checkbox under Max Budget with a tooltip and shows the team admin a hint under the budget field.
* fix(ui): shorten the raise_max_budget tooltip and link it to the docs
* fix(teams): pin the organization in the guarded team budget write
A granted raise is checked against the team's organization at read time, so the write now also requires organization_id to be unchanged. A concurrent move into a budgeted org returns 409 instead of landing an uncapped raise. Also types the new test helpers.
* test(teams): type the team row store methods and the touched race test arguments
* test(teams): annotate the last fixture and parametrize arguments in the raise_max_budget tests
* test(integration): poll the partition lock witness off the event loop
The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting
* test(e2e): retry the reasoning turn until the model emits a reasoning item
Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion
* test(e2e): retry a Bedrock Converse model once when it reports no cache tokens
Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged
* test(e2e): poll until a deleted stored response stops being retrievable
Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx
* test(e2e): give the Together prompt cache five primed attempts
Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed
* test(e2e): allow 30s for the idle RSS reading at session start
A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged
* test(e2e): scope the MCP submission check to its own card
The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card
* test(router): wait for the primary success record before counting it
The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs
* test(proxy): give the spend-log cleanup run a 1s budget
A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves
* test(autorouter): release the slow token count at 2.05s instead of 2.3s
The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack
* test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning
grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting
* test(integration): give the two worker-kill chaos tests the owned-proxy time budget
Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests
* test(integration): compare regional image responses without their created timestamp
Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared
* test(integration): count only the S3 logger's flush task
The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels
* test(integration): send guardrail timeout probes eight at a time
All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits
* test(integration): count a killed Arize chaos worker as gone once it is a zombie
psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget
* test(integration): send the Typesafe connection test its real model and endpoint
The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly
* test(integration): keep DB-stored models out of the usage-routing Redis read test
The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off
* test(vertex_ai): prove the batch upload streams by laziness instead of peak memory
The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests
* test(integration): keep the alias test_connection call as a known bug
* test: type the counting mapper and the Bedrock rerun helper
* test: annotate the new test locals as Final and build them as tuples
* test(vertex_ai): prove pull-driven transforms with the garbage tail alone
* feat(decisions): add hosted_vllm provider
Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move the hosted_vllm placeholder key to constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): never forward the caller's LiteLLM key to MCP servers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): tidy caller-key scrub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): avoid master-key scanner false positive
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): match admission when scrubbing the caller key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): drop header copy churn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): normalize bearer variants in caller-key match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): inject admission header name and cover provider keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): exercise real REST header extraction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover caller-key scrub on the real REST path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): exclude gateway admission header from upstream forwarding
* fix(mcp): skip empty admission headers when scrubbing keys
* fix(mcp): use scrubbed server credentials for auth probes
* fix(mcp): match probe credential precedence to upstream calls
* test(mcp): simplify probe header assertion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(credentials): add credential_alias and make credential_name immutable on PATCH
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): query credential rows by role to stay under the no-node-access budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): expect credential_alias in load_credential_list dump
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drive the credential SearchSelect by placeholder and option roles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass the decrypted CredentialItem to update_db_credential during master key rotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): trim credential_alias in CredentialModal so whitespace-only input clears the alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(credentials): replace credential_alias with display_name and reject edits to config credentials
credential_name stays the immutable reference key. display_name is a nullable, trimmed, max 255
character label set on POST or PATCH (omit keeps it, null clears it, blank is a 400). Reads return
display_name plus an in-memory source tag (db or config), and PATCH or DELETE on a config-defined
credential answers 400 instead of a misleading 404
* feat(ui): show credential display names everywhere and lock config credentials
The credentials table shows the display name with the credential name beneath it, badges config
credentials and disables their edit and delete actions. The edit modal adds an editable display
name next to the read-only credential name and only sends it when it changed. Every credential
picker and reference (add model, model info, models table, vector store form and info) shows the
label while still submitting credential_name
* feat(cli): set display names on credentials and show them in the list
* refactor(credentials): keep the new credential types inside the lint gate budgets
* refactor(cli): print credential command JSON through one helper
* fix(credentials): treat an empty credential_name on PATCH as omitted and pin the 404 for vanished rows
A blank credential_name never renamed anything, so PATCH accepts it again instead of answering 400. New tests pin a trimmed display_name on PATCH, the repository carrying display_name, and a 404 (not the config-owned 400) for a DB credential another worker already deleted. The hydration helper no longer copies display_name, since none of its callers read it, and the CLI update passes display_name straight through because the exactly-one check already makes it None when clearing.
* fix(ui): keep a model's credential when its picker text is emptied and skip the admin-only list for other roles
Clearing the search text in the model edit form's credential picker used to submit null, silently detaching the model's credential on save. None is now the only way to clear it. The models table fetches /credentials only for proxy admins, since everyone else got a 403 on each page load and falls back to the raw name anyway. Also fixes a type error in the re-use credential dialog and adds tests for the display name surviving a provider switch, the 255 character limit, the request payload, and the vector store None choice.
* fix(client): percent-encode the credential name when updating its display name
A name with ? or # was cut short in the URL, so the PATCH landed on a different credential or 404'd.
* fix(credentials): answer 405 with Allow: GET for PATCH and DELETE on config credentials
A config-defined credential exists (GET returns it) but is read-only through the API, which is what 405 Method Not Allowed means. 400 described a malformed request, and 404 would claim the credential does not exist.
* fix(proxy): ignore a display_name set on config credential_list entries so non-string values cannot fail boot
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): count a leading system run through count_tokens' system parameter
Anthropic's count_tokens rejects role "system" at the head of messages, so
/v1/responses/input_tokens with instructions, and /v1/messages/count_tokens
with a system-role message, fell back to the local tokenizer. The shared
Anthropic count_tokens transformation now lifts the leading run of system
messages into the top-level system parameter, the way the chat path sends
it, after any system the caller set. Anthropic direct, Azure AI Anthropic,
and Bedrock Mantle share that transformation.
* test(integration): cover the count_tokens leading-system lift across Anthropic, Azure AI and Bedrock Mantle
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(token_counter): price a base64 PDF document per page instead of as one image
A `document` or `file` block carrying inline PDF bytes was priced like a single
image (85 tokens), so when a provider's count-tokens endpoint rejected the
model (Bedrock Opus) the local fallback answered 116 for a 12-page PDF the
provider then billed at 35941 input tokens. The counter now reads the PDF with
pypdf and prices each page as its extracted text plus the image Anthropic
renders it to (1568 px long edge, 1.15 MP, 750 pixels per token), falling back
to the old image pricing when pypdf is missing or the bytes are not a readable PDF.
* refactor(token_counter): count PDF pages as they are read
Sum each page's text and rendered-image tokens straight from the pypdf
reader instead of materializing a page list first, keep the fallback
to image pricing atomic when a page cannot be read, and annotate the
new tests' locals as Final
* test(integration): audit cells for page-priced PDF documents in count_tokens, pre-call checks and spend
* test(integration): release the held peer when the concurrent budget wait times out
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt
An order-based or configured fallback hop replayed the failed provider's
encrypted reasoning items to the next deployment, which answered 400
(Bedrock Mantle: invalid encrypted reasoning; OpenAI:
invalid_encrypted_content), so every multi-turn Responses fallback for
Codex-style clients failed. The hop now drops the encrypted reasoning
its target cannot decrypt and keeps each item's readable summary. With
encrypted_content_affinity on, the pin narrows to the hop's target
order instead of emptying it, so the hop reaches the next order
instead of failing with no deployments available.
* fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt
Unmarked encrypted reasoning on a hop is attributed to the deployment that just
failed, read from the retry breadcrumb, so a hop to a deployment on the same
api_base and api_key keeps it and a cross-provider hop still drops it. The hop
tests script the upstream at the httpx boundary instead of doubling the handler,
and the router coverage script lists the two hop helpers with their tests
* fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot
* test(router): use a real Router without the origin deployment in the hop strip test
* test(router): cover the hop strip when no failed deployment is known
* fix(proxy): drop the router's fallback hop state keys from the client body
* fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys
* test(integration): audit cells for the fallback hop encrypted reasoning strip
Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): support logging_only mode for the Akto guardrail
* test(guardrails): wait for the spend row before reading Akto calls and cover mixed modes
* test(guardrails): cover Akto logging_only on MCP tool calls
* test(guardrails): type the Akto logging_only unit tests and inject the HTTP handler
* test(guardrails): cover failing Akto replies, provider failures and a mixed outage burst under logging_only
* test(guardrails): cover an unreachable Akto with fail_open under logging_only
#44800 re-raises the provider's own error from the chat adapter, so a bridged /v1/messages stream's error frame now reads litellm.RateLimitError without the litellm.MidStreamFallbackError: prefix that #44989's tests pinned shortly before #44800 merged. The three tests now expect the provider error text once and no sentinel, matching the chat_limited case and every other assertion in the two files
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: replace 61 live logging and otel tests with offline unit and integration coverage
* test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks
Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts.
* test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request
* test: isolate the prometheus latency test on its own deployment so counts do not depend on order
* test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only
* test: drain the redis test's logging and scope otel span checks to the test's own trace
* test: drive the periodic slack flush without a wall-clock interval
* test: scope router hook events to the test, script the db clock, split a nested comprehension
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): shed OpenInference output tool calls individually under the span attribute budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): avoid mutation in Arize OTel v2 integration helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): audit Arize OTel v2 OpenInference spans across endpoints, modes and chaos
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): wait for each exported span before the next request in Arize OTel v2 audit tests
* test(otel): make Arize OTel v2 audit absence and outage checks deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): collect Arize OTel v2 outage spans through an in-order sentinel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): add skipped BUG cells for pre-existing Arize OTel v2 gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): repair Arize OTel v2 regressions from #43698 (linear fit, metadata slot, repr tool args) (#44488)
* fix(otel): keep Arize OTel v2 regressions in check — O(n) fit, metadata slot, repr tool args
Three regressions from #43698's OpenInference tool-call/metadata emission:
1. Metadata evicted indexed message attributes: the new `metadata` key
competed for the 128-attribute span budget, and the fit sheds whole
message groups BEFORE `span.set_attribute`, so the SDK's dropped
counter stayed 0 — an invisible eviction (live A/B: input-message
attributes 86 -> 84). Two-part fix: the fit pins `metadata` behind
every message group (it sheds only once all indexed messages are
gone), and the span budget no longer charges pre-set attributes the
mappers overwrite in place — a boundary-opened LLM span already
carries keys like `gen_ai.request.model`, so the old accounting
reserved slots the fit could never spend. Live: 86 input-message
attributes with the metadata attribute riding alongside.
2. Quadratic shed on long prompts: `_message_shed_groups` rescanned the
full group map once per message (measured on a real acompletion:
0.032/0.128/0.478/1.910s at 1000/2000/4000/8000 messages vs
0.007/0.010/0.022/0.029s at base). Index the tool-call groups once by
(family, message index): the fit is linear again (0.008/0.007/0.014/
0.031s, same rig).
3. Malformed Python tool arguments lost the whole span: provider
adapters and `model_construct` responses hand over raw objects, and
`json.dumps` raises on tuple-keyed dicts (TypeError) and cycles
(ValueError) before the span is exported. Serialize with a repr
fallback; both cases now export with a readable arguments attribute.
The attribute budget change affects every boundary-opened LLM-call span
(strictly more attributes retained, never fewer); the mapper changes
only touch the OpenInference vocabulary.
* refactor(otel): build the tool-call group index in one shot
Review follow-up: the dict.setdefault/append seeding in
_tool_call_groups_by_message violated the no-mutation coding convention
(AGENTS.md: build values in one shot with comprehensions or generators
wrapped in tuple()/MappingProxyType()). Rebuild it as a sorted groupby
comprehension; randomized parity harness confirms the shed order is
byte-identical to the seeded version (400 trials).
Also pin the overflow corner Greptile asked about: a pre-set
indexed-message key the fit sheds keeps its earlier value in place, so
the span total can never exceed the SDK limit (new emitter test).
* fix(otel): key the groupby with an explicit tuple to keep basedpyright at budget
The slice-keyed groupby (group[:2]) widened the key to tuple[str | int],
adding one reportGeneralTypeIssues over the codebase ceiling. Key by the
explicit (family, message index) pair instead; shed order unchanged
(300-trial randomized parity harness).
* fix(otel): read pre-set span keys through a helper typed for both runtime shapes
The SDK annotates ReadableSpan.attributes as a Mapping, but an ended span
hands back a tuple of pairs, so the inline isinstance branch narrowed to
Never and pushed reportGeneralTypeIssues one over the codebase ceiling.
Extract _carried_keys with the runtime union declared on the parameter;
behavior unchanged.
* test(otel): drop the wall-clock bound from the long-prompt attribute fit test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): annotate to_openai_dict as Mapping after the rebase onto main
Main's type-discipline budget tightened since the branch point; the plain
dict return annotation is the one violation the rebased branch adds.
Callers only serialize the result, so the read-only view is accurate.
* chore(otel): drop the restating docstring from to_openai_dict
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): accept team-scoped models by their public name on POST /fallback
create_fallback validated the primary and fallback models against the
router's stored model names only, so a model created through POST
/model/new with model_info.team_id was rejected with a 404 unless the
caller used the generated model_name_<team_id>_<uuid> name. The
endpoint now also accepts the team public model names, which request
time fallback matching already keys on, and lists both kinds of names
in the 404's available_models.
* fix(proxy): read fallback rules fresh and clear the config cache after a fallback write
A second POST or DELETE /fallback within the 60 s config cache TTL started from
a cached copy of router_settings and dropped every rule stored since that copy
was taken, by any instance. Both endpoints now evict the cached row before the
read and invalidate it after the upsert
* test(proxy): type the fallback endpoint tests and prove a team request fails over by public name
* fix(proxy): keep fallback writes working when the Redis config cache is down and type the stored settings read
* fix(proxy): keep the non-standard fallback shapes the router accepts on writes and resolve the config cache at call time
* test(proxy): mark the config cache outage test's result as Final
* fix(proxy): replace a same-key fallback rule in place and read a null rule list as empty
* test(proxy): let the stored router settings fixture carry a null rule list
* test(proxy): audit cells for fallback rules by team public name
* test(proxy): delete the fallback rules the audit cells save
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs
Adds general_settings.spend_logs_metadata_fields with mutually exclusive include and exclude lists. The filter runs on a copy of the row right before it is queued for Postgres, so daily spend rollups, budgets and callbacks still see every key.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): keep excluded auto-router savings keys out of published spend log metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend_logs): cover metadata retention across endpoints, failures, cache hits, batches, runtime updates and auto-router publication
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for spend_logs_metadata_fields
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): filter metadata at DB write so guardrail usage sees full rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend_logs): poll guardrail daily metrics instead of reading once
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend_logs): drop timeout comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_logs): read spend_logs_metadata_fields through typed general settings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): run each mode only on streaming, non-streaming, or both
Add stream_scope so a rail can target streaming inference, non-streaming inference, or both per pre, during, and post mode
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor stream_scope in pipelines and dashboard types
Pipeline steps skipped the stream_scope filter, direct construction ignored mixed-case maps, and schema.d.ts was stale.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): skip unmatched stream_scope steps instead of allowing
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): keep stored stream_scope keys for modes not on screen
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): format stream_scope helpers and update fork MCP unit tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): keep hang cancellation tests from timing out during setup
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor path-defined streaming for stream_scope
Passthrough routes like Gemini streamGenerateContent decide streaming from the URL, so stamp that onto hook data before guardrails run.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): copy AnthropicModelCapabilities instead of cloning
Clippy treats clone-on-Copy as an error, which failed rust-lint on the messages request tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): trust only server stream classification
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): keep streaming marker through deepcopy
scan_raw_request snapshots copy each field, so a plain object() marker would lose identity and skip streaming-only rails.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cost-map): drop duplicate perceptron-mk1.5 row
Two main cost-map PRs both added the OpenRouter model, so the merge left a second key that CI rejects.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): expect native transcription 429 as RustUpstreamError
HTTP status errors from native routes map through route_error_to_pyerr, so the wheel SIGINT child was dying on an outdated RuntimeError check and never reached the hang probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): follow Google Interactions OpenAPI without hardcoded names
The live spec dropped CreateModelInteractionParams and renamed the item path to {interactionsId}. Misc CI failed because the compliance tests still looked those names up as literals.
* test(guardrails): reproduce stream_scope bedrock and passthrough field gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover invalid stored stream_scope reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify bedrock stream actions and keep caller is_streaming_request
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): make stream_scope_allows public and drop mutable builds
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify pass-through and Bedrock stream scopes
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden stream classification and validation
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add stream scope integration audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid mutating passthrough custom body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): run stream scope audit without enterprise license
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): keep stored scope restart cell on one worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): scope stream scope audit sink assertions to the rail under test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert logging_only scope absence behind an ordered barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert one logging_only scan per phase after the barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): set request scope on pass-through stream fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid YAML stream_scope in v1 guardrails list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: merge main into litellm_guardrail_stream_scope_fixes
Update pass-through pre-call test callbacks for main's endpoint_type argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add request paths to pass-through fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover websocket pass-through stream scope and LIT-9050 outage spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): remove unused type discipline suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): script the vertex live upstream in the websocket stream scope test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep stream marker json-serializable and restore pass-through helper names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): allow required Bedrock action re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): strip the stream marker from pass-through payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): drop stream marker by value in scans and snapshots, plain-tuple stream scope state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): render tag-scoped guardrail modes read-only in the custom code editor
A guardrail whose litellm_params.mode is the tag-scoped dict {tags, default}
crashed the Custom Code editor on open: normalizeMode wrapped the dict into
the mode array and StreamScopeFields rendered it as a React child (error #31,
whole dashboard unmounted). Treat a non-string non-array mode as no editable
modes, show formatGuardrailMode(mode) in a disabled input (read-only, matching
the guardrail info view), and keep mode/stream_scope out of the update payload
for such guardrails.
* fix(ui): resolve merge fallout in guardrails components
Deduplicate toModeArray import after the merge, and move the read-only
guardrail details block back into GuardrailReadOnlyDetails (now rendering
the shared mode/logging-only rows plus the stream-scope detail) so
guardrail_info.tsx stays under the 800-line lint budget. Guardrails UI
suite: 298 passed.
* fix(ui): drop duplicate toModeArray import reintroduced by merge
* chore(pass-through): document the deliberate in-place marker strip as mutable-ok
The clear/update on _parsed_body is load-bearing: rebinding to a fresh
mapping instead breaks 75 pass-through tests because the marker-free body
must propagate through the caller's request dict so downstream guardrail
scans and snapshots never observe the server streaming marker.
* fix(guardrails): move stream_scope after timeout in CustomGuardrail init
Inserting stream_scope before the existing timeout parameter shifted the
positional slot of timeout, so positional callers constructed with their
timeout bound to stream_scope (ValueError) and timeout silently None.
Restores the base parameter order; keyword callers are unaffected.
* fix(guardrails): typing pass for the lint gates
stream_scope leaves the declared constructor parameters (restoring the
base positional surface; keyword construction unchanged), unknown config
values crossing the new stream-scope code paths get typed locals or
cast-ok boundaries, and the passthrough payload literals are annotated.
All three lint gates pass against current main; the guardrail suites are
unchanged (236+5019 passing; one known anyio-driver failure pre-existing
on base).
* style: sort cast imports for the ruff gate
* chore(pass-through): drop unused BEDROCK_STREAMING_ACTIONS re-export
The streaming check now uses is_bedrock_streaming_endpoint; nothing in
the repo imports the name from this module.
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: gabriele <gabriele@berri.ai>
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings
* test(gemini): annotate the embedding mock helper and keep the type comment short
* fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly
* test(caching): type the partial-hit cache test and pin its clock
* fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id
* fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs
A bare object `input` answered 500 from both the caching handler and the
transformation; both now answer 400. An unresolved `files/` reference on
vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/
provider. An empty `format` passed through to the provider; it now answers 400
naming file.format. Unknown block keys (`detail`, an unknown video_metadata
key, a top-level block key) are dropped under drop_params, global or
per-request, and still answer 400 without it. The embedding cache counted
file blocks against `max_messages`, so a request with 5 or more blocks was
never cached; file blocks no longer count.
* fix(caching): answer 400 for an object embedding input on the cache lookup
* test(gemini-embeddings): annotate the new embedding tests with return types and Final
* test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final
* test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos
Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends
* test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(voyage): rebrand to VoyageAI by MongoDB, refresh model list, support both contextual input shapes
* fix(voyage): send auto-chunking params for flat contextual inputs
A flat list[str] or bare str for the contextualized embeddings endpoint is
only valid as documents with enable_auto_chunking=True and input_type=document,
or as queries with input_type=query. Default those params for non-query flat
inputs so the request matches the live API contract, while letting caller-set
values win. Nested list[list[str]] still passes through unchanged.
Tests now assert the auto-chunking params instead of only echoing inputs.
* feat(voyage): route MongoDB-issued keys to ai.mongodb.com
Mirror voyageai.util.get_default_base_url from the official SDK: a key with
the `al-` prefix is issued by MongoDB and is only valid on ai.mongodb.com,
every other key on api.voyageai.com. The choice now lives in one shared
helper that the embedding, contextual, multimodal, and rerank configs all
use for both the default base URL and the Authorization header, so the host
and the key always agree.
Also drops the voyage-4-nano and voyage-multilingual-2 model map entries:
voyage-4-nano is not served on the Voyage API and voyage-multilingual-2 is
an older model, so neither belongs in this change.
* fix(voyage): restore MongoDB routing on contextual endpoint and re-add voyage-4-nano after upstream merge
The upstream merge replaced the contextual config's get_complete_url with a
Voyage-only host and dropped voyage-4-nano from the model map. Reapply the
al- key routing via get_default_base_url and add voyage-4-nano (open-weight,
per docs.voyageai.com) so the branch keeps its task changes on top of upstream.
* fix(voyage): drop voyage-4-nano and the contextual tests upstream already covers
voyage-4-nano is not served on the Voyage API, so it does not belong in the
model map. The contextual input tests duplicate
tests/test_litellm/llms/voyage/test_voyage_contextual_embedding.py, which
landed upstream with the auto-chunking fix this branch was carrying.
* test(voyage): drop sys.path.insert from the voyage common utils test
* fix(voyage): route rerank on the key it authenticates with
get_complete_url picked the rerank host from the environment while the auth
header carried the request key, so an explicit MongoDB-issued key was posted
to api.voyageai.com. validate_environment now hands the resolved key to the
config instance and get_complete_url reads it back, so the host and the
credential always come from one key. The config is built per request, so
nothing carries over between them.
* test: allow ultrafast pricing keys in model prices schema
* test: drop ultrafast schema keys now added upstream
* test(integration): prove voyage key-based host routing on the wire
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): forward the inline-tools-2026-09-15 beta to Vertex and Anthropic
* test(vertex_ai): trim the inline-tools beta test docstrings and type their locals
* test(vertex_ai): add audit cells for inline-tools beta forwarding on Vertex Anthropic
* test(vertex_ai): send legal whitespace in the hostile beta header audit cell
* test(vertex_ai): close the SDK clients in the inline-tools audit cells
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): audit the websocket rejection log on every live route
* test(proxy): scan the stopped proxy log for the websocket crash and drop the uvicorn wording pin
* fix(proxy): mark background responses stale_expired when provider returns 404
Responses deleted upstream (store=false / ZDR rows dropped after provider retention) now move to stale_expired on the poll instead of being retried every cycle until the 7 day stale cleanup.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop inline comment in check_responses_cost
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drive _get_response seam in 404 stale_expired tests
Assigning AsyncMock on the instance keeps the new tests inside the TQ002/TQ008 test-quality ceilings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only expire 404 responses polled through a resolved deployment
A NotFoundError from the bare SDK fallback can mean a missing or misconfigured deployment, which a config fix inside the staleness window can still recover. Rows whose poll went through their router deployment are the ones the provider actually dropped, so only those move to stale_expired.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve the deployment before fetching so via_router binds once
Drops the reinitialised loop flag flagged by review and modernizes the moved annotation.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drive the real response fetch in the poll 404 regression tests
* fix(enterprise): expire background response rows on any provider 404 status
The GET path maps the provider's 404 body to litellm.BadRequestError that still carries status_code 404, so an except on NotFoundError never fired and the poll job kept retrying the row every cycle. Key the terminal decision on the status code instead of the class
* fix(enterprise): expire a background response only on a 404 naming it and skip a bad deployment entry per job
* chore(enterprise): record the poll-cycle bound on the managed-object IN lists
* test(integration): cover background response poll retirement on the real proxy
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: move 76 live top-level tests to offline unit and integration coverage
* test: keep the mock-param opt-in guard valid once the top-level senders are gone
* test: address review on offline replacements
* docs(test): drop the deleted harness test file from the harness check command
* test: make the key rebind, team member delete and routes integration tests exercise the legacy paths
* test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing
* test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks
* test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds
* test: scope the admin health check to the test's own deployment
---------
Co-authored-by: yuneng <yuneng@berri.ai>
