* test(e2e): cover tool_reference tool results on bedrock converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): keep tool_reference results as text on converse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(bedrock): format tool reference parser
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover bedrock converse tool_result of only tool_reference blocks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): tag converse tool_reference case with e2e metadata
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): avoid unchecked cast in converse tool_reference mapping
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): keep tool_reference results as text on responses paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): tighten tool_reference mapping on responses paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(completion-extras): expect tool_reference names in responses input
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(bedrock): drop responses converter changes, keep converse tool_reference fix only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: kerry <kerry@berri.ai>
* fix(proxy): keep the mapped status on assistants and threads route errors
* test(proxy): type and annotate the assistants and threads route error tests
* fix(proxy): redact internal details from assistants and threads route error messages
* test(proxy): integration cells for assistants route error mapping
* test(proxy): drop malformed JSON cases that never reach the assistants route handler
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(azure_ai): support Microsoft-Decision-1 on the decisions API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_ai): strip project path and API suffix together in decisions base
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(agents): ban semicolons and colons in human-facing text
Tighten the writing rule so ";" and ":" are only allowed where the sentence would be nonsensical without them, rewrite the existing semicolons in AGENTS.md to follow it, and note that docs (litellm-docs) keep their trailing "."
* docs(agents): rewrite semicolons in nested AGENTS.md files
* docs(agents): hand-maintain the dashboard Next.js warning and sync e2e CONTRIBUTING lines
* docs(agents): point at the PR template path that exists
* docs(agents): restore the generated Next.js block in the dashboard AGENTS.md
* feat(rust): add the DeepSeek Anthropic Messages config
* refactor(rust): drop the tool discriminator without mutation and use rstest in DeepSeek tests
* feat(rust): add the Vertex AI Anthropic Messages config
* fix(rust): normalize Vertex system turns the way Python does
Vertex uses the shared normalize_system_role_messages: a leading run is
hoisted, a later turn stays or is converted in place by the model's
supports_mid_conversation_system, and billing blocks are filtered after the
hoist so one inside a system-role message no longer reaches Vertex.
* fix(rust_bridge): keep non-Claude Vertex Messages calls on Python
The catalog admits every vertex_ai Messages call, but Rust serves only Claude
there and reports anything else as a terminal InvalidProvider instead of a
decline. The dispatcher now bypasses Rust for those models, as
get_provider_anthropic_messages_config does.
* refactor(rust): template the Vertex location and project errors
The location failure is the shared InvalidType detail and the missing project
is MissingSetting with the provider, setting, param and variable name, so no
sentence is spelled at the return site.
* test(rust): use rstest in the Vertex AI Messages tests
* refactor(rust): derive the missing Vertex project error from its spec
* refactor(rust): give Messages configs typed litellm params
* feat(rust): port _normalize_system_role_messages for Messages hosts
mid_conversation_system mirrors Python's module function for function: a
leading run of system turns is hoisted into the top-level system, a later
turn stays in place when the model map flags supports_mid_conversation_system
and otherwise becomes a user turn where it was, never between a tool_use and
its tool_result, and billing blocks are stripped from the hoisted system. The
capability joins MessagesModelCapabilities and the bridge projection. Azure AI
uses it in place of fold_system_role_messages, which hoisted every turn.
* fix(rust): keep the Bedrock runtime endpoint behind a blank api_base
* refactor(rust): import LitellmParams from its crate instead of a re-export
* refactor(rust): template the Messages request decode errors
Both decode failures go through ErrorDetail::invalid with the subject and the
serde error as source instead of a preformatted sentence.
* refactor(rust_bridge): read the module globals the param specs name
The Messages host no longer keeps its own list of litellm globals; a spec
whose spellings the call leaves out names the global to read, so the next
family that has one needs no host change.
* refactor(rust): give each route lookup Python's implicit None
ProviderConfigManager's per-route methods list only the providers the route
serves and fall through to None; the Rust lookups now do the same, so a new
LlmProviders variant touches only the route that serves it.
* feat(decisions): add Databricks ai_decide as a /v1/decisions provider and auto-router decider
* fix(decisions): reject Databricks endpoint names carrying URL separators and keep the provider wiring under llms/
* fix(decisions): reject dot-segment Databricks endpoint names and add Databricks to the dashboard's OSS classifier editor
* fix(ui): name every OSS classifier provider in the classifier radio copy
* feat(decisions): register the Databricks OpenJev decider as an evaluation-mode cost-map entry
* fix(cost-map): declare zero cache rates on the databricks-openjev-qwen35-4b entry
* fix(proxy): reconcile every DB deployment when a registry miss lands on an auto router
A /model/new for an auto router lands on one replica. A request routed to
that auto router on a sibling replica missed it, the model read-through
loaded only the auto router's own row, and the pre-routing hook then picked
a tier the sibling had not loaded, so the caller got a 400 "no healthy
deployments for <tier>" until the next periodic reload. An auto router miss
now runs the full add_deployment reconcile, tiers included. Found by the
audit's two-worker rig on this PR's routing cells; CircleCI's one-worker
proxy never opens the window.
* test(integration): audit cells for Databricks decisions and the Databricks auto-router classifier
Wire cells for /v1/decisions with databricks/<endpoint> deployments, routing
cells for the complexity router's databricks classifier on all three
endpoints with the Postgres rows read back, management cells for the
classifier's spend writes and for the evaluation-mode health check of a
Databricks serving endpoint.
* refactor(cost-map): remove openrouter rows, part 1 of 5 (anthropic to google block)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 2 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 3 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 4 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 5 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(policy-engine): stream detect-only post_call pipeline steps live when buffering is off
* fix(policy-engine): scan live pipeline streams a provider error cuts short and discard rewrites per step
A live detect-only pipeline now scans the chunks the client received when the provider fails mid-stream, then re-raises the error. Each step's rewrite is discarded right after the step, so later steps scan the text the client actually received.
* fix(proxy): keep a guardrail verdict recorded after a stream failure on the failure spend row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): sign RDS IAM tokens for the database's own region
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add per-connection RDS IAM signing region overrides for writer and reader
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): integration cells for per-connection RDS IAM signing regions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): opt-in real RDS e2e cells for cross-region IAM signing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): address review on RDS IAM region cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop recording-front integration tests, mark RDS e2e suite e2e and register its model via /model/new
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert the spend read lands on the replica, not just a live connection
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type generate_iam_auth_token params and retry the replica spend-read check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): derive RDS region from custom RDS Proxy endpoint hostnames
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): add Subject, step labels, frozen rows and Final to the RDS IAM e2e suite
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop the opt-in RDS IAM e2e suite in favor of unit coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
* refactor(cost-map): remove openrouter rows, part 1 of 5 (anthropic to google block)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 2 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 3 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost-map): remove openrouter rows, part 4 of 5
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(github_copilot): sync model catalog with Copilot's served models
* test(github_copilot): assert catalog invariants instead of pinning vendor facts
* fix(github_copilot): keep thinking capabilities on served Claude catalog rows
* fix(github_copilot): keep supports_reasoning on catalog rows that shadow the family fallback
* fix(model-catalog): preserve merged Copilot Haiku metadata
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange
* fix(ui): create a credential for credential-only auth types in Add Model
* fix(microsoft_365_copilot): accept max_tokens and list the chat model
* fix(microsoft_365_copilot): register provider model set
* fix(ui): hide other auth types' fields when credential-only types are filtered
* fix(health): do not inherit stored auth settings when a connection test brings its own api_key
* docs(health): restore connection-test inheritance docstrings
* fix(lint): suppress justified provider boundary casts
* fix(lint): satisfy M365 type-discipline gate
* fix(types): eliminate type-check gate regressions
* style: shorten suppression reasons so ruff format leaves them on one line
* refactor(types): drop shared-file type widening unrelated to the Copilot provider
* fix(health): pass caller headers as secret_fields to connection-test probes
* fix(types): type connection-test probe and token counter locally
* fix(lint): drop cast import from connection-test header getter
* chore: remove explanatory comments flagged by review
* fix(proxy): gate OAuth client credentials behind proxy admin
* fix(router): skip cooldown on caller-scoped OAuth auth failures
* fix(router): skip fallback cooldown on caller-scoped OAuth auth failures
* fix(router): type caller-scoped cooldown lookups
* fix(ci): include Microsoft 365 Copilot tests in unit shard
* fix(proxy): keep saved endpoint when test_connection retests a deployment by id
* fix(m365): collapse doubled Graph replies
* fix(m365): preserve trailing system messages
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): keep end-of-stream output assembly and Redis pipeline logging off the event loop
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(caching): serialize Redis pipeline values with orjson
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(proxy): resolve model-level guardrails once per streamed request
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): type the streamed guardrail cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): fall back to json.dumps when orjson is not installed
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): cover the json.dumps fallback for values orjson rejects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep new guardrail cache and orjson helper within the type gates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(caching): guard Redis print_verbose value interpolation behind is_debugging_on
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(caching): drop the orjson fast path from Redis pipeline writes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(caching): drop the debug-on print_verbose test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert(proxy): drop the per-request guardrail lookup cache from this PR
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(logging): let print_verbose take %s args so Redis cache values format only when verbose
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(caching): type set_cache key/value and redis_version so print_verbose args are known to pyright
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Counts failed gateway requests by HTTP status at the ASGI edge
(LiteLLM_DailyGatewayFailedRequests) and rolls failures up per virtual key,
team, user and model group with the status the logging callbacks recorded
(LiteLLM_DailyRequestErrors, fed from the spend writer, flushed like the
gateway counters, directly or through Redis with a pod lease, and on
shutdown). GET /gateway/daily/activity gains by_status_code; the new
admin-only GET /gateway/errors/activity returns per-day totals with
per-status counts, failures by status code, and keys, teams, users and
model groups ranked by failures with their top status. Dashboard schema.d.ts
regenerated.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): list the account's invocable models behind bedrock/* when check_provider_endpoint is on
* fix(bedrock): sign the listing query RFC 3986 style and raise BedrockError on a failed listing
* test(utils): keep test_utils.py at main's formatting, only the Bedrock discovery test is new
* fix(bedrock): chain BedrockModelInfo's constructor so the passthrough config keeps its event parser
* fix(bedrock): list discovered ids without the provider prefix so partial wildcards filter
get_valid_models returns bare vendor ids for every other provider and for the static bedrock catalog, and the proxy's wildcard expansion adds the provider prefix itself. The lister prefixed its ids, so a partial wildcard like bedrock/anthropic.* never matched the proxy's filter and listed uncallable bedrock/anthropic.bedrock/<id> entries
* test(bedrock): integration cells for wildcard discovery through the proxy and the SDK
* fix(proxy): keep a partial wildcard a filter when its deployment repeats the prefix
The wildcard expansion guessed filter-or-alias by whether any provider id carried the prefix. A deployment such as model_name bedrock/anthropic.* over model bedrock/anthropic.* can only ever route names that share the prefix, so when the account has no on-demand anthropic.* id (every Anthropic model behind an inference profile) the guess fell into the alias branch and listed bedrock/anthropic.<every id>, none of them callable. A deployment whose model repeats the suffix now always filters, and lists nothing when nothing matches
* fix(proxy): prefix every discovered id under a custom wildcard prefix that starts a vendor id
* test(bedrock): Final and read-only annotations in the wildcard discovery tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(bedrock): route Grok 4.7 tools with reasoning to native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): mint unique tool call ids on native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): send a blank text block for Converse tool results with no supported content
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(bedrock): default Grok on Bedrock to native Chat Completions and leave Converse unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): gate tool call id minting to Grok on native Chat Completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(bedrock): check the Grok tool call id gate at both call sites
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): remint positional tool call ids on native Chat Completions for every model
AWS returned call_0, call_1 for GPT 5.6 as well as Grok on 2026-10-09 (and unique ids for GPT 5.6 earlier the same day), so the id format is not fixed per model. Drop the Grok-only gate and remint any call_<digits> id on the route
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): drop stop sequences for Grok instead of forwarding them to a 400
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(teams): let proxy admins grant team admins the right to raise their team budget
Adds a raise_max_budget entry to team_admin_editable_team_fields. With max_budget alone a team admin can still only keep or lower the team budget. Adding raise_max_budget lets them raise it, capped by the organization's max_budget when the team belongs to one, while removing the budget stays with proxy admins. The entry is rejected without max_budget, /team/info reports may_raise_max_budget to the caller, and the Admin UI nests the new checkbox under Max Budget with a tooltip and shows the team admin a hint under the budget field.
* fix(ui): shorten the raise_max_budget tooltip and link it to the docs
* fix(teams): pin the organization in the guarded team budget write
A granted raise is checked against the team's organization at read time, so the write now also requires organization_id to be unchanged. A concurrent move into a budgeted org returns 409 instead of landing an uncapped raise. Also types the new test helpers.
* test(teams): type the team row store methods and the touched race test arguments
* test(teams): annotate the last fixture and parametrize arguments in the raise_max_budget tests
* test(integration): poll the partition lock witness off the event loop
The witness poll ran a blocking psycopg connect and query on the test's event loop, so the partition DDL task only progressed during the 20ms sleeps and missed the 3s deadline on loaded runners. Polling through asyncio.to_thread lets the DDL reach the held lock concurrently, and the deadline is 10s since it only bounds how long the DDL takes to start waiting
* test(e2e): retry the reasoning turn until the model emits a reasoning item
Whether gpt-5.4-mini emits a reasoning item next to a forced function call is up to the model, and it skipped it on four scheduled runs while a same-SHA rerun passed. The first turn now retries up to three times before the unchanged reasoning-item assertion
* test(e2e): retry a Bedrock Converse model once when it reports no cache tokens
Opus on Bedrock Converse reported zero cache tokens on half of the scheduled runs while a same-SHA rerun passed. A model that comes back uncached is asked once more inside the 5 minute window, and the non-zero cache token assertion is unchanged
* test(e2e): poll until a deleted stored response stops being retrievable
Azure kept serving a just-deleted streamed response for a moment, so a single retrieve did not raise. Both delete tests now poll retrieve until it returns an error and still require a 4xx
* test(e2e): give the Together prompt cache five primed attempts
Together documents its prefix cache as best-effort, and three primed attempts came back with zero cached tokens once while a same-SHA rerun passed
* test(e2e): allow 30s for the idle RSS reading at session start
A loaded router replica took longer than 10s to answer the session-start memory read, which pytest reruns cannot recover. The RSS budget assertion is unchanged
* test(e2e): scope the MCP submission check to its own card
The register call timed out at 15s on a loaded stack and leaked a submitted server, so the retries matched two "3 passing, 1 failing" cards. Registration gets 60s and the check reads only the submitted server's card
* test(router): wait for the primary success record before counting it
The shadow fan-out test waited only for the two shadow success events and then asserted exactly one primary success, which the logging worker sometimes had not delivered yet. It needed a rerun on 5 of 59 main runs
* test(proxy): give the spend-log cleanup run a 1s budget
A 0.25s budget could expire before the first batch on a loaded xdist worker, leaving rows_deleted at 0. One second still stops the 50-batch, 5-second loop on the deadline, which is what the test proves
* test(autorouter): release the slow token count at 2.05s instead of 2.3s
The count only has to finish after a quarter of the 8s worker budget, and the planner gives it 3s, so releasing at 2.3s left 0.7s of slack that a loaded runner used up. 2.05s is still past the quarter mark with almost 1s of slack
* test(xai): run the reasoning-effort tests on grok-4.6 with a prompt that needs reasoning
grok-4.7 now returns reasoning tokens without any reasoning text, so reasoning_content was never set and both tests failed on every scheduled run. Called directly, grok-4.6 returned reasoning text 6 of 6 times on a step-by-step arithmetic prompt but only some of the time on a bare greeting
* test(integration): give the two worker-kill chaos tests the owned-proxy time budget
Each test spends about 50s on the burst and then up to graceful_stop_seconds() stopping its owned proxy, which overran the 90s default pytest timeout on loaded runners. They now use the same 2 * graceful_stop_seconds() + 120 budget as other owned-proxy tests
* test(integration): compare regional image responses without their created timestamp
Two identical generations that straddle a second boundary differ only in created, which failed the byte-for-byte comparison. Every other field and both spend rows are still compared
* test(integration): count only the S3 logger's flush task
The set of asyncio tasks created while the logger starts also picked up client close finalizers left by earlier tests, so the count ranged from 1 to 8. The test now counts the periodic_flush task it owns and cancels
* test(integration): send guardrail timeout probes eight at a time
All 44 probes went at once to a two-worker proxy, so on a loaded runner some guardrails used their whole 1s timeout before their request left the proxy, and the sink never saw them. A different set of providers failed on each run. Eight in flight still overlaps the waits
* test(integration): count a killed Arize chaos worker as gone once it is a zombie
psutil reports an unreaped zombie as still running, so the 10s death check failed whenever the supervisor was slow to reap the SIGKILLed worker, and the stop then overran the 90s default timeout. The test now accepts a zombie or a missing process and has the owned-proxy time budget
* test(integration): send the Typesafe connection test its real model and endpoint
The test passed the proxy alias as litellm_params.model, and /health/test_connection lays the request over the stored deployment, so the alias replaced the real typesafe/ model and the check failed with "LLM Provider NOT provided" on every run since it landed in #45481. It now sends the provider model, api_base and api_key directly
* test(integration): keep DB-stored models out of the usage-routing Redis read test
The owned proxy inherited store_model_in_db and the job's shared database, so a model another test left behind joined the router and added its cooldown key to the MGET the test compares exactly. Reproduced locally with one /model/new model present (3 failed), and green with model loading from the database turned off
* test(vertex_ai): prove the batch upload streams by laziness instead of peak memory
The two tracemalloc ratio tests flaked on unrelated PRs because peak memory on a shared xdist worker includes other threads' allocations and garbage from earlier tests. They are replaced by deterministic checks of the same property: the upload stream parses and maps a row only when it is pulled, so a body whose tail is not JSON yields its valid rows first, and a Path source reflects a row rewritten on disk after the upload started. Making the parse, the output, or the file read eager fails these tests
* test(integration): keep the alias test_connection call as a known bug
* test: type the counting mapper and the Bedrock rerun helper
* test: annotate the new test locals as Final and build them as tuples
* test(vertex_ai): prove pull-driven transforms with the garbage tail alone
The two offline tests in tests/local_testing/test_anthropic_prompt_caching.py failed on main because
#24071 started passing logging_obj to client.post, so assert_called_with on the patched
AsyncHTTPHandler.post no longer matched. The request bodies themselves were still correct
Both now live in tests/unit/llms/anthropic/chat and assert the body and headers that reach the
wire through respx instead of patching our own HTTP handler. The coverage allowlist entry keeps the
five tests that still need provider credentials
* feat(decisions): add hosted_vllm provider
Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move the hosted_vllm placeholder key to constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit
Relocate 53 legacy tests that pass with no network or keys into the tests/unit
files that mirror the code they exercise, drop 2 llm_guard tests already covered
by test_llm_guard_call_type_aliases, and delete the emptied legacy files.
* test: freeze the reset clock and run the real reset methods in the moved budget reset tests
Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await
the service-hook tasks the job spawns instead of sleeping. Drive the per-user
failure with a malformed budget_duration row from the prisma mock so the real
_reset_budget_for_key/user/team run instead of patched replacements.
* test: use fixed timestamps in the moved pass-through and spend log tests
* test: mock the moderation HTTP call and type the moved moderation and spend log tests
Serve the OpenAI moderation response through an httpx MockTransport so the real
async_make_request runs, annotate the moved tests' locals with Final, and pass
get_logging_payload typed kwargs instead of an untyped input_args dict.
* test: drive the moved proxy tests through boundaries and annotate their locals with Final
Serve GCS downloads and the Router moderation call through httpx mocks with fake
Google credentials, let the real pass-through success handler fail on the mocked
upstream response, and fold the Vertex live route endpoint check into the existing
route test. Annotate moved locals with Final, build the budget reset rows without
rebinding, and type the remaining fakes without **kwargs
Moves 58 legacy nodes (cyberark, logging callback manager, langfuse handler and helpers, SQS, app crypto, bedrock nova and invoke, cloudflare, litellm_proxy, nvidia nim, replicate, triton, convert_dict_to_response, get_model_info, vcr live-call probe) into the tests/unit files that mirror the modules they exercise. Calls that reached huggingface.co, langfuse or 0.0.0.0 are refused at the HTTP boundary with respx. Tests that need credentials, a license, a spawned server, sleeps, a subprocess, or assert nothing stay in their legacy files
* test: move offline router, latency-routing, retry and exception tests from local_testing into tests/unit
Moves 29 offline nodes into the tests/unit files that mirror the code they exercise.
Hosted-provider calls are mocked at the HTTP boundary with respx, and sleeps that only
built a start/end gap are replaced by explicit timestamps. Legacy files left empty are deleted.
* ci: record the live prompt caching cases left in tests/local_testing as unrun
Moving the router prompt caching test to tests/unit leaves only live Anthropic and Vertex cases in tests/local_testing/test_anthropic_prompt_caching.py, which every CircleCI -k already deselects
* test: freeze the clock in the moved lowest-latency routing tests
The latency logger buckets usage by the current minute, so a run that crosses a minute boundary could grow the cache and fail the memory check. The moved tests now use fixed timestamps and a frozen clock in the logger module
* ci: describe the mixed prompt caching file accurately in the coverage allowlist
* fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): read the serving provider's row for prompt cache breakpoints on the chat bridge and base_model
* fix(anthropic_cache_control_hook): rename the module-level provider resolver so the recursion check passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): follow the credential dialog's new name field and auth method id in the federation e2e spec
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test: wait on conditions instead of wall-clock in guardrail parallelism and scope-option tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: annotate the guardrail barrier locals as Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): never forward the caller's LiteLLM key to MCP servers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): tidy caller-key scrub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): avoid master-key scanner false positive
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): match admission when scrubbing the caller key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): drop header copy churn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): normalize bearer variants in caller-key match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): inject admission header name and cover provider keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): exercise real REST header extraction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover caller-key scrub on the real REST path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): exclude gateway admission header from upstream forwarding
* fix(mcp): skip empty admission headers when scrubbing keys
* fix(mcp): use scrubbed server credentials for auth probes
* fix(mcp): match probe credential precedence to upstream calls
* test(mcp): simplify probe header assertion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* fix(proxy): read the UTC clock once per operation in gateway tracking, PTU rollup and Mavvrik export
* refactor(proxy): default the injected clocks to get_utc_datetime instead of three private copies
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(credentials): add credential_alias and make credential_name immutable on PATCH
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): query credential rows by role to stay under the no-node-access budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): expect credential_alias in load_credential_list dump
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drive the credential SearchSelect by placeholder and option roles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): pass the decrypted CredentialItem to update_db_credential during master key rotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): trim credential_alias in CredentialModal so whitespace-only input clears the alias
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(credentials): replace credential_alias with display_name and reject edits to config credentials
credential_name stays the immutable reference key. display_name is a nullable, trimmed, max 255
character label set on POST or PATCH (omit keeps it, null clears it, blank is a 400). Reads return
display_name plus an in-memory source tag (db or config), and PATCH or DELETE on a config-defined
credential answers 400 instead of a misleading 404
* feat(ui): show credential display names everywhere and lock config credentials
The credentials table shows the display name with the credential name beneath it, badges config
credentials and disables their edit and delete actions. The edit modal adds an editable display
name next to the read-only credential name and only sends it when it changed. Every credential
picker and reference (add model, model info, models table, vector store form and info) shows the
label while still submitting credential_name
* feat(cli): set display names on credentials and show them in the list
* refactor(credentials): keep the new credential types inside the lint gate budgets
* refactor(cli): print credential command JSON through one helper
* fix(credentials): treat an empty credential_name on PATCH as omitted and pin the 404 for vanished rows
A blank credential_name never renamed anything, so PATCH accepts it again instead of answering 400. New tests pin a trimmed display_name on PATCH, the repository carrying display_name, and a 404 (not the config-owned 400) for a DB credential another worker already deleted. The hydration helper no longer copies display_name, since none of its callers read it, and the CLI update passes display_name straight through because the exactly-one check already makes it None when clearing.
* fix(ui): keep a model's credential when its picker text is emptied and skip the admin-only list for other roles
Clearing the search text in the model edit form's credential picker used to submit null, silently detaching the model's credential on save. None is now the only way to clear it. The models table fetches /credentials only for proxy admins, since everyone else got a 403 on each page load and falls back to the raw name anyway. Also fixes a type error in the re-use credential dialog and adds tests for the display name surviving a provider switch, the 255 character limit, the request payload, and the vector store None choice.
* fix(client): percent-encode the credential name when updating its display name
A name with ? or # was cut short in the URL, so the PATCH landed on a different credential or 404'd.
* fix(credentials): answer 405 with Allow: GET for PATCH and DELETE on config credentials
A config-defined credential exists (GET returns it) but is read-only through the API, which is what 405 Method Not Allowed means. 400 described a malformed request, and 404 would claim the credential does not exist.
* fix(proxy): ignore a display_name set on config credential_list entries so non-string values cannot fail boot
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): count a leading system run through count_tokens' system parameter
Anthropic's count_tokens rejects role "system" at the head of messages, so
/v1/responses/input_tokens with instructions, and /v1/messages/count_tokens
with a system-role message, fell back to the local tokenizer. The shared
Anthropic count_tokens transformation now lifts the leading run of system
messages into the top-level system parameter, the way the chat path sends
it, after any system the caller set. Anthropic direct, Azure AI Anthropic,
and Bedrock Mantle share that transformation.
* test(integration): cover the count_tokens leading-system lift across Anthropic, Azure AI and Bedrock Mantle
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* ci: restructure lint, unit and smoke workflows
* ci: preserve source formatting in tier workflows
* ci: retire per-shard coverage flags and tighten the tier workflow guards
* test(ci): freeze the workflow startup safety models
* ci: keep the current required check names running until the ruleset moves to the tier collectors
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(token_counter): price a base64 PDF document per page instead of as one image
A `document` or `file` block carrying inline PDF bytes was priced like a single
image (85 tokens), so when a provider's count-tokens endpoint rejected the
model (Bedrock Opus) the local fallback answered 116 for a 12-page PDF the
provider then billed at 35941 input tokens. The counter now reads the PDF with
pypdf and prices each page as its extracted text plus the image Anthropic
renders it to (1568 px long edge, 1.15 MP, 750 pixels per token), falling back
to the old image pricing when pypdf is missing or the bytes are not a readable PDF.
* refactor(token_counter): count PDF pages as they are read
Sum each page's text and rendered-image tokens straight from the pypdf
reader instead of materializing a page list first, keep the fallback
to image pricing atomic when a page cannot be read, and annotate the
new tests' locals as Final
* test(integration): audit cells for page-priced PDF documents in count_tokens, pre-call checks and spend
* test(integration): release the held peer when the concurrent budget wait times out
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): match deployment pricing ids against the cost map only within the deployment's provider
A deployment model_info.id that equals another provider's catalog key
(e.g. baseten/zai-org/glm-5.2 on an openai-compatible api_base) was merged
into that provider's built-in row by register_model, keeping
litellm_provider=baseten on the entry. _check_provider_match then rejected
the row at request time and the deployment billed $0. register_model now
takes a keyword-only custom_llm_provider used to scope
_get_builtin_model_info_for_registration and
_resolve_builtin_model_cost_entry, and the router passes the deployment's
provider through. The stored entry stays provider-less for
non-colliding ids.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(utils): pass custom_llm_provider straight to the registration lookup
Drop the per-entry lookup-provider expression in register_model per review:
the kwarg alone scopes _get_builtin_model_info_for_registration, and
_resolve_builtin_model_cost_entry keeps its main signature and caller.
_register_custom_pricing_for_request passes the provider through so
router-originated per-request registrations get the same scoped lookup.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(utils): drop docstring and simplify set restore in per-request collision test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): type the colliding-id deployment fixture
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): pin pi compat flags for gateway models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): annotate pi compatibility field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): clarify pi compatibility suppression
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cli): drop the stale mutable-ok suppression on the pi compat block
---------
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt
An order-based or configured fallback hop replayed the failed provider's
encrypted reasoning items to the next deployment, which answered 400
(Bedrock Mantle: invalid encrypted reasoning; OpenAI:
invalid_encrypted_content), so every multi-turn Responses fallback for
Codex-style clients failed. The hop now drops the encrypted reasoning
its target cannot decrypt and keeps each item's readable summary. With
encrypted_content_affinity on, the pin narrows to the hop's target
order instead of emptying it, so the hop reaches the next order
instead of failing with no deployments available.
* fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt
Unmarked encrypted reasoning on a hop is attributed to the deployment that just
failed, read from the retry breadcrumb, so a hop to a deployment on the same
api_base and api_key keeps it and a cross-provider hop still drops it. The hop
tests script the upstream at the httpx boundary instead of doubling the handler,
and the router coverage script lists the two hop helpers with their tests
* fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot
* test(router): use a real Router without the origin deployment in the hop strip test
* test(router): cover the hop strip when no failed deployment is known
* fix(proxy): drop the router's fallback hop state keys from the client body
* fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys
* test(integration): audit cells for the fallback hop encrypted reasoning strip
Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
