* feat(decisions): add hosted_vllm provider
Adds HostedVLLMDecisionsConfig so hosted_vllm/<served model> works on
POST /v1/systemone and POST /v1/decisions against a vLLM server that
serves the System One body (vllm-project/vllm#59299, after v0.31.0).
vLLM only supports choice questions and rejects others, so the base
decisions config gains a per-provider health_check_questions attribute
and the evaluation-mode health probe sends a choice question for
hosted_vllm while every other provider keeps the noul probe.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): move the hosted_vllm placeholder key to constants
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 61e5f2dd3c)
#45214 routed both decision formats through the shared decisions IR in
litellm/llms/base_llm/decisions/transformation.py and litellm/types/decisions.py.
That left litellm/llms/base_llm/decisions/systemone.py (to_system_one_request,
question_keys, to_decisions_response, the SystemOne* models and their adapter)
and litellm/types/openai_decisions.py with no importer outside their own unit
tests, so both modules and both test files go.
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 0e48048bd5)
* feat(ui): add TypeSafe and Strands Decider to the Add Model provider list
* test(public_endpoints): type the provider fields helper against the route's model
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 3a3b8cad80)
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): keep APIConnectionError for unreachable deployments on every provider
`_map_upstream_exception` now maps a status `_handle_error` synthesized for a
non-HTTP failure to `APIConnectionError` before `exception_type` runs, so a
Perplexity (openai-compatible) deployment that cannot be reached no longer
answers `InternalServerError` where main answers `APIConnectionError`.
Audit cells: the unreachable-deployment integration cell is parametrized over
the five providers, OpenRouter connection failures are pinned on chat, stream,
embeddings, decisions and the SDK, a deployment whose provider has no decisions
config gets the 400 naming every supported provider, and a unit cell pins the
mapper alone. The first docstring sentence of the five translation bases is
dropped.
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
(cherry picked from commit 0e09e99146)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(decisions): add the OpenAI Decisions spec types and the System One translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): share one DecisionsModel config and require model and usage on responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): rename the shared pydantic parent to DecisionsObjectBase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): match the SDK on strict choice values and drop the invented min_length bounds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): compose DecisionsRequest from model and body and read the translator top down
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(decisions): use the plural Decisions prefix only for the request and response envelopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(decisions): import assert_never from typing_extensions for Python 3.10
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
(cherry picked from commit 79f62db620)
Cherry-pick of the rc/1.105.0 backport of #45663 (#45894), which already
adapts #45663 to branches that predate litellm/decisions/call.py,
GuardrailReadOnlyDetails.tsx, LiteLLMBaseModel, @/lib/decisionModels and
the TypeSafe Add Model provider. stable/1.104.x has the same decisions
providers as rc/1.105.0.
- The decision model unit test lives in tests/test_litellm/proxy/guardrails/
guardrail_hooks, where stable/1.104.x keeps the guardrail hook tests
- The OpenAPI snapshot and schema.d.ts match a regeneration on this branch
(cherry picked from commit 18047587cb)
(cherry picked from commit 5f9dd38867)
feat(decisions): backport /v1/systemone, OpenAI-format /v1/decisions and the OpenAI Decisions provider to stable/1.104.x for v1.104.2 (#44236, #45184, #45214)
Backport of the #45184 and #45214 guardrail fix.
Cherry-picked from 855f0bb765 (litellm_systemone_endpoint) as merged into #45214 at c19ada66ed (litellm_openai_decisions_provider).
Python 3.11 rejects an unhashable MappingProxyType as a dataclass default, so import litellm raised ValueError there. The same change needs to land on #45214.
The public hidden_params accessor comes from #44668, which is not on stable/1.104.x, so the backported test reads the private attribute like the rest of the file
System One and OpenAI-format bodies now convert to one internal representation, and each provider translates it to its own wire format. OpenAI deployments receive the caller's messages, images, names, typed choice values, level descriptions and safety_identifier unchanged. System One providers return a 400 for image input. Cached and cache-write tokens are priced on both response shapes, and provider response extras survive the round trip.
(cherry picked from commit 0dd207fde1)
System One requests to openai/ models are translated to OpenAI's Decisions shape on the way out and OpenAI's answers are translated back, so both /v1/systemone and /v1/decisions can route to gpt-6-luna.
(cherry picked from commit b94bbbc318)
Backport of #45184 to stable/1.104.x.
Cherry-picked from 3339469e73 (litellm_systemone_endpoint, #45184 is not merged yet). The playground
System One tab and managed agent authorization are not on this line, so those hunks are dropped,
and the lazy-disabled precedence test is dropped as in the #44236 pick.
Backport of #44236 to stable/1.104.x.
Cherry-picked from 8b1990b4bc (main). Adapted to this line: the lazy feature
helpers are ported by hand, managed agent authorization and the integration suites are not on
this line, and the lazy-disabled precedence test is dropped since attach_lazy_features takes no
feature list here.
schema.prisma still declares the LiteLLM_SpendLogs (api_key, startTime) and (litellm_call_id) indexes while their migrations are now SELECT 1, so the v1 resolver's drift script recreated both with a blocking CREATE INDEX at boot. The drift script now drops those two statements before it is applied, matching filter_request_log_index_diff on main. The schema sync test exempts the two indexes
Ported to rc/1.104.0 from stable/1.103.x (#44286), which is not on main: main carries filter_request_log_index_diff from #43948 instead, and its backport here was reverted in #44212. The key metadata recovery hunk is dropped because this line's test file has no Postgres-backed index fixture
(cherry picked from commit 4d245e1d9c)
* test(e2e): align completion, SAIL, and spend-log fixtures with supported contracts (#43902)
Partial backport to rc/1.104.0: only the text completion endpoint test moves off the deprecated gpt-3.5-turbo-instruct to openai/gpt-5.4-nano. The SAIL, spend-log and coverage registry repairs target tests or behavior this line does not need for the e2e suite
(cherry picked from commit e61733b170)
* test(ci): repair stale tests and flaky CI infrastructure (#43983)
Partial backport to rc/1.104.0: only the Together AI structured-output e2e case moves to the hybrid reasoning backend with reasoning off, since the reasoning-only model it picked returned no content. The CI infrastructure and other test repairs are not needed for this line's e2e suite
(cherry picked from commit 6ca90b927c)
* test: repair stale and polluting tests red on scheduled main CI (#44229)
Partial backport to rc/1.104.0: only the owned Redis port, Presidio stored-content check and event-loop lag logging fixes apply here. The other repairs target tests or behavior this line does not have
* test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG
The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context
* test(secret-detection): give the hand-built redaction request an ASGI path
Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path
* test(integration): isolate litellm callback lists per sdk test
usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards
* test(integration): keep the owner-lookup fault proxy off the shared read replica
The owned proxy points DATABASE_URL at a scratch database but inherited
DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared
database and rejected the freshly created key with token_not_found_in_db.
Drop the replica variable like the other scratch-database owned proxies
* test(integration): request every seeded key in the team owner breakdown
The aggregated team activity endpoint now caps breakdown.api_keys at the top
100 keys by default (#43398), so the 300 seeded keys came back as 100 rows.
The test guarantees each key is reported with its own owner, so ask for an
api_key_limit that covers all seeded keys
* test(integration): give every owned Redis its own port in the redis-cache container
On CircleCI every owned Redis ran on the fixed port 16379 inside the shared
redis-cache container. When an earlier server still held that port, the new
one failed to bind, readiness pinged the old server, the pidfile read failed
and cleanup then reported "Owned Redis still serves after shutdown"
Reserve an ephemeral port for the docker-exec path the same way the local
binary path already does, and refuse to start when something already serves
the chosen port so the failure names the real cause
* test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach
The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail
* test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves
vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to
* test(e2e): check only stored message content for a leaked card number
The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there
* test(e2e): assert the proxy decodes token-array embeddings for titan
The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer
* test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking
us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5
* test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests
Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed
* test(e2e): cite the tokenizer and date behind the titan token-array fixture
* test(e2e): let migration seed replicas finish their request-log indexes before cloning
Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema
* test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope
* test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list
test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly
(cherry picked from commit 9b5562f89b)
* fix(proxy-extras): retry P3009 when a peer already recovered the named migration row (#44283)
* fix(proxy-extras): retry P3009 when a peer already recovered the named migration row
* fix(proxy-extras): pin P3009 recovery locals as Final and cover a recovered sibling row
* test(proxy-extras): stop the P3009 ledger fakes shadowing the partition detector cursor
* fix(proxy-extras): annotate the migration ledger reader locals as Final
---------
Co-authored-by: yuneng <yuneng@berri.ai>
(cherry picked from commit 8b11b682f4)
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop queued registry read-throughs spending the resync budget (#44277)
* fix(proxy): stop queued registry read-throughs spending the resync budget
RegistryReadThrough.attempt serializes misses behind one lock, but every
request that was queued behind the first one still spent a unit of the
20-per-5s resync budget and re-ran the DB resync, even though the first
request had already loaded the object. A burst of more than 20 requests
for a model created on another worker therefore exhausted the budget and
the rest got 400 "Invalid model name".
attempt now checks whether the key is already loaded once it holds the
lock and returns early without touching the budget. Models check the
router's model names and deployment ids, guardrails and agents reuse
their existing registry lookups.
* test(proxy): gate the queued read-through test on events and cover each registry's loaded check
The queued-requests test now holds the first resync on an asyncio.Event instead of
a timed sleep and records calls in a recorder with tuple and frozenset state. New
tests show the model, guardrail and agent read-throughs each answer an object that
is already loaded without reading the database, so rewiring any registry's loaded
check now fails a test
* test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture
(cherry picked from commit b877a38e5f)
* chore(deps): bump oauthlib to 4.0.0
* test(integration): answer the model-info refresh GET in the mixed MCP responses wire peer
The proxy's periodic model-info refresh sends GET /v1/models to the deployment api_base, which tripped the peer's /responses-only assertion when a tick landed mid-test
* test(e2e): wait for the api-keys URL after clicking Virtual Keys in onboarding
/ui already renders the Virtual Keys heading, so the helper returned before navigation finished. The late route change moved focus and closed the account menu popover in hideLiteAdmin
* test(proxy): stop two unit modules leaking app.openapi_schema and a session-wide Router
test_custom_openapi cached a stripped schema on app.openapi_schema and never cleared it, breaking later openapi route tests. test_proxy_reject_logging built a module-level Router that stayed in the live router registry all session and re-added cost-map keys during a reload. Reset the schema via monkeypatch and make the Router a function-scoped fixture
(cherry picked from commit a292fd409f)
The failure spend-log row from #42695 is written for pass-through routes that
run as LLM API routes, which a config route only does with auth: true. The
test omitted auth and passed only while config wins (#41779) registered config
entries through the typed model, where auth defaults to true. #43962 restored
the pre-config-wins registration, so the route lost that status and the row
was never written. Set auth: true on the route so the test covers the logging
it was written for without depending on that side effect
(cherry picked from commit 1d9cd9b09c)
