* fix(responses): drop bridge-minted reasoning items from OpenAI replays
* fix(responses): send id-less stored reasoning items without a made-up id
The chat-to-Responses bridge gave a stored reasoning item with no id an rs_<n> id that OpenAI rejects (404 without encrypted content, 400 with it); an id-less item is accepted and verified by OpenAI itself. Decoding encrypted_content now keeps the verifiable thinking blocks of a mixed array instead of rejecting the whole array, and the verifiable-block rule lives in the shared module.
* test(integration): audit minted reasoning item replay across responses, chat bridge and messages
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): stop repeating streamed thinking text in the signature chunk
The Anthropic stream handler emitted, on the signature_delta, a thinking block carrying every thinking delta seen so far plus the signature, after it had already streamed that text as per-delta blocks. Every additive consumer (the Agents SDK, stream_chunk_builder, the Responses bridge) stored the text twice under one signature and replayed the doubled block on the next turn
The signature chunk now carries a signature-only block, matching the Bedrock emitter, so accumulators rebuild the text once and the saved history replays to Anthropic exactly as it was streamed
* test(anthropic): audit the signature-only thinking chunk across wire providers
* test(anthropic): pin the cached replay's thinking shape in the wire audit
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat: add Bespoke Nimble gateway and OSS classifier support
* feat: accept Ollama's nimble model name for the Bespoke provider
* test: exempt the POST-only bespoke decisions route from the all-methods check
test_pass_through_routes_support_all_methods requires every built-in
pass-through route to accept every HTTP method unless it is listed in
PROTOCOL_CONSTRAINED_PASS_THROUGH_ROUTES. /bespoke/v1/systemone is
POST-only like /laya/v1/systemone, so the test failed at this branch
and passed at the merge base. List it alongside Laya.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(proxy-extras): retry P3009 when a peer already recovered the named migration row
* fix(proxy-extras): pin P3009 recovery locals as Final and cover a recovered sibling row
* test(proxy-extras): stop the P3009 ledger fakes shadowing the partition detector cursor
* fix(proxy-extras): annotate the migration ledger reader locals as Final
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* refactor(proxy): extract shared spend log read policy
* test(proxy): use named bindings for spend scope regression
* test(proxy): reuse existing spend log query harness
* test(proxy): cover spend log permission lookup adoption
* chore(proxy): relocate existing spend query baseline
* refactor(proxy): make scope query returns explicit
* refactor(proxy): inject deferred log permission lookup
* test(proxy): cover teamless management compatibility lookup
* refactor(proxy): compose user and team log grants
* refactor(proxy): share generic authorization composition
* refactor(proxy): compose trace read permissions
* refactor(proxy): centralize spend and trace authorization
* refactor(proxy): strengthen spend and trace scope types
* refactor(proxy): flatten log read scope into owned logs
Replace the AnyOf grant tree with a flat OwnedLogs(user_id, team_ids) scope,
and OwnedTraces(logs, api_key_hash) for traces, since every consumer flattened
the tree back into that shape.
A caller with no user id now gets an empty scope instead of matching ownerless
rows through Prisma's IS NULL. The dead request_id guard in ui_view_spend_logs
is removed, and the management facets inject the log team lookup and reuse
read_scope_sql instead of the list shim.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): run spend scope tests through one SQLite emulator
Replace the string-matching payload emulator and the hand-rolled Prisma where
interpreter with one SQLite helper that runs the real scope SQL. Session scope
tests now go through the endpoint, including the no-user caller that must not
match ownerless rows. Drop duplicated lookup-failure and trace mapping cases.
load_permitted_log_team_ids returns no teams without a database instead of
relying on the resolver's broad except.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(proxy): unify log and trace ownership permissions
* test(tracing): align fixtures with ownership read scopes
* refactor(tracing): align query scopes with row ownership
* refactor(spend): make ownership SQL predicates explicit
* test(spend): validate ownership SQL against PostgreSQL
* docs(traces): drop key-row visibility from query help guide
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): reach the empty-memberships branch in team lookup test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate dashboard API types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): stop queued registry read-throughs spending the resync budget
RegistryReadThrough.attempt serializes misses behind one lock, but every
request that was queued behind the first one still spent a unit of the
20-per-5s resync budget and re-ran the DB resync, even though the first
request had already loaded the object. A burst of more than 20 requests
for a model created on another worker therefore exhausted the budget and
the rest got 400 "Invalid model name".
attempt now checks whether the key is already loaded once it holds the
lock and returns early without touching the budget. Models check the
router's model names and deployment ids, guardrails and agents reuse
their existing registry lookups.
* test(proxy): gate the queued read-through test on events and cover each registry's loaded check
The queued-requests test now holds the first resync on an asyncio.Event instead of
a timed sleep and records calls in a recorder with tuple and frozenset state. New
tests show the model, guardrail and agent read-throughs each answer an object that
is already loaded without reading the database, so rewiring any registry's loaded
check now fails a test
* test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture
* refactor(traces): type the ClickHouse query help response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(traces): include agent names and frameworks in named contract round trips
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(traces): cover native query help validation in the storage adapter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): answer the model-info refresh GET in the mixed MCP responses wire peer
The proxy's periodic model-info refresh sends GET /v1/models to the deployment api_base, which tripped the peer's /responses-only assertion when a tick landed mid-test
* test(e2e): wait for the api-keys URL after clicking Virtual Keys in onboarding
/ui already renders the Virtual Keys heading, so the helper returned before navigation finished. The late route change moved focus and closed the account menu popover in hideLiteAdmin
* test(proxy): stop two unit modules leaking app.openapi_schema and a session-wide Router
test_custom_openapi cached a stripped schema on app.openapi_schema and never cleared it, breaking later openapi route tests. test_proxy_reject_logging built a module-level Router that stayed in the live router registry all session and re-added cost-map keys during a reload. Reset the schema via monkeypatch and make the Router a function-scoped fixture
* perf(proxy): aggregate daily model usage per flush instead of upserting per request
* perf(proxy): drain queued model usage in the spend log flush job
* perf(proxy): queue model usage at request time instead of writing to the db
* test(proxy): cover batched daily model usage aggregation and retries
* test(proxy): read back model insights written by the batched flush
* fix(proxy): drain the whole model usage queue each flush so it cannot grow unbounded
* test(proxy): give the mock prisma client a model usage queue
* test(proxy): cover draining a model usage queue larger than one spend log batch
* refactor(mcp): extract upstream preparation and support modern clients
* fix(mcp): reject incompatible upstream transport before saving
* fix(mcp): serialize protocol validation with server updates
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* wip
* wip
* test(traces): separate root status from diagnostic error counts
* test(traces): cover normalization precedence and fallbacks
* chore(cache): remove stray comments from trace PR
* test(traces): name lens test for shared query path
* fix(traces): place query implementation before test module
* test(traces): use unified read scope in migration tests
* ci(rust): allow feature checks to finish
* ci(mcp): allow dependency resolution to finish
* fix(traces): preserve key visibility and safe spend attribution
* test(proxy): stop the proxy_server app fixture leaking LITELLM_LOG
The session app fixture set LITELLM_LOG=ERROR with os.environ.setdefault and never removed it, so later tests on the same xdist worker inherited it. test_drop_params_env_var spawns a subprocess with os.environ and lost the warning it asserts on. Scope the variable to the import with a MonkeyPatch context
* test(secret-detection): give the hand-built redaction request an ASGI path
Since #43975 _read_request_body checks the route path via request.scope, and a scope without path raised KeyError that was swallowed into an empty body, so chat_completion failed with a missing messages parameter. Real ASGI scopes always carry path
* test(integration): isolate litellm callback lists per sdk test
usage-based-routing-v2 Routers register their selector in litellm.callbacks and nothing removes it, not even Router.reset(). The counter TTL and Redis service metrics tests left their selectors behind, and the next usage routing test ran their pre-call checks against its own rpm=1 deployments, raising "Deployment over defined rpm limit". An autouse fixture now gives each sdk test copies of the callback lists and restores the originals afterwards
* test(integration): keep the owner-lookup fault proxy off the shared read replica
The owned proxy points DATABASE_URL at a scratch database but inherited
DATABASE_URL_READ_REPLICA from the replica job, so auth read the shared
database and rejected the freshly created key with token_not_found_in_db.
Drop the replica variable like the other scratch-database owned proxies
* test(integration): request every seeded key in the team owner breakdown
The aggregated team activity endpoint now caps breakdown.api_keys at the top
100 keys by default (#43398), so the 300 seeded keys came back as 100 rows.
The test guarantees each key is reported with its own owner, so ask for an
api_key_limit that covers all seeded keys
* test(integration): give every owned Redis its own port in the redis-cache container
On CircleCI every owned Redis ran on the fixed port 16379 inside the shared
redis-cache container. When an earlier server still held that port, the new
one failed to bind, readiness pinged the old server, the pidfile read failed
and cleanup then reported "Owned Redis still serves after shutdown"
Reserve an ephemeral port for the docker-exec path the same way the local
binary path already does, and refuse to start when something already serves
the chosen port so the failure names the real cause
* test(e2e): skip the Vertex Mistral partner case the e2e project cannot reach
The e2e Vertex project gets a 404 publisher model not found for vertex_ai/mistral-small-2503, so the case can only fail
* test(e2e): skip the Vertex gpt-oss partner case the e2e project never serves
vertex_ai/openai/gpt-oss-120b-maas has hit a 60s read timeout with no response headers on every run in the e2e Vertex project since the case was ported, and no other Vertex partner chat model passes there to switch to
* test(e2e): check only stored message content for a leaked card number
The Presidio spend-log check ran the card-number pattern over the whole serialized response, so a Luhn-valid usage.cost float (0.0003466000000000001) failed the streaming /v1/messages case although the stored content was <CREDIT_CARD>. The check now reads the content and text strings of the stored response, which is where a raw card would land, and still requires the placeholder there
* test(e2e): assert the proxy decodes token-array embeddings for titan
The port in #44120 carried over a legacy SDK-direct test that expected Bedrock to reject token ids with a 400. Through the proxy, /embeddings decodes token arrays to text for providers that cannot embed tokens, so titan answers 200. The test now sends a token array and its decoded sentence and requires the two vectors to match, which fails if the proxy stops decoding or decodes with the wrong tokenizer
* test(e2e): run the Bedrock extended-thinking round trip on a model that honors enabled thinking
us.anthropic.claude-sonnet-5-5 is adaptive-only, so litellm sends thinking.type=enabled with a 1024 budget as adaptive with low effort, and Bedrock returned no reasoning blocks on 5 of 5 identical Converse calls (boto3 direct agreed). us.anthropic.claude-sonnet-4-6 accepts the legacy shape verbatim and returned reasoning on 5 of 5. The non-thinking Bedrock case stays on sonnet-5-5
* test(proxy): stop unit modules forcing DEBUG logging into the event-loop lag tests
Five tests/unit modules set verbose_proxy_logger to DEBUG at import, so every xdist worker that collected them logged the 2.4MB pass-through response from a worker thread, and secret redaction of that line held the GIL for ~0.8s+ inside the timed window. The lag tests now pin the LiteLLM loggers to WARNING and freeze gc while timing, and the module-level DEBUG overrides are removed
* test(e2e): cite the tokenizer and date behind the titan token-array fixture
* test(e2e): let migration seed replicas finish their request-log indexes before cloning
Since #43948 a serving proxy builds the two LiteLLM_SpendLogs indexes on a background thread after it reports ready. The seed fixtures stopped the replica at readiness, so every cloned legacy database lacked an index no real deployment would be missing, and the v2 baseline diff refused it. Seeds now wait until both indexes exist and are valid in the database's schema
* test(passthrough): give the pass-through MockRequest an httpx URL and ASGI scope
#43626 made get_request_route read request.scope during pass-through kwarg setup; the MockRequest in tests/unit/passthrough had neither a scope nor a URL object, so both stream-param tests raised before reaching the code they check. Mirrors the repair #43626 made to the tests/pass_through_unit_tests fake
* test(integration): ignore foreign allow_all_keys MCP servers in the access matrix tool list
test_toolset_gateway_url_serves_a_team_granted_toolset_to_a_key_without_its_own_grant (#43908) registers an allow_all_keys server on the shared gateway, and allow_all_keys servers are listed to every key by design, so a matrix case running on another xdist worker at the same time saw its tools. The matrix now drops tools of allow_all_keys servers it did not create, read from LiteLLM_MCPServerTable before and after listing, and still compares everything else exactly
* fix(lens): run investigations with configured wildcard models
* fix(lens): validate worker model access and pricing before analysis
* fix(lens): bound worker validation and preserve unrelated edits
* fix(scim): apply path-less group PATCH ops instead of storing them under an empty metadata key
A path-less add/replace op (RFC 7644 3.5.2, what Okta Push Groups sends on a
rename) carries a partial Group resource. Each of its attributes now applies as
if sent with that path, so displayName updates the team alias and externalId
and members get their usual handling, and the pushed attributes merge into the
scim_data snapshot the PUT path already writes. A path-less remove or a
path-less op without an object value is rejected with a 400. Any group PATCH
drops an empty metadata key an earlier push left behind, and the Admin UI
metadata form skips an empty key so an affected team can save its settings.
* fix(scim): let a later path op win over an earlier path-less value in the group snapshot
* fix(scim): type the stored team metadata before the JSON object check
* test(scim): run the real group transformation in the path-less replace test
* test(scim): assert the renamed group comes back from the path-less replace
* test(scim): audit the path-less group PATCH on the live proxy
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(exa): fall back to highlights/summary when text is missing (#36905)
* test: type exa search transformation test helpers
* test(exa): drop type: ignore in test helper and trim redundant comments
* test(exa): move snippet fallback test to tests/unit and parametrize it
Uses a real httpx.Response instead of a Mock and annotates snippet as Final.
* fix(exa): inline snippet fallback since Final is not allowed inside a loop
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(proxy): carry key, team and project tags into pass-through spend logs
Pass-through endpoints built their request metadata without the key, team and project controls that native routes apply, so spend rows for configured routes and provider pass-throughs like /anthropic dropped the key, team and project tags and the key and team spend_logs_metadata. The native team and project controls now live in a shared helper that both paths call, key spend_logs_metadata is copied instead of aliased from the cached key, client metadata cannot overwrite user_api_key_ fields, and header tags dedupe with the same merge used on native routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop covers markers from pass-through tag tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): type the shared team and project control helper
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): validate pass-through endpoint list instead of suppressing pyright
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover body, streaming, hostile, forged and native cells for pass-through tags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: mock pass-through request url as httpx.URL after rebase on main
* refactor(proxy): merge spend_logs_metadata sources without a stacked comprehension
* refactor(proxy): name the team and request spend_logs_metadata merge
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): cover non-dict callback_settings.otel and bare EXCLUDED_SERVICES env
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): tolerate non-dict callback_settings.otel and ignore bare EXCLUDED_SERVICES env
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(otel): simplify settings_customise_sources signature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): poll for present spans instead of waiting the full window
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): audit null otel block and bare EXCLUDED_SERVICES across the excluded-services matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): assert per-trace datastore spans in the unconfigured burst cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): keep pydantic-settings runtime options on the OTel v2 env source
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): require post-auth datastore spans at the tenant on the cache-hit twin
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(otel): cover pydantic-settings runtime options in callback_settings.otel through the proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): jwt auto_register map-existing-key repro
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(jwt): auto_register_map_existing_key maps JWT to the user's existing virtual key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): exclude blocked keys from auto_register_map_existing_key reuse
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(jwt): route existing-key lookup through VerificationTokenRepository
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): stop requiring LITELLM_SALT_KEY for the owned JWT gateway
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): gate the owned JWT gateway tests behind E2E_OWNED_GATEWAY
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(jwt): only reuse keys that can call LLM routes in auto_register_map_existing_key
Skip Admin UI session keys and keys whose allowed_routes restrict them to
anything other than llm_api_routes (management, read_only, password-reset
sessions). Mapping a JWT to one of those left the user with 401s or 403s on
every LLM call, since the mapping persists.
* fix(jwt): scope auto_register_map_existing_key reuse to the JWT-resolved team
Only reuse a key whose team_id matches the team auth_builder resolved for
the JWT (no team matches no team), so a personal key can no longer bypass
the resolved team's model and budget limits.
With the flag on, the first JWT request now falls through to the same
virtual-key checks later mapped requests get, instead of returning early,
so a reused key's own limits apply from request one rather than 200 then
403. Flag off keeps the early return unchanged.
* fix(jwt): keep the early return when no master key is set
Without a master key the generic virtual-key path returns a bare
INTERNAL_USER object, so falling through on the first auto-registered
request dropped the key's team, models and budgets. Only fall through when
a master key is configured.
Tests now assert the reused key per team rather than the query shape, and
cover the flag-off early return and the no-master-key case.
* test(jwt): assert on race-loser's returned key, not only mocks (TQ002)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(jwt): close the auto_register_map_existing_key race, shared-claim and expiry holes
A key auto_register just minted is never adopted by a concurrent request, so the race loser's cleanup can no longer delete a key another request mapped and cascade its mapping away (503, user left with no key)
Reuse only happens when the claim value is the JWT-resolved user_id. A shared claim such as azp or client_id falls back to minting, so one user can no longer land on another user's personal key and budget
Only keys that never expire are reused, so an expiring key can no longer pin the claim to a permanent 401
Integration tests on a real proxy and Postgres cover all three. The race test holds the first mapping insert in a Postgres relay, so the interleaving is forced rather than timed. The where-clause shape unit tests are replaced by these, since only a real database proves the filter
* test(e2e): create the reused key in the team the JWT resolves to
The flag only reuses a key in the JWT-resolved team, and this identity's groups claim resolves to its team, so a teamless key was never eligible and the test could not pass
* test(integration): match the held statement across TCP reads
The relay looked for the trigger inside one read, so an insert split across two reads was never held and the race test would fail waiting for it. It now matches one exact trigger over a window that keeps the end of the previous read
* fix(jwt): gate key reuse on the claim field, not on the claim value
Requiring the claim value to equal the resolved user_id skipped reuse for users matched through the sso_user_id or case-insensitive email fallback, whose stored user_id differs from the JWT sub. That is the lookup LIT-5378 asks for. Reuse is now allowed when the virtual key claim is the user_id or user_email JWT field, globally or for the token's issuer, which still keeps shared claims such as azp or client_id on the mint path
* fix(jwt): let an issuer's own user field replace the global one when gating key reuse
An issuer that identifies users by uid no longer treats the global sub field as a user identity claim, so a shared sub under that issuer mints instead of reusing a personal key
* test(jwt): make the flag-off test fail when the flag no longer gates key reuse
The flag-off test used a config where sub was not a user identity claim, so deleting the flag check still passed. Configure user_id_jwt_field=sub so only the flag keeps the lookup off, and drop test docstrings
* chore(lint): drop mutable-ok suppressions that LIT013 flags as no-ops
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Mrinal Chanshetty <mchanshetty@Mrinals-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The auto-router usage view summed the lifetime savings of every session
overlapping the date range, so it disagreed with the Overall savings view,
which sums daily rollups by request day.
Record auto-routed money per UTC request day and router in one new table,
written in the same statement as the session rollup and corrected in the
same transaction as late baseline estimates. The all-router headline reads
the same daily rows and filters as Overall; savings no router day row
accounts for are reported as unattributed and void the baseline comparison.
Session shape and caching stay whole-session and are labelled so; the
savings-per-session tile is removed.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: add Laya gateway and classifier backend
* test: cover the pass-through model_group pin and repair the shard fakes
MockRequest in tests/pass_through_unit_tests gains an httpx.URL and an ASGI
scope, which get_request_route now reads inside
_init_kwargs_for_pass_through_endpoint, and the POST-only /laya/v1/systemone
route joins the protocol-constrained exemptions. A built-in pass-through pins
metadata.model_group to the resolved model so a client cannot choose its own
per-model budget key; test_pass_through_endpoints now proves that on a
non-Laya route and drops a duplicated assertion.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
The migration job runs DatabaseURLSettings.apply_to_env(), which rewrites
DATABASE_SSLMODE=verify-full plus DATABASE_SSLROOTCERT into Prisma's TLS
dialect: sslmode=require&sslcert=<CA>&sslaccept=strict. The request-log
index build then hands that same URL to psycopg, and libpq reads sslcert
as a client certificate, failing with "certificate present, but not
private key file" on every verify-full deployment since #43948.
_strip_prisma_query_params now undoes the Prisma dialect before psycopg
sees the URL. sslaccept=strict (or any value Prisma treats as strict)
becomes sslrootcert=<CA> plus sslmode=verify-full whatever sslmode said,
since strict verifies chain and hostname and libpq only does that in
verify-full; sslmode=disable stays off. Without strict, Prisma verifies
nothing, so the CA is dropped and sslmode is kept as is. A URL that also
carries sslkey is libpq's own client-certificate form and is left alone.
Resolves LIT-9169
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): preserve requested service tier in Responses calls
* fix(chatgpt): avoid extra mutable collections in tier filtering
* test(chatgpt): refresh service-tier cases after main rebase
Keep the tier-preservation regression on the current subscription model and match the moved unit-test suite's formatting. The adapter still forwards preferences without claiming backend priority entitlement.
* refactor(chatgpt): map service tier through a lookup table after the allowlist filter
---------
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(oci): resolve the GenAI endpoint realm from the region instead of hardcoding oraclecloud.com
Government (OC2/OC3/OC4) and other non-commercial realms live under a
different second-level domain, so a request for us-luke-1 was sent to
inference.generativeai.us-luke-1.oci.oraclecloud.com and failed DNS
Delegate the lookup to the OCI SDK's region registry when it is installed,
honour OCI_DEFAULT_REALM otherwise, and keep api_base as the explicit override
* fix(oci): read per-region realm metadata without the SDK and cover the registry path
Replace the global OCI_DEFAULT_REALM fallback, which would have redirected
commercial regions too in a mixed deployment, with the SDK's own per-region
sources: OCI_REGION_METADATA and ~/.oci/regions-config.json. Regions not
described anywhere keep their commercial endpoint
Exercise the SDK registry path with a fake oci.regions module so CI, which
has no SDK, still covers it, and skip the real-SDK test on oci.regions so a
namespace package named oci in the tests tree cannot masquerade as the SDK
* fix(oci): validate regions-config.json entries individually and tolerate undecodable files
One malformed entry no longer discards the valid ones, and the file is parsed
from bytes so an undecodable file is logged and ignored instead of failing
every OCI request built without the SDK
* fix(oci): resolve the realm from the compartment OCID so Government regions work without the SDK
The Docker image ships without the oci package and Government deployments rarely carry OCI_REGION_METADATA, so the reviewed fallback still sent us-luke-1 to oraclecloud.com. Every compartment OCID already names its realm (ocid1.compartment.oc2..), so map that key through the SDK's twenty realm domains first, then the metadata sources, then the SDK registry, then the commercial default.
* fix(oci): consult the SDK registry before hand-parsed region metadata and harden the fallback
Review follow-ups on the realm resolver. Read the compartment realm first, then the SDK registry when it is installed, and only then the hand-parsed metadata sources, so the same file is never parsed twice with different rules. Lowercase metadata values like the SDK does, accept single-label realm domains, and expand ~ with os.path so a container without a home directory cannot raise out of URL building. Type the compartment as str | None at the caller, drop populate_by_name, isolate the legacy region tests from the developer's ~/.oci, and keep the new tests on the immutable style.
With DATABASE_URL set and no way to run the Prisma CLI (not on PATH, not importable), run_server used to print a plain notice and keep booting. The server then crashed later inside the DB exception handler with an unrelated ModuleNotFoundError traceback, and the migration-only entrypoint (--skip_server_startup) exited 0 without migrating. It now exits 1 with a red one-line message naming the missing toolchain and how to install it. The no-DATABASE_URL path is unchanged.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp)!: disable stdio MCP servers by default
stdio MCP servers now only run when the proxy is started with
LITELLM_ENABLE_MCP_STDIO=true. While it is off, existing stdio servers stay
registered but never start: tool listings skip them quietly, direct tool
calls and health checks return a 403 naming the env var, and creating or
updating a stdio server is rejected. The flag is read from the process
environment only, so DB-stored environment_variables cannot turn it on.
The UI reads mcp_stdio_enabled from /.well-known/litellm-ui-config to grey
out the stdio transport, show a banner on stdio forms, and badge stdio
server cards.
BREAKING CHANGE: stdio MCP servers are off by default. Set
LITELLM_ENABLE_MCP_STDIO=true in the proxy environment and restart to keep
using them.
* fix(mcp): ignore stdio flag from config file and read UI flag from the selected worker
LITELLM_ENABLE_MCP_STDIO set under environment_variables in config.yaml is now skipped like the DB-stored value, so only the process environment can enable stdio. The dashboard reads mcp_stdio_enabled from the proxy it is managing, so a control plane shows each worker's own setting.
* test(mcp): cover non-mapping payloads in the shared transport validator
* fix(mcp): skip blocked stdio servers quietly in every listing and keep the UI unchanged until the flag loads
Prompt, resource and resource-template listings now skip a blocked stdio server at debug level like tool listing does, instead of logging a warning per server on every call. The dashboard only treats stdio as disabled once the proxy explicitly reports mcp_stdio_enabled false, so a proxy with the flag on, or an older one without the field, renders exactly as before with no flicker while loading.
* fix(mcp): route blocked stdio tool calls to the flag error and warn once per server
A gateway tools/call naming a blocked stdio server's tool now returns the
LITELLM_ENABLE_MCP_STDIO message instead of "Tool not found".
The "will not start" warning moves out of build_mcp_server_from_table, which
DB reload re-runs on every cycle for rows with a NULL updated_at and which
drafts and test-connection also call. It now fires when a row first enters
the registry or changes transport.
* fix(ui): explain on the server detail page why a stdio server is inert
The Overview and MCP Tools tabs showed "No tools available" with no reason
while stdio is disabled. The detail page now shows the same warning banner
as the edit form, and hands off to the form's banner once editing starts.
* refactor(ui): name the stdio banner conditions on the server detail page
Keeps local/no-long-condition-chain within its budget
* fix(proxy): log the ignored DB-stored LITELLM_ENABLE_MCP_STDIO warning once
The DB config sync re-reads environment_variables on every cycle, so a stored
flag logged the warning on each sync per worker
* fix(ui): register tencent in the Add Model provider dropdown
The Add Model provider dropdown is driven by the proxy's
/public/providers/fields endpoint, which serves
provider_create_fields.json. Tencent was frozen in the test's
ADD_MODEL_UNLISTED_PROVIDERS set, so it never appeared in the dropdown.
Add a Tencent entry (optional api_base + required api_key, matching
TENCENT_API_BASE/TENCENT_API_KEY) and unfreeze it in the backend test.
Register Tencent in the UI Providers enum, provider_map, and placeholder
map so the dropdown resolves the display name and model placeholder.
* fix(tencent): drop test docstring to satisfy comment policy
* fix(ui): bundle the Tencent Cloud logo for the provider dropdown
* fix(tracing): use ClickHouse URL for reads by default
* fix(tracing): unify ClickHouse storage configuration
* fix(tracing): update dashboard setup copy for one URL
* test(tracing): make tests/unit/tracing a package
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(config): drop legacy string tracing store variant
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tracing): own ClickHouse defaults in constants and reject unset env references
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tracing): use raw regex patterns in config tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tracing): read ClickHouse env defaults when tracing config resolves
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): split audit log query guard to fit condition-chain budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): fire MCP client test deadlines on conditions instead of wall-clock time
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): bound MCP client test failure paths independently of the triggered deadline
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): measure outer-deadline cleanup budget from cancellation, not setup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): move live-provider legacy tests into tests/e2e
Port legacy tests that exercise real providers into the tests/e2e suites that own them, using the harness (/model/new plus deferred cleanup) and asserting on what the caller receives. Delete legacy tests already covered at equal or stronger strength by e2e, integration or unit tests, and drop the now empty ocr_testing CircleCI job
* test(e2e): address review on the live-provider test move
Assert the SSE error frame a client actually receives when a post_call guardrail blocks a stream, and require a tool call for every requested city before checking the answer. Restore the OCR matrix and its CircleCI job, the Claude Agent SDK streaming test, and test_async_create_batch, since their SDK-level and callback assertions have no equivalent in tests/e2e
* test(e2e): accept both guardrail block shapes on a blocked stream
A post_call block before the first chunk reaches the client as HTTP 400 with either a JSON error body or a single SSE error frame, depending on whether the block surfaced as an exception or an error chunk. Assert the policy message is present and the blocked output is absent in both
* test(realtime): restore direct SDK realtime tests against OpenAI
The e2e realtime tests go through the proxy and the remaining SDK tests either mock the upstream or assert less, so keep the direct litellm._arealtime tests with and without intent, and TestOpenAIRealtime::test_realtime_connection, in place
* test: make realtime and Nova stream checks deterministic
The direct SDK realtime tests now fail on a refused connection instead of skipping. The with-intent test asserts OpenAI rejects the exact intent value sent, which only happens when the intent is forwarded. The Nova /v1/messages stream test asserts stream structure, stop reason and usage instead of model wording
* test(realtime): own intent forwarding with a unit test instead of a live rejection
Assert litellm._arealtime passes the intent query param into the OpenAI realtime websocket URL, which is the behavior LiteLLM owns, and drop the live test that depended on OpenAI's rejection wording
* fix(azure_storage): keep client call ids from sharing one Data Lake file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(azure_storage): tighten adls_safe_file_name docstring
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(azure_storage): give ids with dot or empty path segments their own Data Lake file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure_storage): cover failure, cache-hit and edge call ids in Data Lake file names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(azure_storage): run the Data Lake file name cells on two proxy workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): straiker v3 routes sk_agt_ keys to v3 and stops reading a missing verdict as allow
An sk_agt_ key always calls /api/v3/detect, even when the guardrail was saved with
api_version 'v1' by the old shared default: the v1 webhook rejects that key with 401.
A 200 with no decision, or permissionDecision 'ask', now takes the failure policy
instead of allowing the request. A detect-mode action is still not a block.
A response-phase block is no longer remembered under the request, so asking the same
question again is scored instead of refused from memory. The replay memory is scoped
by principal and session together, so two principals on one session id never share a
block. A text-only /guardrails/apply_guardrail call relays the text as a user turn.
The agent_ref description now matches the code: the configured value wins.
* fix(guardrails): straiker v3 relays text beside an empty messages list and keys the memory on the key
/guardrails/apply_guardrail sends `messages: []` beside `text`; an empty list is no
conversation, so the text is relayed as the user turn. A verdict whose blocked_by is not
a list states no decision and takes the failure policy, the same way a missing decision
does. A key that names no user is still the caller, so the replay memory is keyed on the
key when no user is known.
* test(guardrails): build the straiker replay-scope request data without mutating it
---------
Co-authored-by: PhimmStraiker <PhimmStraiker@users.noreply.github.com>
* feat(tracing): add SQL queries and schema-aware query help
* test(tracing): verify help requests and sync API types
* refactor(tracing): render query help with Askama
* refactor(tracing): use jinja extension for query guide
* fix(tracing): preserve query help when discovery fails
* feat(tracing): enforce team SQL scope with managed ClickHouse readers
* test(tracing): verify reads with one ClickHouse URL
* fix(tracing): revoke rotated trace reader credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(tracing): streamline query help catalog assembly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(tracing): run query help discovery sequentially
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(tracing): update reader setup request expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Remove the ENFORCE_PRISMA_MIGRATION_CHECK opt-in. When PrismaManager.setup_database returns False (database unreachable, connection retries exhausted, or prisma migrate deploy failing after retries) the proxy now always prints the red failure message and exits 1 instead of serving requests against a database whose schema may be behind the code. The --enforce_prisma_migration_check flag stays as a hidden no-op that prints a one-line deprecation warning so existing container args keep parsing; the env var is no longer read anywhere. The standalone migration entrypoint always runs run_server(("--skip_server_startup",)), and the integration launchers drop the flag.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The daily spend tables store date as text, so a request date that strptime accepts but that is not spelled YYYY-MM-DD (2026-9-24, 2026-09-4, full-width digits) was compared as raw text against canonical rows and matched nothing, and the export route copied it into Content-Disposition, which fails latin-1 encoding and returned 500. A shared parse_canonical_date_range now rejects any spelling whose round trip differs from the input, so every bounded daily activity route (user, team, tag, organization, customer, agent: aggregated, aggregated/keys, search, model top keys, export, cache leakage) and the paginated get_daily_activity path answer 400 before touching the repository, and the export filename is built from the validated dates.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The shared exclusion predicate negated a PostgreSQL ANY comparison without handling NULL, so NULL entity ids evaluated to UNKNOWN and dropped out of the Unassigned bucket whenever exclude_*_ids was set. The paginated daily rows Prisma filter had the same NOT IN shape and gets the same IS NULL OR NOT IN treatment
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): add Microsoft 365 (Graph) server to the MCP catalog
* fix(mcp): signpost the self-hosted Microsoft 365 URL and pin the catalog entry in tests
* test(mcp): pin both shipped copies of a catalog icon to the same bytes
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): persist SSO display name as user_alias on login
Generic/Microsoft SSO already parsed the IdP display_name, first_name and last_name into the SSO result, but the user upsert only wrote user_email and user_role, so the Users table never showed a name. Store the display name (first + last as fallback) as user_alias on first login and on later logins of users whose alias is still empty; never overwrite an alias already set. Whitespace-only names are treated as missing.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep stored user_email when SSO login carries no email claim
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* revert: keep stored user_email change, login writes the IdP email as before
Reverts 47c65cecff. A stored email staying eligible for email-based account linking after the IdP stops sending it is not wanted; the PR goes back to the user_alias fix only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add LITELLM_DISABLE_LAZY_ROUTES to register optional routers at startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): register eager lazy routes at startup so late eager routes keep precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): share one optional-feature install path between lazy and eager registration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* docs(proxy): say eager lazy routes register at worker startup
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): name the import callable passed to _install
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prove a startup hook can drop eager lazy routes for good
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): strip the lazy-routes flag from the lazy-mode control proxies
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover the lazy warmup route registering a feature
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(unit): run tests/unit/proxy/test__lazy_features.py in the proxy-server-core shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for LITELLM_DISABLE_LAZY_ROUTES
Flag spellings, /openapi.json at boot, the warm-up route in both modes, /mcp/proxy ahead of the /mcp mount,
route table and OpenAPI parity with a fully warmed lazy proxy, a broken optional import in both modes, every
client SDK against the completion endpoints, a boot burst with a killed worker, and a restart
* fix(proxy): keep config pass-through routes ahead of eagerly registered features
With LITELLM_DISABLE_LAZY_ROUTES set, features registered before the proxy lifespan
added config pass-through endpoints, so a pass-through overlapping a feature path
(e.g. a self-hosted /langfuse) lost to the built-in route. Restore lazy mode's
registry order once startup finishes, without bringing back routes a startup hook removed
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): accept Converse messages with no content key
A user or tool message whose content key is missing (or null, which the
message cleanup strips) made every Bedrock Converse request fail with
APIConnectionError 'content' before reaching Bedrock. The Converse
transform now reads content with .get for those messages, as it already
did for assistant messages: a content-less user message adds no block
and a content-less tool message becomes a toolResult with empty content.
The str branch also sends the continue message text instead of the
original whitespace-only text.
* fix(bedrock): send the continue message for a content-less user turn
* test(bedrock): type the content-less Converse message test parameters
* fix(bedrock): accept a Converse system message with no content key
* refactor(bedrock): read the system message content with get
* test(integration): audit Bedrock Converse messages without content
Adds the /audit cells for a chat message whose content key is missing or
null on a Converse-routed Bedrock deployment: happy, sad, edge, and chaos
rows through the OpenAI SDK, the Anthropic SDK, and raw httpx against the
scripted upstream, asserting the caller's response, the body the peer
received, and the spend row. The owned-proxy readiness deadline in the
integration harness is now INTEGRATION_PROXY_READY_SECONDS (default 70).
* test(integration): bound stray spend rows in the mid-burst restart cell
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
