Commit graph

35 commits

Author SHA1 Message Date
Joshua Valluru
a886e014bb test(mcp): cite pinned reference payload fixtures
Some checks failed
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
LiteLLM Rust / rust-wheel (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
2026-10-01 18:43:09 -07:00
Joshua Valluru
0b9a1e1b4a test(mcp): call the advertised conformance tool name 2026-10-01 18:29:35 -07:00
Joshua Valluru
1b3627588f test(mcp): require complete reports payloads and process cleanup 2026-10-01 18:29:35 -07:00
Joshua Valluru
0b4f465921 test(mcp): verify forced cleanup of a stalled reference 2026-10-01 18:29:35 -07:00
Joshua Valluru
056c6d927a test(mcp): verify conformance bridge fidelity and runner failure 2026-10-01 18:29:35 -07:00
Joshua Valluru
e50465654a test(mcp): use advertised fixture names without warming discovery 2026-10-01 18:29:35 -07:00
Joshua Valluru
4e05d745ea test(mcp): add pinned official conformance smoke checks 2026-10-01 18:29:35 -07:00
devin-ai-integration[bot]
d7e6154863
fix(mcp): resolve team-granted toolsets for non-admin keys and dashboard sessions (#43908)
* fix(mcp): expand team and dashboard grants when listing and serving toolsets

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): honor team toolset grants on the responses gateway path and expose a public team permission lookup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(mcp): drop the toolset route docstring tweak so the OpenAPI snapshot stays unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): scope inherited toolset grants by the key's own MCP ceiling

A key that declares any MCP grant of its own keeps only its own toolsets, one
that declares none inherits its team's, and require_key_mcp_access_defined
stops a virtual key inheriting while dashboard sessions and admitted users
still do. Adds the direct, no-grant, admin and key-ceiling integration cases
and makes the LLM gateway toolset case discriminate a scoped toolset from the
aggregate grant

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): resolve toolset grants per admitted source and enforce the live team roster

Dashboard sessions and gateway-admitted users now expand into the admitted
subject's per-team sources when resolving toolset grants, so a team-granted
toolset is not capped by the user's own MCP row and is reachable on the
namespaced route. A cached team id no longer grants a toolset unless the live
roster still lists the user, a team lookup fault only drops that team's
inherited grant, and /team/member_add evicts the cached team object so the new
member is authoritative immediately

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): read a key's named object permission before letting it inherit team toolsets

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(mcp): inject the toolset grant resolver into scope helpers so tests stop patching MCPRequestHandler

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): drop the duplicate admitted_subject_sources wrapper after merging main and follow its renamed resolvers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): honour fresh policy and the session resource scope on pinned toolsets

A pinned toolset scope now reads the toolset through the writer when the admitted session
requires fresh policy, so a tool revoked from the toolset is gone on the next request. A
gateway bearer scoped to one server can only open a toolset that names that server

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep operator-open servers out of toolset gateway urls

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): audit cells for team-granted toolsets across every surface

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): bound toolset edit convergence by both cache layers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): drop toolset integration cells that test behavior this PR does not change

Repeat-read byte identity, 20 concurrent calls, a stopped peer and a killed worker are covered
generically by test_mcp_resilience.py and test_mcp_user_env_vars.py. The toolset edit cell asserts
pre-existing cache propagation and flaked locally with connection resets while polling

* chore(mcp): drop mutable-ok suppressions that main's LIT013 now flags as unused

* chore(mcp): keep the require_key_mcp_access_defined read from adding an unknown-argument type error

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-01 14:15:06 -07:00
yuneng-jiang
6ca90b927c
test(ci): repair stale tests and flaky CI infrastructure (#43983)
* test(ci): add used_client_oauth_token to the GCS pub/sub spend-log golden

#43063 stamps used_client_oauth_token into spend-log metadata, so
test_async_gcs_pub_sub_v1 failed on main with an extra metadata key

* test(ui): give the auto-router threshold save wait room for the availability debounce

#42625 keeps Save disabled while a 300ms-debounced availability check runs.
This test waits for Save right after the change, so the whole debounce lands
inside waitFor's 1s default and it times out under CI load. It is the
recurring UI Unit Tests failure on main since #42625 landed

* test(e2e): expect no pricing tier on bills for streamed calls OpenAI served at default

#42870 added both the rule that a served default or standard tier bills at
base pricing and records no service_tier, and streamed tests expecting the
row to record 'default'. They have failed on every scheduled litellm-e2e run
since. The tests now map the served tier to the pricing basis the bill must
record and check input is billed at that basis's rate; the messages case
registers custom rates so the rate check has something to compare against

* test(e2e-ui): wait for the call-id search before hovering the logs row

The row the spec hovers is already on the unfiltered first page, so it was
found before the search request returned. The search response then
re-rendered the table under the mouse, and the Base UI tooltip never opened.
Reproduced with Playwright against a local proxy: hovering right after the
fill never shows the tooltip, hovering after the search response shows the
call id every time

* test(e2e): run the Together structured-output case on the hybrid Qwen with reasoning off

The case picked the cheapest Together row flagged supports_response_schema.
DeepSeek-V4-Flash-0731 hit its cost-map deprecation date on 2026-09-29, so the
pick moved to GLM-5.3-Flash, a reasoning-only model that spends the 1024-token
budget thinking and returns content=None. Qwen3.5-9B is the pinned hybrid model
the reasoning_effort=none case already exercises, and Together lists it with
structured output support

* test(integration): read the agent 365 guardrail status by its own name in spend logs

The MCP shard runs under xdist against one database, and a sibling file creates a
default_on pre_mcp_call content filter there. The owned proxy reloads DB guardrails, so
that filter's 'success' entry could land first in guardrail_information and the test
read it instead of the agent 365 verdict

* test(unit): ignore asyncio's leaked-task records in the budget limiter push-failure log check

gc.collect() inside the caplog window can collect a pending task an earlier test left
on a closed loop, and asyncio logs 'Task was destroyed but it is pending' into this
test's records. The check still counts every LiteLLM logger, and unretrieved task
exceptions on this loop still go through the asserted exception handler

* test(e2e-ui): fill the create-tag fields inside the dialog

#42949 added 'Filter by tag name' and 'Filter by description' inputs to the Tag
Management page, so page-wide getByLabel('Tag Name') and getByLabel('Description')
match two elements and Playwright's strict mode fails the create step

* test(integration): run integration proxies with the CI license

Multi-worker proxies start each uvicorn worker in a fresh process, so every
worker reads the license from its environment. Forward LITELLM_LICENSE into the
proxy and test runner environments

* ci: save GitHub Actions caches only from main and bump codecov-action to 5.5.5

Every pull request saved its own uv, maturin, Rust and Prisma caches, about
4.5 GB per PR, so the repository's 10 GB cache budget evicted main's entries
within minutes. Pull request jobs then missed every cache, downloaded all
dependencies from PyPI and hit the install step timeouts. Pull requests now
restore only, and main keeps the caches warm for them. test-linting and
check-ui-api-types run only on pull requests and keep saving

codecov-action 5.5.4 imports its signing key from the deleted codecovsecurity
keybase account, so every upload failed signature verification. 5.5.5 reads it
from codecovsecops; the key ID matches the one signing the current CLI

* test(unit): join the session-minting thread before collecting the handler

asyncio.to_thread resumes the test as soon as the worker sets its result,
while the pool thread can still hold the work item and through it the
handler. gc.collect() then cannot finalize the handler and the session stays
open. A pool that shuts down before the test continues drops that reference

* test(integration): relaunch owned proxies that lose their port, expire idle gateway connections early

owned_proxy_process released its reserved port and the proxy bound it only
after full startup, so another xdist worker or an outgoing connection could
take it first and the proxy exited with 'address already in use'. The launch
now retries on a fresh port when that happens and stops every failed attempt.

uvicorn closes idle keep-alive connections after 5 seconds and httpx expired
them at the same 5 seconds, so a request sent right at that mark could reuse a
socket the server was closing and get 'Connection reset by peer'. Gateway
clients now drop idle connections after 2 seconds

* ci(circleci): give the base SDK wheel build the same 30 minute no-output window as the Windows build

The release profile builds with fat LTO and one codegen unit, so the final
link of litellm-cache-s3 runs silently for minutes. Successful builds take
711 to 749 seconds, right at the default 10 minute no-output limit, and about
30% of recent runs were killed there

* test(integration): model the budget-reset database outage as 10 seconds instead of 5 refused connections

The proxy retries the database about every 30 seconds and each retry opens
roughly one connection, so a 5-connection outage took 3 to 4 retries to clear
and recovery landed between 60 and 90 seconds, straddling the test's 80 second
reset window. A fixed 10 second outage still refuses the immediate reconnect
and recovers on the next retry

* ci: move the unit-test uv cache split into a composite action

check_workflow_startup_safety sums every setup step's timeout, so the save and
restore variants each counted 5 minutes although only one runs. One composite
step keeps the setup ceiling at 35 minutes

* test(unit): point tiktoken at the bundled cache for every unit test

The rust_bridge tokenizer tests loaded o200k_base before any test in their
xdist worker had imported default_encoding, so tiktoken fell back to the
temp cache and tried to download under pytest-socket. Move the session
fixture from litellm_core_utils/conftest.py to the root unit conftest.

* test(integration): answer model discovery probes in the hosted_vllm wire tests

The router's periodic upstream model info refresh sends GET /v1/models to
hosted_vllm deployments, so a wire server that is live during a refresh
sees an extra request. Answer the probe with an empty model list and leave
it out of the provider-call assertions, matching the responses bridge
tests.
2026-10-01 17:46:43 +00:00
devin-ai-integration[bot]
c129ea4fc9
fix(mcp): scope OpenAPI listings to the exact server prefix and drop upstream OAuth metadata when a server is saved (#43608)
* fix(mcp): key discovery caches per caller correctly and drop stale caches on server updates

Discovery-list cache identity now uses the hashed token instead of the raw
api_key and treats MCPJWTSigner-signed servers as per caller. Server
definition changes also drop the cached upstream OAuth metadata. OpenAPI
listings look tools up under the normalized registry prefix with the
separator, so an overlapping sibling prefix no longer leaks into the list.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep the discovery cache digest call unchanged so CodeQL matches the existing alert

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): guard OAuth metadata cache writes with a per-server generation and drop unproven per-caller discovery keys

An upstream metadata fetch that started before a server edit could store its stale reply after
invalidate_oauth_metadata_cache ran. Invalidation now bumps a per-server generation and the fetch
only stores when the generation it captured before I/O is unchanged.

The MCPJWTSigner-based per-caller discovery classification and the api_key to token key change had no
reproduction (the signer only injects on tools/list, and UserAPIKeyAuth hashes api_key in place), so
both go back to the merge-base behavior.

Integration coverage under tests/integration/mcp: overlapping OpenAPI aliases, a config-declared
server name with a space, OAuth metadata refetch after a save, and the in-flight stale-write race

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep OAuth metadata generations only while a fetch is in flight

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): count queued OAuth metadata fetchers so invalidation survives lock handoff

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(mcp): keep a held OAuth metadata lock registered even when no fetcher slot claims it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): prove a peer worker drops stale upstream OAuth metadata after a save elsewhere

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 17:25:39 -07:00
devin-ai-integration[bot]
9dfa42dcde
refactor(types): replace Any with proven types in 7 files (#43704)
* refactor(types): replace Any with proven types in 11 files

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(types): revert Any changes that broke existing callers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(types): drop prompt factory helper wrappers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): cover typing sweep surfaces

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): tighten sweep audit tests

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 06:12:58 -07:00
devin-ai-integration[bot]
3726ce2cfc
refactor(guardrails): fix agent 365 to the production endpoint and log the opt-in fail_open at error level (#43189)
* feat(guardrails): fail open by default when Agent 365 cannot evaluate and count it in Prometheus

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(tests): ruff format the Prometheus fail-open registry test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(guardrails): add Agent 365 authority host override, fail-open integration test and per-guardrail YAML default

Add `authority_host` to the Agent 365 config (also read from AGENT365_AUTHORITY_HOST, then AZURE_AUTHORITY_HOST) so sovereign clouds and the integration test can point the OBO exchange at a different Entra host.

Add tests/integration/mcp/test_mcp_agent_365_guardrail.py, a real proxy test with Postgres, Redis, a scripted MCP upstream and local Entra and Agent 365 doubles covering the default fail-open, explicit fail-closed and fail-open, Defender Skipped, policy denial, persisted status and Prometheus counter.

Use PrometheusLogger.get_instance for the fail-open metric lookup instead of a hand-rolled callback scan. Clarify the config description: gateway credential failures fail open, caller token failures block.

Extract the dashboard YAML preview into teamGuardrailConfigYaml.ts so the effective per-guardrail default is unit tested and the "default" hint only shows when nothing was set explicitly.

Regenerate the lazy OpenAPI snapshot and schema.d.ts for the new field.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): default a scheme-less Agent 365 authority host to https and treat a null fallback as unset in the YAML preview

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): audit cells for the Agent 365 fail-open default across entry points, Entra faults, throttling and two workers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): prove both Agent 365 workers serve and that a killed worker is replaced

Each fresh connection reports its worker pid from /debug/memory/summary and its MCP catalog on the same
connection, so the two-worker readiness wait covers both workers by identity. The kill test now kills a
pid the proxy reported as a worker and waits for a replacement pid, instead of the first psutil child

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): drop the prometheus fail-open counter from the agent 365 guardrail

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): keep agent 365 fail closed by default and make fail_open an explicit opt-in

Restores the shared unreachable_fallback default and the sibling guardrail initializers, drops the Admin UI YAML preview that only existed for the per-guardrail default, and reworks the unit and integration tests so the default blocks with HTTP 503 while unreachable_fallback: fail_open lets availability failures through as Unscanned

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): append authority_host after the existing Agent365Guardrail parameters

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* feat(guardrails): agent 365 fails open by default and hides the production overrides from the UI form

Agent 365 sits in the runtime path of every MCP tool call, so an Entra or
Agent 365 outage now lets the call through unscanned (logged at error level,
recorded as Unscanned with guardrail_failed_to_respond) instead of blocking it.
unreachable_fallback: fail_closed stays as the opt-in strict mode. Policy
blocks, throttling, 4xx rejections and a rejected caller token still block

The shared unreachable_fallback field becomes nullable so each guardrail owns
its default; every sibling still resolves None to fail_closed and typesafe
keeps failing open

api_base, resource_app_id and agent_id have production defaults and leave the
dashboard form (ui_hidden); they stay available in config.yaml and env. The
authority_host override and its env keys are gone, the OBO exchange always
uses login.microsoftonline.com. The integration suite keeps only the cells
that need no Entra double, the evaluation paths live in unit tests with an
injected handler

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(ui): regenerate openapi snapshot and schema.d.ts for the nullable unreachable_fallback

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): fix agent 365 to the production endpoint and keep fail_closed as the default

Remove api_base, resource_app_id and agent_id from the Agent 365 config model, their AGENT365_* env fallbacks and the _is_ui_hidden helper: the evaluation URL and the Agent Tools app id are fixed production constants and the agent identity is always the caller's key alias. Revert the fail_open default; unreachable_fallback: fail_open stays an explicit opt-in. Restore the shared unreachable_fallback field, the sibling guardrail initializers and typesafe to main. Move the Entra dependent cells from the subprocess integration suite to unit tests with an injected HTTP handler.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(guardrails): warn when agent 365 yaml still carries the removed override keys

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(guardrails): inject the http handler into the agent 365 initializer instead of assigning it after construction

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-28 12:05:47 -07:00
joshua-berri
3743c8563e
fix(mcp): adopt shared server resolution and caller authorization (#43263)
* test(mcp): characterize server resolution and authorization

* refactor(mcp): extract shared server resolution

* fix(mcp): adopt shared resolution in management endpoints

* fix(mcp): scope credential metadata resolution outside loop

* test(mcp): pin catalog isolation and batched credential permissions

* fix(mcp): restrict catalog detail and batch credential permissions

* test(mcp): enforce identity isolation in database fixtures

* test(mcp): name resolution tests by behavior

* test(mcp): describe detail access assertion failures

* chore: keep agent naming discipline local

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-09-26 14:37:32 -07:00
joshua-berri
0300fc4ab2
test(mcp): pin server resolution and authorization behavior (#43261)
* test(mcp): characterize server resolution and authorization

* test(mcp): pin catalog isolation and batched credential permissions

* test(mcp): enforce identity isolation in database fixtures

* test(mcp): name resolution tests by behavior

* test(mcp): describe detail access assertion failures

* chore: keep agent naming discipline local

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-09-26 12:59:11 -07:00
yuneng-jiang
14f4c34c61
fix(ci): stop stale CI reds, keep unit tests off the host env, retry CyberArk policy conflicts (#43294)
* fix(ci): stop five stale or flaky CI reds and retry CyberArk policy-load conflicts

The Langfuse redaction unit test exports to a local OTLP capture instead of
polling Langfuse Cloud through a recorded lookup. The passthrough worker-kill
test only requires spend rows for requests the surviving worker served. The
spend-routes sweep treats the intentional /spend/capture_rate 503 as expected.
CyberArk retries a 409 policy load in Python, Rust and the e2e Conjur helper
instead of reading it as "variable exists". The integration egress guard now
matches the script's own cgroup, so it no longer blocks the CircleCI agent,
which runs as the same user.

* fix(ci): keep the policy-load backoff typed as float

* fix(ci): retry CyberArk policy loads without blocking the event loop and tighten the worker-kill and Langfuse tests

* fix(secrets): load CyberArk policy one request at a time per manager

* test(secrets): pin that non-conflict CyberArk policy failures are not retried

* test(unit): run tests/unit with only an allowlisted host environment

CircleCI's unit job inherits every project env var, so real provider keys,
REDIS_HOST, DATABASE_URL and AWS or Azure credentials reached tests that
assume none are set. Locally, litellm's import-time load_dotenv did the same
from any .env up the tree. The unit conftest now drops every variable outside
a small allowlist and disables dotenv before litellm is imported.

* test(e2e): name a failed search and the stuck batch status instead of misattributing them

The websearch session test read an empty web_search_tool_result_error block as a
successful search, so a failing search tool surfaced as a session billing bug.
The batch cancellation timeout now reports the last status the proxy returned.

* fix(ci): scrub the host environment per unit test instead of for the whole pytest process

GHA shards run tests/unit next to other suites in one process, so the import-time
scrub deleted MCP_TEST_PEER_PYTHON before tests/mcp_tests read it and the MCP
upstream fell back to the SDK2 interpreter. The two websearch tests that called
OpenAI and Perplexity live are removed: tests/unit no longer sees their keys.

* fix(ci): scrub only the host variables present before litellm is imported

The per-test scrub also deleted TIKTOKEN_CACHE_DIR, which litellm sets at import to
its bundled encodings, so tokenizer paths tried to download them and hit the
socket guard. The prisma setup test now passes its own database URL instead of
reading one another test leaked into the process environment.

* fix(ci): stop the order-dependent unit reds and settle logging tasks on their own queue

LoggingWorker marked a task done on whichever queue was current when the callback
finished, so a callback that outlived an event-loop change raised "task_done()
called too many times" or undercounted the new loop's queue. It now settles the
queue the task came from.

The rest are test isolation fixes for failures that only appeared when another
file ran first on the same xdist worker: a replaced user_api_key_cache, breaker
metrics unregistered by prometheus tests, semantic_router's health-check filter on
uvicorn.access, logging tasks carried over from bedrock tests, a Router-written
model_cost entry, and a stray post captured by the langflow test. The token
counter check now asserts bounded chunking instead of wall-clock time.

* test(e2e/ui): wait for the logout redirect before visiting a protected page

Logout revokes the session server-side before clearing cookies and navigating, so an immediate page.goto either ran with the cookie still set or was aborted by the logout redirect (net::ERR_ABORTED).

* test(unit): restore the prometheus metrics config per test and settle logs carried from earlier tests in the a2a cost tests

* test(router): pin the router clock in the usage counter tests so a minute rollover cannot empty the read

* test(e2e/ui): wait for logout to clear the token cookie instead of for a login redirect

* test(integration/mcp): answer the model-info probe another test's proxy sends to the model double
2026-09-26 09:25:13 -07:00
joshua-berri
6b7688869e
feat(mcp): configure protocol versions and capability discovery (#43169)
* feat(mcp): configure protocol versions and capability discovery

* test(mcp): return SDK initialization result in REST pagination fixture

* test(mcp): arm cancellation deadlines after TCP calls start

* fix(mcp): avoid serialized discovery and listing spend logs

* test(mcp): isolate default protocol header policy

* fix(mcp): honor preview protocol pins and refresh migrated tests

* fix(mcp): preserve edited protocol pins in saved OAuth previews

* fix(mcp): retain saved protocol pins when previews omit versions

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
2026-09-25 13:13:52 -07:00
yuneng-jiang
2cbaa4c9b5
test(mcp): stop a comprehension variable from shadowing the body() helper (#42906)
test_malformed_bodies_missing_users_and_foreign_servers_are_rejected used
`body` as a comprehension variable and then called the module-level `body()`
helper a few lines later. CPython 3.12.2, which the CircleCI integration job
runs, compiles that later call as a local read, so the test raised
UnboundLocalError on every integration-mcp run since #42652. Newer 3.12
patch releases and 3.13 compile it as a global read, which is why it passes
locally.

Renaming the comprehension variable makes both reads unambiguous.
2026-09-24 11:56:56 -07:00
devin-ai-integration[bot]
ce582affaa
fix(mcp): reject duplicate MCP server names and aliases (#42791)
* fix(mcp): reject duplicate MCP server names and aliases

MCP server_name and alias were unchecked at write time, so two servers
could share one tool prefix and tool routing resolved to an arbitrary
winner. Writes now run inside an advisory-locked transaction that
rejects a collision on either column case-insensitively with a 400
naming the colliding identifier, covering create, edit, connector
import and restricted-admin submission. Server reload logs one warning
per identifier already shared in the database.

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(ui): block duplicate MCP server names and aliases before submit

The create and edit forms now check the normalized name/alias against
the loaded server list (case-insensitive, spaces to underscores, own
row excluded on edit) and show a field error instead of submitting.
Structured proxy error bodies are unwrapped so a 400 no longer renders
as 'Error: [object Object]'.

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(mcp): check identifier conflicts when an alias is cleared

Clearing an alias drops the tool prefix to the stored server_name, so
that name must go through the conflict check too; an explicit alias:null
is now treated as an identifier write. Also narrows the new db tests to
behavioral assertions instead of pinning prisma where shapes.

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(mcp): treat an empty alias as a clear in conflict checks

An empty-string alias was written unchecked even though the prefix falls
back to server_name; the update path now treats any falsy alias like a
clear. The edit form likewise compares a cleared alias as empty instead
of re-checking the alias being removed.

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(mcp): cover clearing an alias to an empty string

Co-Authored-By: bot_apk <apk@cognition.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
2026-09-23 17:50:27 -07:00
devin-ai-integration[bot]
6593566cdb
fix(key_management): invalidate cached object permissions on key update (#36719)
* fix(key_management): invalidate cached object permissions on key update

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(key_management): broadcast permission cache eviction and cover key regenerate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(key_management): evict object permission rows through evict_and_broadcast

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(integration): cover mcp tool permission widen, narrow and clear on both workers

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(key_management): evict object permission rows before the key object

Co-Authored-By: bot_apk <apk@cognition.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
2026-09-23 15:55:09 -07:00
devin-ai-integration[bot]
4b50e8b236
fix(mcp): keep tool attribution on guardrail-blocked REST calls (#42790)
* fix(mcp): keep tool attribution on guardrail-blocked REST calls

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(proxy): keep content enforcers in the pre-call walk when guardrails are skipped

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(proxy): accept skip_guardrails kwarg in pre_call_hook test doubles

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(proxy): drop section comment flagged by repo comment policy

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(proxy): drop docstrings from skip_guardrails tests

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(proxy): wrap pre_call_hook mocks under the line limit

Co-Authored-By: bot_apk <apk@cognition.ai>

* refactor(proxy): drop skip_guardrails docstring sentence

Co-Authored-By: bot_apk <apk@cognition.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
2026-09-23 15:40:56 -07:00
devin-ai-integration[bot]
49d0ece934
fix(mcp): forward caller bearer on REST oauth_delegate tool calls (#42787)
* fix(mcp): forward caller bearer on REST oauth_delegate tool calls

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(mcp): only forward caller bearer on REST for client-forwarded-token servers

Co-Authored-By: bot_apk <apk@cognition.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
2026-09-23 14:27:54 -07:00
devin-ai-integration[bot]
dc14e76147
fix(mcp): return 401 challenge for REST token-exchange tool calls without a subject token (#42782)
* fix(mcp): return 401 challenge for REST token-exchange tool calls without a subject token

Co-Authored-By: bot_apk <apk@cognition.ai>

* fix(mcp): keep tool_server_mismatch when server_id disagrees with the tool prefix

Co-Authored-By: bot_apk <apk@cognition.ai>

* test(mcp): type the token-exchange challenge test helpers

Co-Authored-By: bot_apk <apk@cognition.ai>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
2026-09-23 14:13:16 -07:00
devin-ai-integration[bot]
5db8543817
fix(anthropic): preserve MCP tool results in the non-Anthropic Messages bridge (#42783)
The tool_result user message built by the /v1/messages MCP loop used tuple
content, which the Messages to Chat Completions adapter silently dropped, so
non-Anthropic models re-requested the tool until the iteration cap or the
provider rejected the follow-up. Emit list content so the existing tool_result
branch translates it into a role tool message keyed by tool_call_id.

Resolves LIT-8474

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: bot_apk <apk@cognition.ai>
2026-09-23 13:03:25 -07:00
devin-ai-integration[bot]
9fd25b2228
fix(ui): keep per-user MCP credentials updatable and clearable after setup (#42652)
* fix(ui): keep per-user MCP credentials updatable and clearable after setup

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): keep card keyboard activation off nested credential buttons

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): confirm before clearing saved per-user MCP credentials

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(ui): reset the clear confirmation when the credentials modal closes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): cover per-user env var edge paths in browser and integration contracts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): wait for the peer process to grant the key before listing its MCP tools

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(e2e): drop the mutable removed flag from the deleted-server browser contract

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 11:51:05 -07:00
devin-ai-integration[bot]
e26a6450c8
test(integration): add MCP gateway coverage wave 1 with a dedicated mcp shard and proxy coverage artifact (#42711)
* test(integration): drop the contracts.json manifest and the covers requirement

Groups live as a GROUPS literal in run.py, the browser expectations move next to the
browser tests, and the runner fails only on pytest failure, collection errors or a
selected file that collects zero tests. The covers marker stays registered for the
existing tests but is no longer checked. The mcp directory gets its own group

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): run mcp as its own shard with xdist and a peer proxy

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): INTEGRATION_COVERAGE=1 runs the proxy under coverage for the MCP modules

The mcp shard sets it. The proxy and its peer start under coverage run in parallel mode,
get SIGTERM after the tests so coverage flushes, and the combined text and HTML reports
land in the suite results that CircleCI already stores as artifacts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(integration): let the test proxy flush coverage when uvicorn re-raises SIGTERM

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add SSE, stdio, scripted, OpenAPI and OAuth 2.1 MCP peer doubles

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP transport and access-control matrices

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP credential and OAuth flow coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): add MCP LLM endpoint, accounting, guardrail, resilience and lifecycle coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): stop the same-URL grant test from counting a late initialize as a leaked call and satisfy the test-tree lint

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): assert the REST denied-server listing is refused or empty

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): pin the REST denied-server listing to 403 access_denied

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 07:48:46 -07:00
Joshua Valluru
fdc4ad54c5 docs(mcp): remove duplicate regression coverage inventory 2026-09-19 18:18:37 -07:00
Joshua Valluru
479360ae87 test(mcp): restore scoped execution and credential isolation regressions 2026-09-19 16:41:31 -07:00
Mateo Wang
57c757e9d7
Revert "test(mcp): verify scoped execution and OAuth credential isolation" 2026-09-19 16:10:47 -07:00
Joshua Valluru
a41b60cf77 test(mcp): align live regressions with discovery and error contracts 2026-09-19 13:03:00 -07:00
Joshua Valluru
e0b6bae516 test(mcp): cover scoped execution and OAuth credential isolation 2026-09-19 12:14:32 -07:00
Devin AI
6247b75543 test(mcp): keep the import block as merged on main
Co-Authored-By: bot_apk <apk@cognition.ai>
2026-09-19 00:51:51 +00:00
Devin AI
e3755a88e7 test(mcp): assert the misconfigured credential message on fail-closed rejections
Co-Authored-By: bot_apk <apk@cognition.ai>
2026-09-19 00:50:00 +00:00
Joshua Valluru
f83992f786 test(mcp): reject every unauthorized server in health results 2026-09-17 18:44:20 -07:00
Joshua Valluru
15b45839e1 test(mcp): enforce security regression contracts through live gateway 2026-09-17 17:37:40 -07:00
Yuneng Jiang
c00f1b4a5c
test: add extension and browser integration contracts
Some checks failed
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
LiteLLM Rust / rust-lint (push) Has been cancelled
LiteLLM Rust / rust-test (push) Has been cancelled
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
2026-09-16 13:15:53 -07:00