* test(responses): audit #47445
Integration cells for billing every round of a streamed MCP auto-execute on one spend row, on a native Responses upstream and the chat bridge, across tool rounds, a rejected or broken follow-up, a post-call guardrail block, a client hang-up, the response cache, and the next bridged turn
* test(responses): check a broken follow-up bills its own nonzero usage
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Adds a real-proxy case where the MCP follow-up call fails on every router retry, checking that the one failure row bills the first call of every attempt on chat completions and /v1/responses
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover the max_parallel_requests slot a streamed /v1/messages MCP auto-execute keeps
A streamed /v1/messages request that auto-executes MCP tools replays its answer
as a stream after its model rounds already logged success. The stream path frees
the request's max_parallel_requests slot only on a client disconnect, so after
message_stop the slot stays taken and the key's next request answers 429
The test streams such a replay to the end and expects the slot to be free. It
fails until the fix lands
* fix(proxy): free the max_parallel_requests slot when a replayed /v1/messages MCP stream completes
A streamed /v1/messages request that auto-executes MCP tools runs its model
rounds without streaming and replays the final answer as a stream. The first
round claims the request's only success log, and its limiter callback keeps the
slot because the tool call reused it afterwards. The stream path released the
slot only on a client disconnect, so after message_stop the key stayed at its
max_parallel_requests limit and its next request answered 429
Stream cleanup now releases the stashed slot when the stream completed and its
logging object already claimed the success log, which only happens when the
stream replays a response that was logged before it started. A real stream
still releases through its own success callback
* test(mcp): free the key's only parallel slot after a streamed auto-execute ends
Run a streamed MCP auto-execute on a key with max_parallel_requests 1 on
every LLM surface, then send the next request on that key and expect 200.
On main both /v1/messages surfaces answer 429 here, while /v1/responses and
/v1/chat/completions already pass
* test(proxy): keep only the MCP stream regression tests, since #48628 already frees the slot at stream end
* test(proxy): require the MCP stream regression test's response to stream and reach its end event
* refactor(mcp): drop the experimental label from the MCP server and client paths
Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims
pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): upgrade to MCP SDK 2.3.0 and raise the per-event SSE size cap
MCP SDK 2.x reads Streamable HTTP responses through httpx2's SSE parser,
which rejects any single event over 1 MiB, so a large tools/call result
failed with 'SSE stream ended without a response'. mcp 2.3.0 exposes
max_sse_event_size, and MCPClient now passes a 64 MiB default,
configurable with LITELLM_MCP_MAX_SSE_EVENT_SIZE_BYTES
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): fall back for invalid SSE event limits
* test(mcp): verify configured SSE limits and recovery
* chore(mcp): remove redundant dependency comment
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* test(mcp): cover one spend row billing both rounds of a chat completions MCP auto-execute
* fix(mcp): bill both model calls of a chat completions MCP auto-execute on one spend row
* fix(mcp): keep provider usage fields when summing both chat rounds of an MCP auto-execute
* fix(mcp): log each auto-executed MCP tool call under its own spend-log id
A chat MCP auto-execute logged its tool calls under the request's call id, which is also the id of the request's failure row. When the follow-up call failed, the database kept the tool row and dropped the failure row that bills the paid first call
* test(mcp): cover the margin, cache, failure row and parallel tool rows of MCP auto-execute billing
* fix(mcp): keep the gateway handler off the SDK path when no tool is an MCP tool
completion(), responses() and anthropic messages imported the MCP gateway handler
whenever tools were passed, and responses() did so on every call. Since the handler
began importing the MCP catalog at module level in #42568, that import needs the
optional mcp package, so a base SDK install failed with ModuleNotFoundError on any
function-tool call and on every responses() call
A new import-safe pre-check skips the handler unless some tool is a dict of type mcp,
which is the only shape the handler's gateway decision can accept. MCP gateway tools
still reach the handler as before
Adds in-process regression tests for the three entry points and a function-tool
completion to the base SDK install check
* fix(mcp): leave one-shot tool iterables unread in the gateway pre-check
responses() declares tools as an Iterable and the handler scans the same
object a second time, so a pre-check that read a generator would hand the
handler one tool fewer. The pre-check now scans only collections and reports
anything else as possibly naming the gateway, which keeps the previous path
The Anthropic messages regression test now mocks the httpx layer with respx
and turns off the aiohttp transport, instead of replacing AsyncHTTPHandler
* fix(mcp): scan only lists and tuples in the gateway pre-check
The pre-check read any Collection, but a Collection that yields its items only
once would still be exhausted by the handler's second scan. Only the list and
tuple builtins are guaranteed to iterate the same way twice, so they are the
only inputs read now. Everything else keeps the previous path and is reported
as possibly naming the gateway
The unread-input test now covers a generator, an iterator, a one-shot
collection and a deque
* fix(mcp): scan only exact list and tuple types in the gateway pre-check
isinstance accepted list subclasses, and a subclass that iterates only once
would still be consumed by the handler's second scan. The fast path now reads
only objects whose type is exactly list or tuple. Every other input, subclasses
included, keeps the previous path and is reported as possibly naming the gateway
The unread-input test gains a one-shot list subclass and a tuple subclass
* fix(sdk): decide MCP gateway routing without importing the proxy handler
The pre-check now holds the handler's exact gateway test, so remote MCP tools, mcp 1.x installs and any tools iterable stay off the handler, and the base SDK check calls responses() with a remote MCP tool.
* fix(sdk): send /mcp/<name> remote MCP URLs to the provider when the gateway registry cannot load
A path-only /mcp/<name> match now counts as a gateway tool only when the MCP server registry imports, so SDK installs without the proxy extras pass remote MCP servers like Zapier to the provider. Also drops a dead test patch.
* test(sdk): send a /mcp/<name> remote MCP tool in the base SDK install check
* style(sdk): drop docstrings from the MCP gateway pre-check and its tests
* test(mcp): cover /mcp/<name> gateway routing on the proxy and Messages tools on a base SDK install
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* test(mcp): move the MCP OAuth happy path from a live Linear e2e to an integration test
GitHub Actions now runs unit tests only, and the MCP OAuth happy path needed a
captured Linear browser login to click through a real consent page. The OAuth
flows it exercised are already integration tests against an owned
authorization-server double, except one behavior: the stored per-user token
must be served by a proxy process that never saw the consent, with no shared
Redis to fall back on. That case is now in test_mcp_oauth_flows.py
The e2e test, its owned-gateway helper, its workflow, the mcp_oauth_live marker
and the JUnit gate script only that workflow used are removed
* test(mcp): chat completions run an OAuth server tool with the caller's own stored token
* test(e2e): drop the Linear chat-completion MCP OAuth e2e and its helpers
The behavior it checked, a chat completion running an OAuth server's tool with
the caller's stored per-user token, is now an integration test in
test_mcp_llm_endpoints.py that needs no saved Linear browser login
* test(mcp): decode the stored per-user credential as urlsafe base64
The plaintext check used b64decode, which drops the - and _ of a urlsafe value
and raised on roughly one ciphertext in five
* test(mcp): drop the chat OAuth integration test that duplicates test_mcp_slots F2-chat
security/test_mcp_slots.py already sends a per-user OAuth token through
/v1/chat/completions and asserts the peer receives it. Every mutant the new test
killed was also killed by the existing suite, so it added no signal
* fix(test-harness): preserve sibling auth modules in isolated worker configs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect servers in MCP connect-flow descriptions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): expect ptu_hours in daily activity metrics after #43043
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): kill every serving worker once the SCIM push blocks on the row lock
* fix(cost): price exact cost-map names before price-free capability rule matches
* test(streaming): price a priced alias stream at its deployment model and keep the unpriceable case unmapped
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): price an explicit base_model under its own provider after candidate reordering
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): retain priced generalized candidates and skip explicit-pricing lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(cost): assert custom pricing makes no remote model-info requests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): keep the per-candidate pricing provider final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): pin a Bedrock application profile prompt for the one-hour ttl its base model sends
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the Nova Sonic tool-call argument delta carries the full arguments
* fix(cost): sort cost candidates from the static cost map and rules, without provider lookups
* test(integration): bill route-priced Bedrock chat streams and responses, and pin application profiles for an hour on messages and cache hits
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* fix(responses): log and bill every round of non-stream MCP auto-execute
Both model rounds run as internal sub-calls on the request's logging object, and the outer call logs the response the client received with usage, cost, and cost breakdown summed across the rounds. The internal-call gate now also covers the sync success callbacks, so an inner round no longer takes their first-wins slot
* fix(caching): let the parent call log and bill an internal sub-call's cache hit
A round of non-stream MCP auto-execute that the response cache served fired the
cache-hit callbacks on the shared logging object, so the spend log kept round 1's
function_call row and dropped the billed row for the answer. Internal sub-call cache
hits now fire no callbacks and carry a zero cost, and the summed cost breakdown leaves
out rounds served from the cache
* fix(logging): keep a parent row unique when its response came from the response cache
A RAG query or an MCP turn whose final round is served from the cache hands back the cached response's id, which already belongs to the row that cached it, so SpendLogs dropped the new row as a duplicate. Such a row now gets the cache-hit id suffix, and it is logged as a cache hit when it cost nothing.
* fix(logging): skip the zero-cost warning for a response served from the cache at no charge
A non-stream MCP auto-execute request whose rounds the response cache served is priced
at $0 before the parent marks it a cache hit, so it logged a false 'priced at $0'
warning and counted in litellm_zero_cost_requests_total. The diagnostic now treats a
response the cache served at no charge the same as a marked cache hit
* fix(responses): bill the paid first MCP round when a later step of the request fails
* fix(responses): log each MCP tool call under its own id so a failed follow-up keeps its spend row
The non-stream auto-execute path passed the request's call id to the tool call, so the tool's spend row and the request's failure row shared one request_id and the failure row was dropped on insert, leaving the paid first round unbilled.
* fix(responses): bill round 1 on a client disconnect, keep MCP tool rows in the request's session, and charge a fixed margin once per request
* fix(responses): keep bridged MCP auto-execute in one session and trace and continue its conversation
The bridged follow-up no longer chains on round 1, so a thinking model keeps its reasoning and the request stays in one session. The tool call log continues the request trace without renaming it, the tools are fetched once, and a later turn finds the MCP turn under the id the client got, skipping the tool rows.
* fix(responses): log the non-stream MCP tool listing in the request's session and trace
* fix(responses): find a bridged MCP turn by the id the client holds and log its listing in the bridged trace
* fix(responses): derive the MCP tool row id from the request's call id
The tool call's spend row now uses <call id>_<tool call id>, so it stays
linked to the request without colliding with the request's failure row.
Also tighten the internal cache-hit test to a fixed clock and typed params.
* fix(responses): bill a retried MCP attempt for its own rounds
A router retry or fallback reuses the request's logging object, so a failed attempt's stashed round 1 usage made the next attempt's success row log stale tokens. Each non-stream MCP attempt now starts without that stash.
* fix(responses): bill every failed MCP attempt's paid first round on the request failure
A retry or fallback now sets a failed attempt's stashed round 1 aside instead of dropping it, and the proxy's failure row adds it to the last attempt's own, so a request whose later attempt fails on its first call still bills the round the provider charged. A cached final round also keeps the paid first round's cost breakdown.
* fix(responses): keep MCP tool catalog and results out of the bridged next-turn replay
* fix(responses): keep MCP tool spend, client body keys, and Langfuse traces working on auto-execute
Count the MCP tools an auto-executed Responses API call ran in the tool spend rollup, accept a client body key named previous_response_client_ids without a 500, and point MCP listing and tool spans at the trace Langfuse gives the parent request, including x-litellm-session-id requests
* fix(responses): join MCP spans to the Langfuse trace when the request sends body metadata
* test(integration): cover non-stream MCP auto-execute billing, failure rows, and bridged follow-ups
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
* fix(responses): open faked stream items empty so tool arguments and text arrive once
* fix(responses): open faked stream items in progress with a fresh empty list, and open mid-stream guardrail blocks empty
* fix(responses): open faked refusals, reasoning summaries, custom tool calls and MCP calls empty so each value arrives once
* test(integration): cover the empty-open Responses stream items on every fake, replay and block path
Integration cells for the faked /v1/responses stream, the chat and messages bridges, cache replay, guardrail block events and the MCP gateway call. Each opened item starts empty and each value arrives once, checked with raw httpx, the OpenAI SDK and the Anthropic SDK, plus upstream errors, 5 KB values, a client drop, a dropped upstream connection and a worker SIGKILL
* test(integration): cover mcp_runtime endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(integration): ruff format and import order in mcp runtime tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert out-of-path calls and untrusted forwarded headers are refused
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): move the untrusted forwarded-header discovery case into its own test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): hash the token row lookup, widen spend polling, and union open servers after listing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tighten mcp runtime contracts per review (sdk client, toolset scope, forwarded trust, callback html, session tools/call)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert the sdk unknown-path error without pinning the sdk's exception wrappers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover trailing-path and group csv mcp paths, rfc 9207 iss-less and error-iss callbacks, and id_token exchange
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover lite logout spelling, team-mapped and unmapped IdP subjects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover per-server mcp-session resource and root /token spelling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): unpack the sealed client code in the step-up tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover mcp_management endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): require server_not_found for pending and rejected submissions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin per-worker submission reloads, toolset and alias-collision spellings, raw key overrides
Assert the approving and rejecting worker immediately and only wait on the peer, add an outsider key and exact upstream calls, list by toolset_name, cover a config alias shadowing a DB server_name, send key_hash as a raw sk- key, and add BUG skips for API disallowed_tools and overrides on keys or teams without object_permission
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin XFF trusted-peer, load balancer chains and sibling refusal body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): enforce output_policy trust chain, JSON-RPC XFF views, MCP tool-policy BUG pins
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): force a registry reload during the pending window, pin the admin default listing and the whole tools/call body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give chaos and timing tests a harness API and gate raw signals
Chaos tests killed and suspended proxy workers with raw os.kill and psutil
calls and bounded elapsed time with fixed literals, so they raced the uvicorn
worker healthcheck and failed on slow CI boxes
OwnedProxy now offers kill_worker and replacing_worker, which return only once
a replacement answers readiness with its own pid, and suspended, which is
refused unless the proxy was launched with a lenient worker healthcheck.
timing.py bounds durations against a measured control or the injected delay.
run.py measures the proxy import time once and derives readiness deadlines and
the per-test timeout from it, writing boot-calibration.json
check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time
asserted under a numeric literal) for tests/integration outside _support, and
every call site is migrated so both report zero with no suppressions
* test(integration): steer the assistants wire burst onto both workers
The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split
* test(integration): restore product-enforced timing bounds lost in the migration
Some TQ011 migrations dropped or loosened a bound whose time is the behaviour
under test. The websocket session cap only kept its floor, so a cap closing at
300 s would have passed, and several timeouts were bounded by the injected stall
instead of the configured product value
assert_enforced_within bounds an elapsed time by the configured value plus
explicit slack, with an optional floor. The websocket cap, the MCP validation
timeout, the guardrail litellm_params timeout, the prisma command timeouts, the
baseline projection lock timeout and the key metadata statement timeout use it.
The device code refusal keeps its 15 s margin, and the langfuse retried prompt
fetch is now bounded by a cold fetch of a steady prompt as well as by the
ignored retry-after
* fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations
The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat
in a ping on the killed worker for up to that long before respawning it. It now uses
the graceful stop budget, and a worker kill returns only after the victim is reaped,
so pooled connections no longer race the dead worker's socket.
Several tests on main picked the multiprocessing resource tracker as their victim and
never killed a worker. Now that they kill a real one, presidio counts streams cut by
the death, delete_model waits for the config reload that removes a deleted model from
a replacement worker, and the wedged coordination Redis test kills before the wedge
and waits for every parked publish to fail before resuming Redis.
The redis cluster test imports the harness through the tests package so it collects
in the unit job, and the per-model fallback rig exposes the owned proxy it now needs.
* test(observability): only require S3 deliveries the killed worker could not have buffered
A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
The probe test asserted that every request the peer saw carried the forwarded token, so a credential-less tool discovery session from either gateway (startup mapping, an OpenAPI registration elsewhere, or the 30s MCP reload) landing in the window failed it intermittently on the CircleCI mcp layout
The caller now sends an allowlisted x-integration-caller marker, which the gateway forwards on the caller's listing and call sessions but never on the probe or on background discovery. Probes and marked requests must carry exactly the forwarded token, the single tools/call must be one of the caller's, and anything left over may not carry any Authorization at all. The admission key scan still covers every request
Verified on a local two-gateway rig sharing Postgres and Redis while a thread registered and deleted OpenAPI servers on both gateways. The old assertions failed 207 of 240 runs there with the CI signature, the new ones 0 of 240. Dropping the token on the call session, on the listing session, or on the probe still fails all six cases
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* test: migrate proxy security, schema migration and MCP legacy tests
* test: run the MCP proxy e2e upstream under the SDK1 peer in integration-mcp
* test: warm the baseline store before timing the locked projection
* test(e2e): hold the stack exclusively while the semantic filter flag is flipped
* test(mcp): scope the in-process proxy e2e fixtures to their module
The session-scoped autouse fixture deleted DATABASE_URL and overrode LITELLM_MASTER_KEY for the rest of the worker, which broke every later test in the integration mcp suite
* test(e2e): assert the semantic filter keeps listed tools instead of a ranking
Which Datadog tool an OpenAI embedding ranks first is not something litellm owns, so the test now checks that the kept tools are a non-empty subset of the key's listing within top_k
* test(mcp): keep the in-process proxy e2e tests on one worker
loadgroup spread the file's tests across workers and interleaved them with other files, so the module-scoped in-process proxy was torn down and restarted between tests and requests timed out
* test(mcp): name the upstream requests that carried the wrong token
The bare all() assertion hid which request broke the forwarded token, which made an intermittent CircleCI failure undiagnosable
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* test: move guardrails, image, audio, batches, search legacy tests to unit
* test: restore the custom guardrail and remaining chunk 1 unit rows
* test: move the ocr matrix to e2e
* test: drop comments from moved tests and fix vertex ocr credentials
* test: move litellm utils and unified google legacy tests to their tiers
* test: move logging callback and pass-through unit legacy tests to their tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move store_model_in_db legacy tests and start the admin ui migration
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: finish the proxy job rows and drop their CircleCI jobs
Moves the remaining multi-instance, openai endpoint, otel, pass-through and
top-level proxy tests into tests/integration with scripted upstreams, rewrites
the weak replacements to assert the outbound wire and exact responses, and
removes the build_and_test job group this PR owns from CircleCI
* test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names
* test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist
* test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in
* test: type the e2e beta header, prompt cache and secret manager values
* test: assert the missing model error message on the OpenAI routes
* test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks
* test: make the langfuse fixture directory a package for the layout check
* test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane
* test: keep the ocr e2e pdf fixture next to the test
* test: put meta below covers on the streaming tool search e2e
* test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures
* test: count only OCR calls in the OCR matrix recorder
* test: pin OCR parse bodies and billed cost, group new e2e cases into classes
* ci: restore the python image anchor dropped in the main sync
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(ci): distribute unit cases and reuse security sweep connections
* revert(ci): reject first speed trial after new failures
* perf(ci): avoid repeated test discovery and parsing
* perf(ci): isolate parallel security workers
* revert(ci): restore measured security execution isolation
* perf(tests): reuse source scans and deduplicate cache cleanup
* perf(ci): restore parallel security with safe log batching
* test(ci): isolate fixture state and schema comparisons
* test(ci): make schema and scheduler fixtures deterministic
* test(ci): drain queued logging before fixture teardown
* test(ci): preserve logging delivery without idle worker restarts
* test(ci): keep guardrail sinks available during parallel bursts
* test(ci): isolate full proxy lifecycle from gateway imports
* test(ci): preserve first failure responses and sweep evidence
* test(ci): restore documentation URL settings between cases
* test(ci): isolate open MCP tools in private gateways
* test(ci): group private MCP cases and handle exited workers
* test(ci): retain proxy exits and MCP failure details
* test(ci): repair merge interactions with main
Key the schema shard's full clone on artifact-name, since main removed the
unit-flag input. Give each owned Redis its own log file again. Hand the
realtime transcription tests main added a logging object whose dispatch
handlers are awaitable, so the per-test logging drain no longer waits on a
MagicMock task forever
* perf(ci): shard owned integration cases and close lifecycle leaks
* fix(ci): qualify shard execution and harden interrupted cleanup
* fix(ci): bound worker supervision and own fault traffic
* perf(ci): reuse unparametrized fixtures across bounded case groups
* fix(ci): preserve boot fault allowances and complete exit records
* test(ci): tolerate startup retries and retain readiness details
* perf(ci): balance integration shards by recorded duration
* fix(ci): load migration database helpers only for workers
* fix(mcp): never forward the caller's LiteLLM key to MCP servers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): tidy caller-key scrub
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): avoid master-key scanner false positive
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): match admission when scrubbing the caller key
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): drop header copy churn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): normalize bearer variants in caller-key match
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): inject admission header name and cover provider keys
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): exercise real REST header extraction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): cover caller-key scrub on the real REST path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): exclude gateway admission header from upstream forwarding
* fix(mcp): skip empty admission headers when scrubbing keys
* fix(mcp): use scrubbed server credentials for auth probes
* fix(mcp): match probe credential precedence to upstream calls
* test(mcp): simplify probe header assertion
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
* feat(guardrails): support logging_only mode for the Akto guardrail
* test(guardrails): wait for the spend row before reading Akto calls and cover mixed modes
* test(guardrails): cover Akto logging_only on MCP tool calls
* test(guardrails): type the Akto logging_only unit tests and inject the HTTP handler
* test(guardrails): cover failing Akto replies, provider failures and a mixed outage burst under logging_only
* test(guardrails): cover an unreachable Akto with fail_open under logging_only
* test(mcp): fix stale bridge-hook, applied-guardrails and pagination-revoke integration tests
* test(mcp): clear the spare direct grant when restoring the access-group policy
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(mcp): refresh server catalog for each gateway operation
* fix(mcp): reject configuration changes during scoped dispatch
* fix(mcp): refresh shared catalog state for each operation
* fix(mcp): refresh catalog before native alias routing
* docs(mcp): clarify native route resolution order
* test(mcp): align streaming fixture and generated API documentation
* fix(mcp): preserve discovery published during catalog refresh
* fix(mcp): coalesce queued catalog reads without a stale window
* fix(mcp): reconcile concurrent route changes when publishing catalog
* test(mcp): provide catalog scope in post-call hook fixtures
* fix(mcp): retain valid live routes and handlers during refresh
* fix(mcp): keep refreshed OpenAPI operation membership authoritative
* test(mcp): preserve logging fixtures after catalog integration
* fix(mcp): isolate transport test admission state and exhaust OAuth outcomes
* refactor(mcp): narrow catalog consistency change to ticket scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(mcp): gate catalog refresh on a database revision marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): refresh waiters that observed a newer catalog revision
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(mcp): awaitable catalog revision doubles in prisma mocks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(mcp): satisfy catalog refresh lint and type gates
* test(mcp): provide catalog scope in toolset fixtures
* test(mcp): exercise temporary OAuth through catalog operations
* fix(mcp): share active catalog snapshots across discovery tasks
* fix(mcp): retain discovered routes across cached catalog operations
* fix(mcp): preserve local tool ownership across discovery
* refactor(mcp): satisfy tightened immutability lint ceiling
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): authorize local handlers by registered server ownership
* fix(mcp): check registered handler freshness and isolate fixtures
* fix(mcp): read catalog revisions and snapshots from the writer
* fix(mcp): resolve access groups from the operation catalog
* fix(mcp): preserve empty access group restrictions
* fix(mcp): retain rediscovered routes across concurrent updates
* fix(mcp): enforce registered ownership for local tool dispatch
* fix(mcp): preserve issuer discovery across catalog snapshots
---------
Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
