* fix(spend_tracking): key completion spend logs off the per-call id so duplicate provider response ids cannot drop rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): narrow the provider response id with isinstance instead of cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gcs_bucket): store request payloads under the spend log request id so the logs viewer can find them
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend_tracking): validate callback fields for current type gates
* test(gcs_pubsub): assert provider response id in spend metadata
* fix(ollama): normalize reused response ids automatically
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf streams take a reasoning or tool-call delta as the first token, tool-call replies had no time_to_first_token
* test(e2e): perf users start at a random offset and jitter each interval by up to 25%, no more lockstep bursts at the planned rate
* test(e2e): small_bodies_5rps sends from the mixed case's callers (two teams, JWT users, keys in and out of teams) through the pre and post guardrail
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
The route host now reports get_configured_request_timeout() in MessagesSettings, and prepare uses the call timeout first, then that setting, then the 600s default. Before this, litellm_settings.request_timeout and REQUEST_TIMEOUT were ignored on the native path, so a long turn that works on Python timed out after 10 minutes
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): add require_key_agent_access_defined for least-privilege agent access
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): align A2A model visibility assertions with supported routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep A2A model entries on their existing routes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): point the listing model at the harness upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): coerce require_key_agent_access_defined like the settings validator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject agent access lookups instead of patching the handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename require_key_agent_access_defined to agent_access_default_deny
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: ryan <ryan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): drop unknown legacy thinking keys under drop_params when rebuilding adaptive thinking
* fix(bedrock): drop unknown thinking keys under drop_params on display updates rebuilds
Bedrock Invoke copied every caller thinking key when adding display updates,
so an unknown key still reached the provider with drop_params on. Filter that
copy the same way, and cover each chat call site plus the global
litellm.drop_params path in tests
* fix(anthropic): drop budget_tokens from rebuilt adaptive thinking under drop_params
* test(anthropic): cover drop_params sources for rebuilt adaptive thinking on /v1/messages and Bedrock Invoke
* feat(gemini): continue MAX_TOKENS decodes with continuationToken across chat, streaming, and interactions
* fix(gemini): continue long decode on CONTINUATION and cancel interactions at /cancel
* feat(gemini): add public gemini-4-argon cost-map rows for AI Studio and Vertex AI
Add gemini/gemini-4-argon and vertex_ai/gemini-4-argon alongside the existing
early-access row, carrying the same limits and capabilities. The AI Studio row
mirrors the early-access entry; the Vertex row differs only in provider and
pricing source. Prices follow once Google publishes them.
Extend the mapped metadata test to cover all three ids, that each routes to its
provider, that its backup copy matches, and that the public rows carry the same
limits as the early-access row.
* feat(gemini): raise gemini-4-argon to the 1M long decode cap and report initial_prompt_tokens across hops
* chore(prices): build on the merged gemini-4-argon rows and only raise the long decode cap
* fix(gemini): follow continuation hops in a loop instead of recursion
The unary hop collectors and the stream iterator called themselves once per continuation request, which the recursion check in CI rejects and which would hit the interpreter's call depth on a very long output. They now loop, and a test follows more hops than the interpreter allows nested calls
* fix(gemini): bill Google Search grounding from every continuation request
Google sends search grounding once, on the request that ran the search, and does not repeat it on a continuation request. The merged response kept only the last request's grounding, so usage.web_search_requests and the per-search fee dropped every earlier search. Unary merges and the stream iterator now carry the grounding of every request in the chain
* fix(gemini): continue long gemini-4-argon answers on streams and Vertex AI
A continued stream stopped after the first chunk of its second request. The stream wrapper asks the iterator for a fresh start before every chunk, and that reset it to the first request's finished stream. The iterator now reads from the newest request's stream
On Vertex AI the continuation request went to the v1 API version, which rejects continuationToken with a 400. A request that carries the token now goes to v1beta1, which accepts it, including a token that v1 issued
On a stream, initial_prompt_tokens came from a usage chunk in the middle of the stream, which reports a smaller prompt count than the last chunk. It is now the count the first request's stream ends with, the same number a unary call reports
continuation_token is no longer forwarded on the Interactions API, which answers 400 for the field. It is dropped there as it was before this branch
* chore(gemini): name the type errors the dual-mode stream iterator trips
The stream iterator holds a sync or an async line iterator in one untyped attribute. Pointing it at a continuation request's stream gave the type checker a concrete union for it, so six existing lines that only ever run in one of the two modes started to report errors
This adds a rule-named suppression with a reason on each of those lines and changes no code. Splitting the class into a sync and an async iterator would remove them and is left for a follow-up
* fix(gemini): continue only when Google ends a request with CONTINUATION, and stop streaming mid-answer tokens
A long decode now sends the next request only when Google's finish reason is CONTINUATION, so an answer cut at the caller's own max_tokens returns in one request with its continuation_token instead of drawing a rejected extra request
A request for more than one choice is never continued, since Google rejects a continuation with more than one candidate
Chat-format streams no longer carry the tokens Google sends mid-answer. Only a stream that ends cut off carries a continuation_token, on its last content chunk
The response cache key now includes continuation_token, so a resend with a token is not served the earlier segment
* fix(gemini): keep one response id across a continued answer
A continued chat stream changed its chunk id each time the next request to Google started, and a unary answer came back under the last request's id. Every chunk and the unary answer now carry the first request's id, including the chunk that ends a cut-off stream
A request whose n or max_tokens arrives as a string is sent once and never continued, where it used to fail with a 500 on the comparison. An empty continuation_token no longer changes the cache key
* test(gemini): add integration cells for long decode continuation
Covers continued answers on chat, /v1/messages and /v1/responses, streamed and not, on the Gemini API and Vertex AI against a scripted Google, plus concurrency, outage and worker restart runs
Also moves the Google stand-in cancel route to /cancel, the URL this PR now calls for Interactions cancel, and checks the cancel test posts there exactly once
* fix(vertex_ai): fetch the access token again before each continuation request
A long Gemini 4 answer on Vertex AI can run for over half an hour of follow-up requests, and every one of them reused the bearer token fetched before the first request. Once that token expired Google answered 401 and the caller got a cut-off answer with finish_reason length
Each Google request in a chain now asks for the token again, which is a cache hit until the token nears expiry, on the streamed and unary paths, sync and async. A refresh that fails counts as a failed follow-up, so the caller still gets the answer so far with its continuation token. A caller's own Authorization in extra_headers still wins on every request, and the Gemini API key path is unchanged
* fix(vertex_ai): start each continuation stream with an empty parser buffer
When a Google stream ended on an unfinished frame, its leftover text stayed in the stream parser and every frame of the next continuation stream was appended to it. None of them could be decoded, so the whole continuation was billed and dropped without an error
Each continuation stream now starts with a fresh buffer, and a test covers a stream that ends on an unfinished frame, sync and async
* fix(vertex_ai): treat a continuation stream with no readable frame as a failed follow-up
A continuation stream that answered 200 and then sent Google's error frame first, sent a body that is not JSON, or dropped before its first frame ended the caller's stream with `stop` or an error and no token, so a cut answer looked complete or was lost. It now ends the way a refused follow-up does, with the answer so far, `finish_reason` length and that request's token
Every continuation stream request now also gets the caller's per-request timeout, which #46349 added to the first one, so a follow-up that never answers ends at that timeout the same way
* fix(vertex_ai): keep every follow-up's citations and URL context in a continued answer
A non-streamed answer that took several Google requests kept citationMetadata and urlContextMetadata from the last request only, and left every later request's offsets counting from that request's own text. So a URL read on the first request was missing from the answer, citations from earlier requests were dropped, and a later request's citations and search supports pointed at the wrong part of the joined answer
The joined answer now keeps every request's citations (Vertex AI's `citations` and AI Studio's `citationSources`) and URL context, and shifts each later request's citation and search support offsets by the UTF-8 bytes of the answer text sent before it, thought parts excluded, since Google counts these offsets in bytes
* fix(vertex_ai): answer when a candidate sends null citationMetadata or urlContextMetadata
A Gemini candidate whose citationMetadata or urlContextMetadata was JSON null made the call fail on every Gemini model, non-streamed with a 422 or 500 and streamed with an error event, because the per-request metadata helper handed the null back as is and the caller tried to extend a list with it. A continued answer also failed when a follow-up's citation or search support came with a null startIndex
A null field is now kept as one entry on a single request, as it was before this branch, a follow-up's null URL context adds nothing to a continued answer, and a null startIndex counts from the start of that request's text. Tests cover missing, null and empty fields on one request and on a continued answer, streamed and not
* fix(vertex_ai): pass through spans with no end index and point a continued stream's citations into the whole answer
A later request's citation or search support with no endIndex made the whole non-streamed answer fail with a 500, so a span Google sends without an end index now comes back as Google sent it, the way one request for the whole answer returns it. A span with an end index but no start index still counts from 0
A continued stream handed callers each later request's citation and search support offsets counted from that request's own text. The stream now adds the answer bytes of every earlier request, so the offsets point into the whole answer like the non-streamed path and one request for the whole answer
Adds the audit's wire tests for both, which fail 24 cells before this change
* test(vertex_ai): cover a later request whose cited span starts at its first byte with no start index
Google leaves out a zero startIndex, so the second request of a continued answer can cite its own opening with only an endIndex. These wire cells check that span lands where one request for the whole answer puts it, on Gemini and Vertex AI, chat and Responses, unary and streamed
* fix(vertex_ai): log a continued answer's raw response as one object
A non-streamed answer joined from several Google requests logged its raw response as a JSON list of every request's body. The OpenTelemetry logger reads that raw response as one Gemini body to set its `llm.gemini.*` attributes, so on a continued answer it hit a validation error and set none of them
It now logs the merged body, which carries only the last request's token, and a single request still logs Google's body unchanged. The new unit test fails with the change reverted
Also adds integration cells for a cut chat stream's token chunk, an answer joined from 41 requests, a caller that gave up on a non-streamed call, counts and caps that are not numbers or are sent twice, `n` sent as true, a partial answer cached after a refused continuation, the token in another provider's cache key, and the sync SDK path with its Vertex token refetch
* test(integration): check a continued answer logs one merged raw response
A non-streamed Gemini answer that took several Google requests now hands logging one merged response, so the OpenTelemetry span carries the llm.gemini attributes. These 12 cells fail on the merge base and on the commit before that fix, and pass on the branch
* test(vertex_ai): cover a follow-up tool call and its thought signature in long decode
* test(gemini): pin that a one-request answer logs Google's body unchanged
* test(integration): assert the streamed usage.cost matches the spend row on every cost case
* test(integration): type the streamed usage frames as JsonValue
* test(integration): only require streamed usage.cost when the client received a usage frame
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(anthropic): send detail on input_image parts bridged from /v1/messages to Responses
* fix(ui): send detail on playground Responses image parts and code snippets
* test(messages): cover image detail on the Responses bridge wire
* fix(oci): skip non-function tools and gate reasoning_effort on models that reject it
Fixes#31449.
Two OCI GENERIC request-building defects that made Grok-on-OCI unusable
for tool-using coding-agent clients (Codex over the Responses API, Claude
Code over the Anthropic Messages API):
1. `adapt_tool_definition_to_oci_standard` raised `OCIError("OCI only
supports function tools")` for any tool whose type is not "function",
aborting the whole request when a client mixed a built-in/server tool
in with its function tools. Non-function tools are now skipped with a
warning; if none remain, `tools`/`toolChoice` are omitted from the
request instead of sending an empty array.
2. The GENERIC param map forwarded `reasoning_effort` to every non-Cohere
model, but OCI rejects it for all served xAI Grok models
(`Model grok-4.20 does not support parameter reasoningEffort`) and for
OpenAI's non-reasoning models (`Unrecognized request argument supplied:
reasoning_effort` on gpt-4o / gpt-4.1); gpt-5, the o-series, gpt-oss-*
and Meta/Google models accept it. `_model_supports_reasoning_effort`
encodes that, `get_supported_openai_params` stops advertising the param
for those models, and `map_openai_params` follows the standard
unsupported-param contract (dropped under drop_params, clear error
otherwise) so `drop_params` / `additional_drop_params` now work as
documented instead of the 400 surfacing from the service.
Behaviour verified against OCI GenAI in us-chicago-1 (2026-09-11) per model.
The two existing reasoning_effort normalisation tests move from a Grok
model to openai.gpt-5, which is the case they exercise. 21 new tests; the
OCI suite is 406 passed.
* fix(oci): drive reasoning_effort gating from the model catalog and drop tool_choice for skipped tools
Review follow-ups on #40732:
- `_model_supports_reasoning_effort` now reads the `supports_reasoning` flag
of the model's `oci/` catalog entry through `get_model_info` instead of
hardcoding xAI / gpt-oss branches. Catalogued models without the flag
(xAI Grok, Meta Llama) no longer receive `reasoningEffort`; models absent
from the catalog keep the historical pass-through.
- Catalog: flag `oci/google.gemini-2.5-{flash,pro,flash-lite}` as reasoning
models and add `oci/openai.gpt-oss-120b` / `oci/openai.gpt-oss-20b`
(pricing from Oracle's public price list, 128K context per OCI docs).
- `_model_uses_max_completion_tokens` routes on the `openai.*` prefix only.
OCI-hosted Gemini accepts `maxCompletionTokens` but ignores it (the output
cap silently disappears) and Llama rejects it with 400, so the catalog's
reasoning flag must not drive that switch.
- After tool adaptation, a `FUNCTION` tool_choice naming a tool that was
skipped (built-in / server tools) is dropped with a warning instead of
being sent to OCI, which rejects the unmatched choice.
Verified live against OCI GenAI (us-chicago-1): Grok with mixed tools plus a
forced built-in now completes and still calls the function tool; Gemini and
gpt-oss forward reasoningEffort with maxTokens honoured; Llama drops it under
drop_params; gpt-5 keeps maxCompletionTokens.
* style(oci): use an immutable set for the tool-name lookup
Keeps the type-discipline gate (LIT002, mutable collections) within its
budget; the lookup is read-only.
* fix(oci): fail closed when a forced tool_choice names a tool OCI will not receive
Review follow-ups on #40732:
- When tool_choice forces a function that is not among the adapted tools
(a skipped built-in or a misspelled name), send `toolChoice: NONE` instead
of removing the choice. Falling back to the service default would let the
model call any other supplied function even though the caller forced a
specific one, which an agent processing untrusted content must not do.
- Move the #31449 regression tests into the mapped test module
(`test_oci_chat_transformation.py`) instead of a separate file.
Verified live against OCI GenAI: Grok and Llama with a forced skipped
built-in now complete with no tool call; a forced kept function is still
honoured.
* test(oci): point reasoning_effort comments at the in-module gating tests
The referenced module was folded into this file.
* fix(oci): drop non-function tools only under drop_params and gate reasoning_effort on declared levels
Non-function tools now follow the unsupported-param contract: a 400 naming
drop_params without it, and dropped with a warning under it, with a forced
dropped tool turning tool calls off. reasoning_effort is gated on the model's
reasoning_effort_levels catalog entry, so OCI Llama keeps forwarding it and the
Grok 4.20 models OCI serves, now in the catalog with their Oracle prices, stop
sending a param the service rejects
* fix(oci): gate reasoning_effort on oci/xai.grok-4.20, which rejects it
* fix(oci): disable tool calls when tool_choice forces a dropped custom tool by name
* fix(oci): drop tool_choice that forces a dropped tool and keep a kept function's name
* fix(oci): keep Gemini 2.5 Flash Lite unflagged for reasoning so released proxies keep its max_tokens cap
* fix(oci): answer 400 when a request depends on a dropped tool and leave Gemini max_tokens routing to its own PR
Under drop_params, non-function tools still drop when function tools remain and
tool_choice does not force a dropped tool. A request whose every tool is dropped,
or whose tool_choice forces one, now answers 400 instead of a tool-less reply the
model can fill with invented results, or a swap to another tool.
The maxTokens routing change for OCI Gemini goes back to base, since it is
outside this fix and an open PR carries it.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
GET /v1/mcp/server takes an optional user_id (admin, or the user themselves) and returns the union of the servers each of that user's teams reaches plus allow_all_keys servers. A user in no team gets the unscoped list, a deleted team still listed on the user is skipped, an unknown user is a 404 and a restricted virtual key is refused like for team_id. team_id keeps precedence when both are passed.
The dashboard MCPServerSelector passes the team and user it edits (key edit sends keyData.team_id and user_id, key create sends the picked team and owner, user edit sends the user) and narrows the server options to members of the access groups selected in the same control, keeping allow_all_keys servers and servers already granted so they can still be removed.
Resolves LIT-7784
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
fastapi_sso opens a bare httpx.AsyncClient for the non-PKCE token exchange and
userinfo call, so litellm_settings.ssl_verify and SSL_VERIFY never applied and
an IdP on an internal CA failed with an opaque 503. A LiteLLMHttpClientSSO mixin
replays that exchange on litellm's shared SSO client for the generic, Google and
Microsoft providers and surfaces IdP connection and TLS failures as a 401
ProxyException naming the endpoint.
Resolves LIT-9445
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): name the successor model for deprecating deployments
* chore(ui): add successor_model to the generated deprecation api types
* fix(proxy): keep the cost map successor when a deployment overrides only the date
* fix(proxy): read the cost map successor from metadata.successor
The cost map schema rejects a top-level successor_model key, and curated
entries already carry the successor under metadata.successor. Read that key,
and look it up even when the cost map entry has no deprecation date, so a
deployment that sets only its own date still names the successor.
* fix(proxy): keep the first deployment when deduping, let a blank successor hide the cost map one, and name a served successor by its model group
* fix(proxy): name only the deployment's own successor_model, since cost map successors are LiteLLM ids rather than served model names
* fix(proxy): leave out a successor_model the deployment's callers cannot call, with a warning
* fix(proxy): report successor_model as the admin set it and only warn when no model matches it
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): strictly honor disable_end_user_cost_tracking in SpendLogs
* fix(proxy): drop end-user id on failed requests and ClickHouse rows when end-user cost tracking is off
* fix(clickhouse): drop the end-user id from spend log metadata when end-user cost tracking is off
* test(spend): cover disable_end_user_cost_tracking across endpoints, failures, cache hits and chaos
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(anthropic): count a replayed server tool call's cache mark the way modify_params sends it
With modify_params on, a trailing tool search turn whose call has a kept result
got a placeholder answer from the factory and went out as a marked tool_use,
but the cache control hook counted marks before that placeholder existed and
skipped the mark, so a request at the cap reached Anthropic with 5 marks and
failed with a 400. The hook now finds client answers on the sanitized messages,
the same list the factory reads.
Also make the replay checks cheaper (prefix check before validation, early
return without provider_specific_fields, and look up client answers only when
a tool call carries cache_control), drop the two audit cells that cover old
gaps outside the tool search fix, and derive the pass-through test timeout
from the graceful stop budget.
* fix(anthropic): keep the duplicate tool result warning out of the cache control hook
The hook finds client answers with only the step of the modify_params sanitize
pass that pairs tool calls with their results, so a request with a duplicate
tool result logs the warning once again instead of once per mark count.
Also restore the Snowflake non-stream and cached-stream tool search cells as
known bugs with pytest.skip instead of deleting them, and put the Anthropic
pass-through test timeout back to 240 seconds.
* fix(anthropic): count cache marks next to a message the pairing step cannot read
With modify_params on and a marked tool call, the cache control hook paired tool
calls with their results before counting. A message without a get method (a
stray string or null in extra_body.messages, or an openai SDK
ChatCompletionMessage) made that pairing raise AttributeError before any
provider call, where main counted the marks and let the request through.
The hook now pairs only when every message can be read the way the pairing
step reads it, and otherwise counts on the messages as sent, like main did.
* fix(anthropic): count cache marks after placing a later system message the way the request builder does
* test(anthropic): cover a leading system prompt and a system message before the answer in the cache mark census
* test(anthropic): align the tool search replay cells with main and cover a system message after the search turn
The malformed kept result cells now expect the unanswered search call to be dropped, as main does since #45882.
The stream cut cell checks streamed tool call names, since the kept search result names get_weather as a tool
reference. The streamed response cache hit cell runs again because #46104 fixed it. The modify_params cap cell
expects the four client marks, since the placeholder answer sends the search call as a plain tool_use with its
mark, and a new cell covers a system message after the trailing search turn in both stream modes
* fix(anthropic): count marks on the raw messages when they cannot be placed, and find client answers once per request
* test(anthropic): cover marked tool calls next to a malformed tool id and a later system message
* perf(anthropic): look up client tool answers only when a marked tool call is a server tool call
Answered ids only decide whether a marked srvtoolu_ call is rebuilt as server_tool_use, so a request whose marked
tool calls are all client calls no longer places system messages or runs the modify_params pairing step to count
marks
* test(anthropic): reach the client answer lookup with a marked search call in the malformed tool id cell
* perf(anthropic): place mid-conversation system messages in linear time for models that keep them
* fix(anthropic): count cache marks with the model's own system message placement and pair tool results in linear time
* test(anthropic): cover modify_params dropping a tool answer that answers no call of the latest assistant turn
* fix(anthropic): count marks with in-place system placement when the model lookup cannot confirm the flag
* test(anthropic): cover a bare model with custom_llm_provider filling the cache mark budget through litellm.completion and Router
* fix(anthropic): give unmapped models the request builder's converting placement and survive any model lookup failure
* test(anthropic): cover an unmapped model name counting cache marks with the request builder's converting placement
* fix(anthropic): count server tool call cache marks the request builder's way only on routes that build Anthropic Messages requests
Databricks, OpenRouter and Bedrock Converse send every marked tool call as is, so the hook keeps the census main uses there. The seeding step carries custom_llm_provider on the injection points when a server tool call is marked, so a bare model name still resolves its route.
* test(anthropic): cover two answered marked server tool calls before a system message on Databricks, OpenRouter and Bedrock Converse
* fix(anthropic): keep main's cache mark count on GitHub Copilot and ChatGPT routes without resolving them, so a sync call with no stored login never starts the device flow
* test(anthropic): cover a marked server tool call reaching the GitHub Copilot and ChatGPT login hosts like an unmarked one
* fix(prompt_templates): do not mutate messages in-place in function_call_prompt
* refactor(test): avoid local list mutation and satisfy type discipline gate
* style(prompt_templates): keep line length under 120 chars for ruff format
* style(prompt_templates): type _with_function_prompt with AllMessageValues for ANN401
* style(prompt_templates): eliminate basedpyright errors in function_call_prompt
* fix(prompt_templates): keep tuple and None system content when adding the function prompt
Missing system content becomes the function prompt alone, a tuple keeps its parts, and the old test that pinned the caller-side mutation now asserts the caller's messages stay unchanged
* fix(prompt_templates): reject unreadable system content without echoing it
Only list and tuple content is read as parts. Any other non-string content raises a TypeError that names the type, so error logs never carry the system prompt
* fix(prompt_templates): leave non-list system content failing as before
None and tuple content fail the same way they did before this change, so the only behavior change is that the caller's messages are no longer mutated
* fix(prompt_templates): answer 400 for unreadable system content, send the function prompt alone under drop_params
* test(prompt_templates): pin the function prompt on every system message
* test(integration): cover function prompt fallbacks, retries, cache and drop_params on ollama
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(websearch_interception): intercept the bare web_search tool type on chat completions
* fix(websearch_interception): match the bare web_search type only on chat completions
/v1/messages keeps its existing detector, so a non-Anthropic model there no
longer short-circuits a bare web_search tool into raw search results
* test(websearch_interception): cover the bare web_search type on chat and through the messages adapter
* refactor(typing): narrow Any sources in proxy, guardrails, bedrock and responses code
Turns untyped json.loads, response.json(), header lookups and forwarded
kwargs into object or precise types at their source, so downstream code
type-checks without reportAny / reportExplicitAny. No runtime behavior
changes.
* refactor(typing): narrow Any sources in caching, MCP, providers and proxy request code
Decoded JSON, cache reads and header lookups now enter as object or
precise types, prisma tables used by the shadow-eval logger get typed
Protocol views, and a few SDK parameters move from dict[str, Any] to
Mapping[str, object]. No runtime behavior changes.
* refactor(typing): narrow Any sources in proxy server, router, logging and shared utils
* refactor(typing): narrow Any sources in provider transformations, integrations and proxy endpoints
* refactor(typing): narrow Any sources across providers, proxy and router
Deepens the Any cleanup on the hottest files: typed helpers in
core_helpers for attribute and item access, identity TypeAdapter(object)
wraps at untyped boundaries, explicit return annotations on shared
provider and proxy helpers, and Protocols in place of Any parameters.
No intended behavior change.
* test(integration): wait on the right signal in five flaky integration contracts
The Prometheus shared-label tests waited for the latency metric and then read the
request counter, which the logger writes after an await. They now wait for the counter
The auto-router session poll returned on the first 200 while the caller expected two
turns, so it now waits for the expected turn count
The S3 queue cap test let its failing sink recover on a 15s clock that could expire
mid-flush. The sink now stays down until a flush fails after every event is queued
K3 normally takes 168-175s against a 180s limit, so the limit is now 300s
The MCP env var browser spec reused an idle keep-alive socket that the proxy closes
after 5s, so its API calls now send Connection: close
* test(integration): add until_asserted so a wait polls the exact asserts it guards
eventually() takes a predicate separate from the asserts that follow it, which lets a
test wait on one value and then assert on another written later. until_asserted reruns
a block of asserts until they all pass and re-raises the last failure on timeout
The Prometheus shared-label tests and the auto-router session check now use it
* fix(vector_stores): stop sharing one list across default-constructed registries
VectorStoreIndexRegistry and VectorStoreRegistry took a mutable list as their
default argument. Python evaluates that once at class definition, so every
registry built without arguments shared one list. The proxy builds one that way
during startup, so an upsert into it leaked into every registry created after,
and into the class default itself.
Default to None and build a fresh list per instance. A caller-supplied list is
still used as given, including an empty one, so nothing that passes its own list
changes behaviour.
Fixes#38874
* test(vector_stores): annotate locals Final and mutate registries through their public API
The leak tests now go through add_vector_store_to_registry and upsert_vector_store_index rather than appending to the registry's list directly, which is closer to how the proxy hits the bug.
* fix(vector_stores): drop the last mutable list default and identity-only tests
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(e2e): out-of-process mock OpenAI, Anthropic, Bedrock and guardrail servers for the perf cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): coding-agent request corpus and real provider captures for the perf mocks
Request bodies come from a deterministic coding-agent session (system prompt, Read/Edit/Write/Bash/Grep/Glob tools, tool_use and tool_result turns carrying real source files picked by the request marker) in OpenAI chat, Responses and Anthropic formats. captures/ holds one real response per provider, route, shape and stream mode (gpt-6-luna, claude-sonnet-5, us.anthropic.claude-sonnet-5), cold prompt cache, body only, raw stream chunks with time offsets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): replay module for the perf mocks, captured provider responses picked by route, mode and request size
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mocks replay the captured provider responses, knobs shrink to flake and speed
The OpenAI, Anthropic and Bedrock mocks answer with the captured body or stream chunks for their route, mode and nearest request size, ids swapped per request, on the captured clock scaled by the p knob. The synthetic reply builders, the per-token stream pacing and the hand-rolled Bedrock event-stream encoder are gone
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mock request log names its provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf guardrail mock remembers the request marker by call id for the response hook
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mock captures retaken cold, one marker and system-prompt prefix per cell
Run 2 shared one marker across cells, so every stream capture was served from the provider's prompt cache (cache_read_input_tokens up to 523946) and the json cells had small prefix hits. Every cell now carries its own marker, also prepended to the system prompt, and all 30 captures report zero cached tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf corpus reads tests/e2e files so the lane's runner image has them
* test(e2e): perf long_context json captures for Claude retaken with a 16k output cap, no longer cut mid-thinking
* test(e2e): perf mocks keep only what the replay path and the test read
* test(e2e): perf mocks interpolate the provider's time to first byte from the captures around the request size
* test(e2e): perf captures at 100k (short) and 500k (long) tokens with tools, the fixed-size set removed
* test(e2e): PerformanceTestCase schema, deterministic request plan and wire records for the perf cases
RequestShape names a corpus session (input_tokens, tools, output_tokens) and wire.request_body builds it with setup/corpus/session.py, so a 300k request is 85 file reads of real source in the endpoint's own format. The plan no longer predicts reply text, the mocks replay captured provider responses and the runner joins what the client saw against the mock's log by marker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf Metric gains time_to_first_token_seconds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf User loses the unused role field, every JWT user is an internal_user
* test(e2e): perf RequestShape sizes requests from an input-token range, short and long stock shapes
* test(e2e): perf case runner (provision, Locust plan replay, observe) and the 50 rps mixed-provider case
provision.py checks the deployed config against the case and builds the callers, run.py replays the plan from a separate Locust process (locustfile.py) recording status, header, first-byte, first-token and total time per request, observe.py joins the records with what the mocks received and the workers' RSS to score every Limit, and compares each served body with the capture the mock replayed for it as a diagnostic. The case sends 15,000 requests at 50 rps over three model groups, two guardrails and the three request shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf case asserts time_to_first_token_seconds at percentile 50 and 99
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf provision creates every JWT user as internal_user
* test(e2e): perf observe reads only the mock log fields that remain
* test(e2e): 50 rps case sends short and long coding-agent sessions sized per request
* test(e2e): perf run keeps the per-request records as perf-records.jsonl under E2E_ARTIFACT_DIR
* test(e2e): the mixed small-gateway case runs at 25 rps (50 saturated one 4-worker pod on main and on v1.105.1)
* test(e2e): small_bodies_high_rate perf case, the github issue 46276 shape (11 to 17 KB chat bodies, keys only, 200 rps)
* test(e2e): the mixed small-gateway case runs at 10 rps for 6,000 requests (25 rps still saturated one pod on v1.105.1)
* ci: python lint job gets 20 minutes, the e2e harness phase (625 tests plus the replay harness suite) pushes a run past 15 when tests/e2e changes
* test(e2e): small-bodies perf case runs at 5 rps so gateway_added_seconds is per-request overhead, not queue wait
* test(e2e): mixed-provider perf case runs at 5 rps x 3,000 requests from 100 users
* test(e2e): mixed-provider perf case sends happy-path traffic only (no flaky deployment, no guardrail interventions or blocks)
* test(e2e): perf gateway model_list is three happy-path deployments, one per provider
* test(e2e): Postgres and Redis work per request as perf case limits
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): perf mocks wait a case-level latency with jitter instead of replaying captured timings, cpu_seconds_per_request limit from metrics-server
* test(e2e): perf suite PR carries the small_bodies_5rps case only, the mixed-provider case moves to its own PR
* test(e2e): small_bodies_5rps limits from the ten v1.105.1 baselines (1.5x on datastore counts, 2x on latency and cpu, absolute on errors and memory)
* test(e2e): perf mocks keep only the short captures the small case reads, captured timings and the Bedrock invoke route removed
* test(e2e): perf mocks generate each reply from the request (provider schema, usage, ids, event order, chunk sizes) instead of replaying captures
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(agents): support Databricks Agent Platform agents in the agent gateway
Add a databricks_agent provider that speaks the MLflow ResponsesAgent wire
(Responses-shaped input, output_text items, output_text.delta SSE) to Model
Serving endpoints (/invocations or the unified /serving-endpoints/responses)
and Databricks Apps (/responses), with a PAT or OAuth M2M client credentials
Register it through the agents config block or the Admin UI agent catalog and
call it through /a2a/<agent> (message/send, message/stream) or model:
a2a/<agent> chat completions. The chat route now runs every completion-bridge
agent through its own provider, and attaches the minted Databricks OAuth token
for both bridge and URL agents. Consumed OAuth settings never reach the
provider call
* fix(agents): harden databricks_agent input, output, and OAuth handling
Non-text content parts answer 400 unless drop_params is set, a reply without
an output list answers 502, flat OAuth fields resolve an os.environ api_base,
the bridge debug log no longer echoes the request model, routing tests stub
only the HTTP boundary, and the catalog description fits the dropdown
* fix(agents): merge agent and client extra_headers on the a2a chat route, and reject an empty databricks_oauth block
* fix(agents): bridge only databricks_agent agents on the a2a chat route, keep the card url over a client api_base, and require workspace_url for a Databricks App
* refactor(agents): read the a2a chat bridge providers from a constants set instead of branching on a provider name
* fix(databricks_agent): let the configured credential win, reject malformed client extra_headers, and keep a missing endpoint name empty
* test(agents): drive the flat OAuth a2a send through the real bridge
* fix(agents): keep an empty databricks_oauth block ignored on url agents, as before this change
* fix(agents): accept bare App hosts, env service principals, and os.environ refs on Databricks agents
* fix(databricks_agent): address review caveats on chat bridging, params, and auth
Map max_tokens and max_completion_tokens to max_output_tokens and pass
temperature, top_p, and user, so /v1/messages works without drop_params.
Send A2A message metadata only with string values, answering 400 or
dropping the rest under drop_params. Join message items with a blank line
in both stream and non-stream replies. Cache the env service principal
token until shortly before it expires. Keep databricks_agent replies out
of the response cache.
On the chat bridge, let request fields win over the agent's stored params
while pinning the agent's api_base and api_key, and apply the agent's
static_headers between client and agent extra_headers, as /a2a does.
Resolve os.environ/ only for databricks_agent api_base and api_key instead
of every bridge provider. Ignore an unusable databricks_oauth block on URL
agents on chat, as at the merge base, instead of answering 400.
* test(internal_context): list the databricks agent token cache as process-local
* fix(databricks_agent): keep the agent's endpoint, input, and target out of client control
extra_body model and input answer 400, or are dropped under drop_params.
A client api_base or base_url never replaces the agent's card URL on the
bridge. Chat to a URL agent keeps its merge-base call, and an incomplete
databricks_oauth block on a URL agent keeps its merge-base ValueError.
* test(databricks): expect the merge-base 400 for a client api_base on agent chat
* fix(databricks): keep agent-owned inputs and identity headers out of a chat caller's reach
* fix(databricks): bound the agent-over-caller merge to two levels so it needs no recursion
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
