Under otel_v2 a single MCP tool call surfaced in APM as two disconnected
traces joined only by a span link: the HTTP transport transaction
POST /{mcp_server_name}/mcp and the tools/call span carrying
error.type=MCPToolResultError. resolve_mcp_span_context parented the MCP
span to the W3C trace context the client propagates in params._meta
(SEP-414) and recorded the transport as a link, so with no traceparent
propagated (the common case today, including MCP Inspector) the span
started its own root trace.
Nest the MCP span under the transport span when nothing is propagated, so
the call stays in one trace; the propagated-context path is unchanged and
still parents to the remote context and links the transport per the OTel
GenAI MCP semconv.
The transport has to be resolved per message rather than read from the
request-root ContextVar. A stateful streamable-HTTP session runs every
message on the single task the session's initialize POST spawned, so that
ContextVar is frozen at initialize inside the handler: live capture on
staging showed the tools/call span linking the initialize POST rather than
the POST that carried it, and nesting on that anchor would hang every tool
call of a session off the first request's already-ended span. The gateway
now resolves the current request's span on the ASGI task and carries it to
the handler on the authenticated-user object, the same way per-request auth
already crosses that boundary.
* fix: handle explicit outputInfo: null in Vertex AI batch response
Vertex AI can return HTTP 200 for a create_batch/get_batch call with an
explicit "outputInfo": null body (the output directory is assigned
asynchronously and may not be populated yet at response time).
_get_output_file_id_from_vertex_ai_batch_response did:
response.get("outputInfo", OutputInfo()).get("gcsOutputDirectory", "")
dict.get(key, default) only substitutes default when the key is absent,
not when it is present but explicitly None, so this crashed with:
AttributeError: 'NoneType' object has no attribute 'get'
surfaced to callers as an opaque openai.InternalServerError 500 from
litellm.create_batch()/retrieve_batch() for any Vertex AI batch job,
regardless of whether the job ultimately succeeds.
Fixed by guarding with `response.get("outputInfo") or OutputInfo()`,
matching the existing null-safe pattern already used by the sibling
_get_input_file_id_from_vertex_ai_batch_response for inputConfig. The
existing outputConfig fallback branch (a few lines below) already
handles this case correctly once it's reachable - it just never was.
Added 2 regression tests covering outputInfo: null with and without an
outputConfig fallback available.
* test: drop explanatory comment from regression test
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails
Pre-call guardrails run sequentially because each may mutate the request
payload and later guardrails depend on earlier mutations. Deployments with
several slow block-only pre_call guardrails (external moderation, Bedrock,
LLM-judge) therefore pay the sum of their latencies. during_call guardrails
run concurrently but alongside the LLM call, so a violating payload has
already been sent, which is unacceptable when the request must never reach
the model.
This adds a per-guardrail run_in_parallel flag (default off). Guardrails that
opt in are pulled out of the sequential loop and run concurrently via
asyncio.gather after every sequential (payload-mutating) guardrail has run, so
they observe the mutated payload and still form a hard barrier before the LLM
call; the first to raise blocks the request. Their returned data is discarded
since they are declared block-only.
The flag is wired from LitellmParams onto the guardrail instance at the same
generic choke point in initialize_guardrail that already sets
skip_system_message_in_guardrail, so no per-provider initializer needs to
change.
* feat(guardrails): extend run_in_parallel opt-in to post_call guardrails
post_call_success_hook ran guardrails sequentially for the same reason
pre_call did: response-modifying guardrails thread the response forward. But
block-only output scanners (which read the response and reject on violation
without changing it) serialize for no benefit and add latency.
This reuses the existing run_in_parallel flag for the post_call hook. Opted-in
post_call guardrails are pulled out of the sequential loop and run concurrently
via asyncio.gather after the sequential (response-modifying) guardrails and
before the non-guardrail CustomLogger callbacks, so they inspect the final
response and still block it from reaching the client if any raises. Their
returned response is discarded since they are block-only.
The apply_guardrail path sets data["guardrail_to_apply"] immediately before
awaiting, and unified_guardrail pops it before its first suspension point, so
concurrent guardrails never race on that key under asyncio's cooperative
scheduling.
* fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes
Addresses review feedback on the run_in_parallel opt-in.
asyncio.gather propagated the first exception without cancelling or awaiting
the siblings, so a block at t=0 left the other guardrails running as
unobserved background tasks (wasted external calls plus event-loop warnings),
and a fast SensitiveDataRouteException/ModifyResponseException could return a
reroute or passthrough before a slower block finished, letting crafted input
bypass the block. Both the pre_call and post_call parallel batches now gather
with return_exceptions=True so every guardrail runs to completion, then raise
any blocking exception ahead of a flow-changing one.
The registry choke point wrote bool(None)==False onto every instance when the
config omitted run_in_parallel, silently disabling a constructor-set default;
it now only writes when the config provides an explicit value.
* fix(guardrails): record lifecycle logs for every concurrently-run guardrail
The log_guardrail_information decorator skipped its auto-record when it saw
that the count of standard_logging_guardrail_information entries in the shared
request_data had grown during the wrapped call, taking that as proof the
wrapped function had recorded its own richer entry. That heuristic breaks the
moment guardrails run concurrently (parallel pre_call/post_call, during_call):
a sibling guardrail's append inflates the shared count, so a guardrail that did
not self-record wrongly concludes it already did and drops its own entry. The
result is that enabling run_in_parallel silently loses per-guardrail lifecycle
logs, so the Admin UI Request Lifecycle timeline and downstream loggers
(Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent
guardrails.
Replace the shared-count heuristic with a ContextVar flag set when a guardrail
records its own entry. asyncio copies the context into each gathered task, so
the flag is isolated per concurrent guardrail while still catching the
self-record-then-skip-auto-record case within a single invocation.
* test(guardrails): declare run_in_parallel on post_call guardrail mocks
The post_call partition reads run_in_parallel on every CustomGuardrail
callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is
set in __init__, not on the class) so the attribute access raised, and even
a class-level default would return a truthy child mock that wrongly routes
the double into the parallel batch. Declare the flag False on the shared
mock factories so these pre-existing hook tests exercise the sequential
path they assert on.
* fix(guardrails): harden run_in_parallel reads and address review feedback
Read run_in_parallel via getattr(..., False) in the pre_call and post_call
partitions so a third-party CustomGuardrail subclass that overrides __init__
without chaining super().__init__() no longer raises AttributeError on a path
that previously worked. Drop the redundant in-function GuardrailEventHooks
import in _run_parallel_post_call_guardrails (already imported module-level).
Remove the flaky wall-clock upper-bound assertions from the two concurrency
tests; the all-start-before-any-end overlap assertion is the timing-independent
signal that actually proves concurrency.
litellm already supports Google, Microsoft and generic OIDC SSO through
fastapi-sso, which has no SAML support; AuthMethod.SAML existed only as an
unused enum value. This adds real SAML 2.0 single sign-on for the admin UI.
A new SAMLAuthHandler validates signed assertions with the OneLogin
python3-saml toolkit and maps them onto a CustomOpenID, then reuses the
shared post-login path every other provider goes through, so provisioning,
role/team mapping and the UI session JWT are unchanged. Both SP-initiated
and IdP-initiated HTTP-POST flows are supported. SP-initiated logins are
bound to the browser that started them via an HttpOnly state cookie plus a
cached AuthnRequest id, and the ACS rejects any response whose InResponseTo
doesn't match; unsolicited (IdP-initiated) responses cannot be browser-bound
so they are rejected unless SAML_ALLOW_UNSOLICITED=true. Replays are rejected
by a consumed-assertion guard whose lifetime tracks each assertion's
NotOnOrAfter, and both the replay guard and the login-state binding go
through the proxy's shared in-memory + Redis cache for multi-instance
deployments. The ACS honors DISABLE_ADMIN_UI and re-applies the
free-SSO-user Enterprise gate after the assertion is validated, so an
unvalidated POST can no longer drive the billable-user count query.
SAML is configurable from the admin UI SSO settings (IdP metadata URL or
inline XML, SP entity ID, and an allow-unsolicited toggle), which persists
the SAML_* environment variables the handler reads, exactly like the Google,
Microsoft and generic OIDC providers.
python3-saml is kept as an optional saml extra; its xmlsec and lxml wheels
bundle the native libraries so no system packages are required, and the
import is guarded so the proxy still starts without the package with the
SAML routes returning a clear 501.
Resolves LIT-4016
The gateway image installed --extra proxy/proxy-runtime/extra_proxy/semantic-router
but not bedrock-realtime, so aws-sdk-bedrock-runtime was absent. Bedrock Nova Sonic
speech-to-speech uses InvokeModelWithBidirectionalStream (which boto3 cannot do) via
that package, so realtime requests failed at startup with 'Missing
aws_sdk_bedrock_runtime'. Add the extra to both uv sync stages; it is already in
uv.lock so --frozen resolves, and the package/marker (python>=3.12) matches the
python3.13 image.
The accumulated-JSON fallback ran json.loads over the whole buffer after every fragment and, on failure, kept the buffer without resetting it. A buffer that ever held more than one concatenated JSON value could never parse (json raises on trailing data), so it returned None on every subsequent chunk while growing without bound - an unrecoverable per-request CPU spin. Parse one value at a time from the front with raw_decode and keep the remainder, draining trailing values on later calls and at end of stream.
The async streaming error paths fired the sync failure_handler in a thread
and the async_failure_handler via create_task at the same time, so both
mutated the shared logging object concurrently and could crash pydantic-core.
Route failure logging through a single guarded dispatch_failure_handlers, so
the sync handler only runs after the async one completes.
* fix(guardrails/model_armor): handle None metadata in post_call _process_response
On batch routes data["metadata"] is normalized to None (present key, None
value), so request_data.get("metadata", {}) returned None and _process_response
raised 'NoneType' object has no attribute 'get', 500ing every /v1/batches create
with a post_call Model Armor guardrail (regression from v1.93.0 activating the
post_call hook). Coalesce a falsy metadata to {}
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Clean up test case documentation
Remove regression comment from test_process_response_with_none_metadata_does_not_crash.
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The a2a completion-bridge tests registered the agent with only
custom_llm_provider and model, so the bridge's litellm.acompletion had no
api_key and relied on the gateway resolving ANTHROPIC_API_KEY from its ambient
env. When that env var is absent, POST /a2a/{id} returns 500 with
"Missing Anthropic API Key" and every message/send test (completion bridge,
pinned v0.3/v1.0 message shapes, semver serves) fails while the
register/discovery/rejection tests still pass.
Give A2ABridgeParams an optional api_key and register the bridge agent with
api_key="os.environ/ANTHROPIC_API_KEY", matching how the rest of the suite
wires anthropic-backed models (e.g. the ratelimit redis tests). The agent now
carries its provider key explicitly instead of depending on ambient gateway env.
Registers claude-opus-5 across the cost maps and provider lists so the model
prices, reports its real 1M/128K limits, and advertises its capabilities instead
of falling through the generalization patterns at zero cost.
Adds the first-party entry plus the Bedrock (base, global, us, eu, au, jp),
Vertex AI, and Azure AI variants. Pricing matches Opus 4.8 at $5/$25 per MTok
with the usual 1.1x regional premium on the cross-region inference profiles, and
fast mode is priced at 2x through provider_specific_entry on the first-party
entry only.
Two fields deliberately differ from Opus 4.8: prompt_cache_min_tokens drops to
512, and bedrock_output_config_effort_ceiling is omitted because Bedrock accepts
output_config.effort="max" for Opus 5.
The suggested CIDR chip was a click-only span both before and after the shadcn
migration, so keyboard users could not reach or activate it. Render it as a
Button, which brings focus and Enter/Space activation with it, and cover the
keyboard path with a test that fails against the old span.
A template with no LLM enrichment rendered every parameter field twice: the
shared list already covers them, because nonEnrichmentParams is the full
parameter list when there is no enrichment, and a second no-enrichment branch
mapped the same list again.
Predates the shadcn migration and was carried forward by it. The test now
asserts exactly one field per parameter, and fails if the duplicate branch
comes back.
Replaces antd and Tremor with shadcn primitives across the 18 files these three
routes exclusively own. Markup only: no behaviour, data flow or copy changed, and
no shared or form-bearing component is touched, so the blast radius stops at
these pages.
The 12 tests covering these components are unchanged from the previous commit and
still pass, which is the evidence that the rewrite preserved behaviour. Also
prunes the six antd no-restricted-imports suppressions these files no longer
need.
Markup-only migration of the 17 files these three routes exclusively own,
replacing antd and Tremor with the installed shadcn (base-vega) primitives and
lucide icons. No route behaviour changes; the tests written in the previous
commit are untouched here and pass against both the old and the new markup.
Colour now comes from tokens rather than from hardcoded utilities, so the
health-check button, the alerts and the badges no longer pin their own palette.
email_settings also loses an invalid DOM nesting (a table cell inside a div, and
a div inside a paragraph) that React had been warning about.
Two modals on the policies page moved from the Policies panel up to the panel
root. Base UI Tabs mounts only the active panel, unlike Tremor, and both are
opened from the Templates tab, so leaving them nested would have made "Use
Template" do nothing.
Retires 53 antd import suppressions from the eslint baseline.
Greptile caught a real regression in the shadcn migration: starting to edit organization
settings and switching to another tab silently discarded the unsaved input.
antd Tabs and Tremor TabGroup mount a panel lazily and then keep it mounted, so a
half-filled form or a search history survives leaving the tab and coming back. Base UI
unmounts inactive panels instead. Its keepMounted escape hatch is not equivalent either:
it mounts every panel eagerly, which renders work the user may never ask for and, on the
organization view, put the organization name on screen twice.
useVisitedTabs reproduces the original semantics by tracking which tabs have been opened
and keeping only those mounted. It is applied to the two tab strips whose panels wrap
stateful children: organization Settings, and the vector-stores Create and Test tabs,
where an in-progress upload or a search history was equally exposed. The access-group
detail tabs render lists derived from props, so they stay lazy.
The added regression test fails without the fix and passes with it, and it also passes
against the pre-migration antd component, so it pins parity rather than the new markup.
Establishes the regression net for the upcoming markup migration of these
three routes. Every assertion here is written against the current antd and
Tremor components and passes against them, so it carries no knowledge of the
markup that replaces them and stays meaningful afterwards.
Adds characterisation tests for the seven components that had none, and
rewrites cache_dashboard's chart-card lookup to anchor on each chart's own
title instead of asserting a global count of card nodes, which would break the
moment another card appears on the page.
No component is touched in this commit.
Moves the nine files these three routes exclusively own off antd and Tremor onto the
shadcn primitives in src/components/ui. Scope came from the migration analyzer's import
closure, so nothing reached by a second route is touched and every file carrying an antd
Form is left alone until #34195 lands.
access-groups gets the page header, search box and the whole detail view; vector-stores
gets the tab shell, the store picker and the tester panel; organizations gets the
organization detail view and the three filter controls.
Two changes are behavioural rather than cosmetic. The vector-stores tab strip moves from
Tremor, which mounts every panel at once, to Base UI, which mounts only the active panel;
that is the correct behaviour and the reworked test now opens the tab it asserts on. The
antd Select on the Test Vector Store tab becomes a combobox rather than a plain select so
its showSearch type-ahead survives.
organization_view keeps one antd import, the ColumnsType used to build the extra columns
it hands to the shared MemberTable; that is dictated by the shared component's API and
goes away when MemberTable migrates. eslint-suppressions.json ratchets down accordingly:
eight files lose their no-restricted-imports entry and organization_view drops from three
to one.
Every test passes unedited across the migration, and the visual gate reports the three
migrated routes changed with the other 32 pixel-identical
Rewrite the two markup-coupled assertions off antd class selectors and onto
role/text queries, and add characterisation tests for the nine route-owned
components that had none. Both rewritten tests and all nine new ones are green
against the current antd and Tremor components, so the migration that follows
can be judged by tests it never touched.
Replaces antd and Tremor with the installed shadcn primitives on the three
route-exclusive panels: Tremor tabs, buttons and text on budgets; the antd
delete Modal and Tremor button on skills; the Tremor card, inputs and buttons
on ui-theme.
Markup only, no behaviour change. The characterisation tests added in the
previous commit are untouched and stay green, and the ui-theme inputs now
carry real label associations.
Shared components stay on antd; they are reached by other routes and are
migrated separately. The form-bearing files on these routes are left alone.
Prepares the shadcn migration of these three routes by removing every assertion that
depends on the current component library, so the same tests can gate the migration
without being edited.
FiltersButton and its OrganizationFilters consumer both asserted on the ".ant-badge"
wrapper class; they now assert the active-filter indicator element itself, and
FiltersButton additionally asserts that it is absent when there are no active filters.
TestVectorStoreTab drove the antd Select with fireEvent.mouseDown and picked options by
node; it now clicks through the combobox role and the option text, which works against
any listbox implementation.
The vector-stores index test relied on Tremor mounting every TabPanel at once, so it
read the Manage tab's table without ever opening that tab. It now clicks the tab
first, which is what a user does and what any tabs implementation supports.
VectorStoreTester had no test at all, so this adds a characterisation suite covering
the empty state, the blank-query guard, the search call and its rendered result,
result expansion, Enter versus Shift+Enter, the failure path and clearing history.
All of these pass against the current antd and Tremor components
Adds a role/text-based characterisation test for UIThemeSettings, which had
none, and extends the skills panel test to cover the delete confirmation.
Both are green against the current antd/Tremor components so they can prove
the shadcn migration keeps behaviour identical without being edited.
The dashboard already receives the requested model name as model_group on
every spend-log row, but LogEntry dropped the field, so nothing distinguished
an auto-routed request from a direct one.
Surface it precisely rather than by comparing requested against resolved:
model_group differs from model for plain aliases and wildcard deployments
too, so a bare mismatch tags almost every row and identifies nothing. The
indication is driven instead by which deployments are auto-routers, resolved
from every page of /v2/model/info and shared through context.
The request drawer header names the router in a badge next to the provider;
the session sidebar swaps the entry's leading icon. Rows that no auto-router
served render exactly as before.
* fix(proxy): restore atomic user upsert when adding team members
Parallel /team/new calls naming the same not-yet-existing member were
returning 500 "Unique constraint failed on the fields: (`user_id`)".
The upsert in add_new_member passed an empty update branch. Prisma only
compiles an upsert down to a single INSERT ... ON CONFLICT when that branch
writes something; with an empty one it emits SELECT-then-INSERT instead, so
concurrent requests all read "no such user" and all insert. Postgres
statement logs confirm it: the empty form logs BEGIN/SELECT/INSERT/COMMIT,
the non-empty form logs INSERT ... ON CONFLICT ("user_id") DO UPDATE SET.
Re-state user_id in the update branch as a no-op so the native upsert path
comes back. The teams append stays in the filtered update below it, so an
already-existing member still cannot pick up a duplicate team id.
tests/test_team.py::test_team_new failed 9 of 15 runs against a live proxy
before this and 0 of 15 after. The existing unit test asserted only that
upsert had been called on a mock, so it passed either way; it now pins the
shape of both branches and fails when the update branch goes back to empty.
* test: point the live codex tests at gpt-5.3-codex
OpenAI deprecated gpt-5.2-codex, so test_openai_codex and
test_openai_codex_stream started failing against the live API with
model_not_found. gpt-5.3-codex is the current codex model; both tests pass
on it. The remaining gpt-5.2-codex references in the suite are mocked
transformation tests and are unaffected.
* test(e2e): update models page specs for the shared DataTable
The DataTable migration in #34363 changed three things the models page
specs were pinned to, and five tests went red.
Row click no longer opens the detail view; the Model ID cell owns that
now, so both specs click its `model-id-<id>` test id instead of the row.
The search box placeholder switched from an ASCII "..." to a real
ellipsis, so the specs use getByPlaceholder with a substring instead of
an exact attribute match that punctuation can break again. The results
count moved from `models-results-count` ("Showing 1 - 50 of 137 results")
to the shared pagination's `pagination-range` ("Showing 1-50 of 137").
The Team-BYOK test also filtered rows on the team alias, which the Team
ID column has never rendered in either the old or the new table; it
filters on the team id now, which is what the column actually shows and
what the assertion's own comment intends.
Verified against a local proxy serving a fresh build with the seeded
e2e postgres and mock upstream: all five failing tests pass, and the
full suite is 82 passed / 4 skipped at CI parity (workers=1).
combine_usage_objects iterates prompt_tokens_details model_fields and sums each;
with cache_write_tokens and cache_creation_tokens now mirroring each other via
__setattr__, the pair was summed twice, doubling cache creation counts for
Anthropic batch cost calc, mid-stream fallback usage merges, and realtime usage.
Collapse the mirrored pair to one representative before summing.
On the /v1/responses path the response usage is not chat-Usage-shaped, so
additional_usage_values could not derive cache tokens from response_obj.usage
and the Admin UI Logs cache-creation token row stayed empty. Fall back to the
normalized standard_logging usage_object's prompt_tokens_details for both the
cache-read and cache-creation counts.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The Responses API (/v1/responses) usage transform rebuilt prompt token
details and dropped OpenAI's input_tokens_details.cache_write_tokens, so
gpt-5.6 cache-creation tokens were never logged or billed via that route.
Map it in the transform, and make PromptTokensDetailsWrapper keep
cache_write_tokens and cache_creation_tokens in sync on assignment.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject request when budget reservation write fails under fail_closed_budget_enforcement
With general_settings.fail_closed_budget_enforcement set to true, the read-time
spend check already returns 503 when spend cannot be verified, but the atomic
pre-call reservation still failed open: reserve_budget_for_request swallowed
_CounterReservationUnavailable per counter and degraded to read-time-only
enforcement, so concurrent requests could all pass the same under-budget read
during a Redis outage and overspend past the configured budget.
Now the strict flag is threaded into reserve_budget_for_request and a failed
reservation write raises 503, releasing any counters that already reserved.
Default behavior with the flag absent or false is unchanged.
Fixes#33923
* fix(proxy): pass 503 budget-enforcement detail as plain string
