get_sanitized_user_information_from_key copied UserAPIKeyAuth.metadata
verbatim into user_api_key_auth_metadata, so the key's callback
configuration - including the integration credentials inside callback_vars -
reached the StandardLoggingPayload every integration receives. The two other
sites that stamp key/team metadata into request metadata did the same.
Sanitize at those sources with strip_callback_config, which drops the
`logging` and `callback_settings` slots and leaves everything else (notably
`priority`, read back by the dynamic rate limiter) untouched. Those slots are
resolved from UserAPIKeyAuth during pre-call setup and never read off the
logged copies, so nothing downstream loses input.
This makes the scrub in scrub_sensitive_keys_in_metadata dead - it only
matched the string "logging" under one of the two field names and never
covered callback_settings - so it is removed.
Separately, LangSmith set the run's `inputs` to the raw StandardLoggingPayload
while redacting only `extra`, so redact_user_api_key_info left every
user_api_key_* field in inputs.metadata. Both now go through one
_redact_metadata helper, which also covers the nested requester_metadata copy.
Validating the cursor whenever `after` was non-None turned `?after=` into a
400, which the listing has always read as "no cursor". Only a cursor the
client actually sent is looked up now, matching the sibling managed-resource
listing.
An `after` that does not resolve to a batch the caller can list now returns
400 instead of an empty page. An empty page is indistinguishable from the end
of the list, so a stale or malformed cursor silently truncated a client's batch
list. The lookup is scoped to the caller's own rows, so a Prisma cursor can no
longer be anchored to another user's batch.
`has_more` now comes from whether an extra row exists rather than from whether
the page came back full. Reporting fullness made every client fetch one extra
empty page when the batch count was an exact multiple of `limit`, and made a
page shortened by an unparseable row look like the end of the list, hiding the
older batches behind it.
Also drops the unreachable `target_model_names` oversampling branch; that
argument raises a few lines above it.
translate_thinking_for_model duplicated the same summary/auto_summary
wrapping logic as _translate_thinking_to_openai without the
disabled-thinking guard, so it could still wrap "none" into an
{effort, summary} dict when reasoning_auto_summary is enabled (caught
by Cursor Bugbot). Extract the wrapping rule into one shared
_apply_reasoning_summary_wrapping helper used by both call sites so
this invariant can't drift apart again.
Codecov flagged the empty-list early-return in
_translate_stop_sequences_to_openai as an uncovered line in the diff —
add a regression test asserting stop_sequences=[] does not set
new_kwargs["stop"].
Two follow-ups from review on the upstream-reported usage contract.
An unusable cost header fell through to the endpoint's flat cost_per_request
instead of the zero the contract promises, so a target that contradicted itself
got billed an estimate it had just disowned. A target that speaks this contract
now owns the cost for the request whether or not the value it sent parsed.
The reported total also cannot be split into prompt and completion, so reading
one out of it under token_rate_limit_type input or output yielded zero and left
the TPM window uncharged; pass-through traffic then ran past a limit it is
meant to share with the general API. Usage that carries no split now charges
its total under every limit type, while usage that does carry one is untouched.
Guard against reasoning_auto_summary wrapping "none" into a dict when
thinking is disabled — there's no reasoning trace to summarize, and
non-Claude providers (e.g. Fireworks) expect reasoning_effort as a
plain string.
Claude Code's auto-mode classifier sends stop_sequences and thinking:
{type: disabled} on /v1/messages. The Anthropic adapter passed
stop_sequences through unchanged instead of mapping it to OpenAI's stop,
which Fireworks' OpenAI-compatible endpoint rejects with HTTP 400. It also
dropped disabled thinking instead of mapping it to reasoning_effort: none,
so the model spent its output budget on reasoning it was told to skip.
Resolves LIT-4798
Restore the original PR's behavior on every path that did not already
resolve: no database, a lookup error, a missing managed-file row, or a
row without a storage_url all fall back to dispatching the original id,
which the managed-files deployment hook still maps. This drops the 404
and 503 fail-closed responses I had added, which were the only behaviors
that diverged from litellm_internal_staging.
The change is now strictly additive: when a managed-file row with a
storage_url exists, the unified batch branch substitutes it so providers
like Vertex receive a real gs:// path instead of the opaque token; every
other path behaves exactly as before. Verified live that non-managed,
managed-owner, multi-model load-balanced, and missing-row requests are
byte-identical to base
PassThroughGenericEndpoint.cost_per_request defaults to 0.0, so every
config-defined endpoint forwards a flat 0.0 even when the operator never
configured one, and the success handler applied it over whatever cost was
already established. That silently zeroed the cost an upstream reported for
the request. The flat value is an estimate for targets LiteLLM cannot price,
so it now yields to a target that priced the request itself; it still applies
unchanged when no cost was reported.
The cap was an env-tunable knob in constants.py. Nothing needs to tune it:
it exists so DISTINCT cannot run over an unbounded row set, and picking a
value is a correctness decision, not deployment configuration. An env var
also makes the bound unverifiable, since the same code can behave very
differently between two proxies.
It is now a plain constant next to its only caller, mirroring how
SPEND_LOGS_PAGINATION_COUNT_CAP sits beside ui_view_spend_logs, and it takes
that constant's value: both reads of LiteLLM_SpendLogs now stop at the same
depth. constants.py goes back to matching staging exactly.
The existing test only asserted the parameter equalled the constant, which
is tautological; raising the constant to a billion kept it green while
removing the bound. A second test pins the value against the logs page's
cap, so an arbitrary change to either one fails.
* fix(proxy): attribute spend to org for team-linked keys minted without org_id
Keys attached to an org-linked team but minted without an organization_id
produced spend that was never credited to the org: the spend writer reads
user_api_key_dict.org_id with no team fallback, while the org budget check
resolves the org from the team. The check therefore ran against a counter
fed by almost none of the org's traffic and never tripped.
Backfill org_id from the freshly fetched team object in
_run_centralized_common_checks, per request only, so the spend writer and
the budget check read the same org. A key with an explicitly pinned org_id
always wins, and the cached key row is never mutated, so moving a team to
a different org takes effect on the next auth once the team cache
refreshes.
* test(proxy): cover CLI session-token org backfill from team
CLI session tokens from /sso/cli/poll are minted with a real team_id but
no org_id, and their auth path decrypts the blob without the combined_view
team join that fills org for DB keys. Spend from these tokens reached the
team but never the org, so org budgets never tripped. The regression test
mints a real CLI token, runs it through the centralized checks, and
asserts the credential leaves auth with the team's org.
A pass-through target that fans a single HTTP request out to several models
internally cannot be priced from its response body, so LiteLLM had nothing to
record and every such request landed in the spend logs with zero cost and zero
tokens. The target now reports the totals for the whole request in
x-litellm-response-cost and x-litellm-total-tokens response headers, and
LiteLLM records those values as-is rather than recomputing them.
The headers are read on every upstream response, so a request that burned
tokens before failing still books its spend on the failure row instead of
being dropped for having a 4xx/5xx status. Only what the upstream actually
reported is written, so a target that sends a cost but no token count keeps
the token count LiteLLM derived on its own; a target that sends neither header
is untouched, which is the normal case for Anthropic, Vertex and friends.
Two supporting fixes fall out of this. The rate limiter only pulled token
counts off response shapes it models, so pass-through usage never charged the
TPM window and a team could exceed its shared token limit through pass-through
traffic alone; it now falls back to combined_usage_object. And the streaming
success path reset response_cost unconditionally before the assembled response
recomputed it, which discarded any cost a pass-through handler had already
established (the pass-through branch right below it has always intended to
preserve exactly that).
With end_date omitted the floor was end-anchored to now including its
time-of-day, so an explicit start_date exactly 30 days back parsed as
midnight, compared below the floor, and was invisibly clamped to a
mid-day instant: up to a day of spend disappeared while the response
start_date still printed the full calendar date. Anchoring the floor to
today's UTC midnight makes every comparison in the window derivation
date-pure.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The scope predicate expanded permitted team ids into an IN list with one
placeholder each, copied from /key/aliases. ui_view_spend_logs, which owns
the same page and the same scoping rules, builds ("user" = $X OR team_id =
ANY($Y::text[])) instead: a single array parameter whatever the team count,
and no placeholder arithmetic to keep in step with the rest of the query.
Same semantics, but the two clauses now read identically, so a future change
to how spend logs are scoped is harder to apply to one and miss in the other.
The clamp floor is end_date minus 30 days, serving up to 31 calendar
dates inclusive: deliberately the same width as the endpoint's default
window, so the dashboard's own default range never triggers the clamp
note. The docstring, card note, and test name now state that invariant
instead of the misleading 'most recent 30 days'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: resolve unified_file_id to real storage_url before dispatching batch create
litellm.create_batch() against a Vertex AI-backed model crashes with an
opaque error when the input file was uploaded as a LiteLLM-managed
'unified file' (multi-model file upload). The base64-encoded
unified_file_id token is a LiteLLM-internal identifier, not a real
provider-side file reference, but the batches_endpoints create_batch
handler forwards it unchanged to llm_router.acreate_batch() /
litellm.acreate_batch() for the unified_file_id branch. Provider-specific
code that expects a real file location (e.g. Vertex AI's batch
transformation, which parses a 'publishers/' segment out of the GCS URI)
then fails on the opaque token.
Resolve the unified_file_id to its real backend location
(LiteLLM_ManagedFileTable.storage_url) before dispatch, mirroring the
same lookup already used by the files retrieve/download endpoints for
managed files. Falls back to the previous (unchanged) behavior if no
managed-file record exists.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(proxy/batches): null-guard await on find_first for sync MagicMock test harnesses
* fix(proxy/batches): enforce ownership and correct lookup key when resolving managed input_file_id
The adopted resolution queried LiteLLM_ManagedFileTable with the decoded
litellm_proxy string, but the unified_file_id column stores the raw base64
file id (see schema.prisma and the enterprise managed-files hook), so the
lookup never matched in production and silently fell back to the opaque id.
Query with the raw id instead and lock the key with a regression test.
Move the resolution above the dispatch branches so the load-balanced router
path receives the resolved storage_url too, enforce managed-file ownership
with the same can_access_resource semantics the files retrieve and download
endpoints use (404 on denial), and downgrade database failures to a logged
fallback instead of aborting batch creation. Unresolved ids still dispatch
unchanged because the managed-files deployment hook can map them via
model_file_id_mapping
* fix(proxy/batches): fail closed when the managed file ownership lookup errors
A lookup exception previously fell back to dispatching the original
unified id with the ownership gate unexecuted; the managed-files
deployment hook maps unified ids from cache without re-checking
ownership, so a database outage let a caller dispatch another tenant's
file. Raise a clear 503 instead and lock the behavior with a regression
test. No-database and no-row cases still fall back unchanged
* test(proxy/batches): default harness prisma_client to None
The batch routing harness left proxy_server.prisma_client at its module
global, which a sibling test in the same shard can leave as a MagicMock.
The unified-file rows that do not opt into managed-file resolution then
entered the resolver and awaited a non-awaitable mock, surfacing as a
503. Patch prisma_client to None by default so those rows stay a no-op;
resolution tests still override it explicitly
* fix(proxy/batches): keep unified resolution in its own branch and fail closed on missing row
Cursor flagged that hoisting the storage_url substitution above the
load-balanced dispatch branch broke two things on that path: the
model_file_id_mapping deployment filter keys on the original unified id,
and the response returned the internal storage_url instead of the
unified id. Move the resolution back inside the unified branch and
exclude unified ids from the load-balanced branch so a managed file
always takes the resolving path (which restores input_file_id and the
unified_file_id hidden param on the response), and a load-balanced batch
keeps the original id for deployment filtering.
Also fail closed with a 404 when a unified id has no managed-file row
while a database is present: the id cannot be ownership-verified, and
dispatching it would both bypass the gate and hit the Vertex
publishers-segment IndexError. Owned rows without a storage_url (legacy)
still dispatch the original id
* fix(proxy/batches): do not divert unified files off the load-balanced branch
Excluding unified ids from the load-balanced branch (and not
unified_file_id) regressed a path that works on the base revision: a
multi-model managed file dispatched with an explicit router model under
load balancing was routed into the unified branch, which raises a 400
for anything other than exactly one target model. Verified live against
base (200, managed-files deployment hook remaps the unified id per
model) versus the guarded branch (400 Expected 1 model, got 2).
Restore the original three-condition load-balanced branch so that path
keeps working unchanged. Unified-file storage_url resolution and the
ownership 404 still apply on the non-load-balanced unified branch, which
is the common managed-batch flow; the load-balanced managed path retains
its existing behavior and its pre-existing enterprise-hook ownership gap,
unchanged from base
* refactor(proxy/batches): scope managed-file handling to resolution, drop ownership check
Narrow this PR to its one problem: resolving a managed unified input_file_id
to its backend storage_url so provider batch handlers (Vertex parses a
publishers/ segment) receive a real location instead of the opaque token,
and failing closed with a 404 when the token has no backing row so it is
never dispatched into the provider crash.
Remove the cross-tenant ownership check (can_access_resource) added earlier.
Batch-create had no ownership enforcement before this PR, and the gap spans
every managed-file call type, so it belongs in the enterprise managed-files
pre-call hook (its acreate_batch branch) where files, batches and
fine-tuning are covered uniformly, not partially in this one endpoint. Filed
as a follow-up. This also removes the load-balanced-path ownership
inconsistency the bots flagged, since there is no ownership branch to skip.
Drop the inline comments flagged against the no-comments rule; behavior is
documented in the helper docstring and the test docstrings
* fix(proxy/batches): fail closed with 503 when the managed-file lookup errors
A lookup exception previously fell back to dispatching the unresolved
unified token, which defeats the fail-closed guarantee: the token still
reaches the provider and can hit the same publishers-segment IndexError
the resolution prevents. Treat a lookup error like the missing-row case
and fail closed, but with a retryable 503 since the condition is
transient. No-database and no-storage_url rows still fall back unchanged
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two issues Greptile raised on the filter window and the capped scan.
A preset date range ends at "now", which the logs query re-reads on every
fetch, so live tail keeps moving the table's end bound. The filter window
was memoized on the date controls alone, so it pinned whichever "now" it
was first built with: an end user that started sending traffic afterwards
showed up in the table but stayed missing from the dropdown until
something remounted it.
formatLogsWindow now takes the preset end bound as an argument, and
getLogsWindowEndBound derives it from the logs query's last fetch, rounded
up to the next minute. Rounding up rather than down means the filter window
never trails the table; bucketing means the query key holds steady between
ticks instead of churning once per render. The panel reads it from
logsQuery.dataUpdatedAt so it advances exactly when the table refreshes,
falling back to the stored end time before the first fetch. Deriving it
from Date.now() during render is what the purity rule forbids.
The capped inner scan ordered by startTime alone, so rows sharing a
timestamp could be cut differently between two requests and successive
OFFSET pages would disagree about the set they were paging through.
request_id now breaks the tie, which the (startTime, request_id) index
already covers.
Drift from rows genuinely arriving inside the window between page fetches
is left alone. Removing it means keyset pagination over the distinct set,
which cannot keep the inner row cap, and that cap is what stops this
query from degrading into a full scan of LiteLLM_SpendLogs.
GET /v1/tool/spend aggregated LiteLLM_SpendLogToolIndex joined to
LiteLLM_SpendLogs with a start_time-only predicate the composite
(tool_name, start_time) index cannot serve, and the dedup total query
left the outer SpendLogs scan unwindowed, so every dashboard load
walked both per-request tables end to end.
- clamp the window to the most recent 30 days ending at end_date; the
response start_date reflects the effective window and the dashboard
notes the clamp
- index SpendLogToolIndex on start_time (all schema copies + migration)
- window the SpendLogs side of both queries (1s margin: the two writers
can disagree by ~1ms on the same request)
- expire SpendLogToolIndex rows on the spend-log retention cutoff via a
parametrized batch-delete engine shared with the SpendLogs cleanup
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(guardrails): add /v1/messages support for Straiker plugin
- Pass prepared response data to Anthropic Messages streaming post-call hooks (litellm/llms/anthropic/chat/guardrail_translation/handler.py)
- Normalize Straiker request, tool, finish-reason, and mode fields across Chat Completions, Messages, and Responses APIs
* fix(guardrails): gate cross-surface message resolution and cover streaming request data
Resolve request messages only for surfaces that have a mapped translation
handler. The unguarded fallback tried every registered handler in turn, which
raised AttributeError out of the guardrail's error handling on list-shaped
`input` bodies, and synthesized a chat message that was never sent for bodies
it happened to parse.
Prepare request data on the mid-stream Anthropic branch as well, matching the
terminal branch and the OpenAI handler, so guardrails that scan before
end-of-stream still receive identity metadata.
Read usage from Anthropic dict responses so non-streaming /v1/messages reports
token counts instead of null.
Add regression coverage for the streaming request data on both the terminal and
mid-stream branches; reverting either now fails.
---------
Co-authored-by: cs-mehta <chandra@straiker.ai>
anthropic_messages was missing from the cache's supported call types, so every /v1/messages request went to the provider. Adding it alone is not enough: the cache key is built from the OpenAI-ish param set, which has no system, top_k or stop_sequences, so two requests differing only by system prompt shared an entry and the second got the first one's answer. The Anthropic Messages request shape now feeds the key set as well.
Streaming responses return to the caller before async_set_cache runs, so they are teed on the way out and the SSE events are stored verbatim once the stream reaches message_stop without a provider error. A hit replays those bytes and logs the request as a cache hit with zero cost.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A toolset row is {server_id, tool_name}. The server is already identified by
server_id, so the stored name is the tool's own name and there is nothing for a
prefix to disambiguate. Resolution nevertheless reduced the stored name by the
server's wire prefix, which is a guess about a string that carries no such
marker.
The wire prefix is added on the way out and is not part of any tool's identity,
so when a native tool name happens to begin with it the guess renamed the tool:
a row for greyhound_internal_events on a server prefixed greyhound resolved to
internal_events. That is a different tool on the same server, so the selected
tool disappeared from /toolset/<name>/mcp and an unselected sibling was served,
and executed, under the selected tool's wire name. Toolsets are the tool-level
permission boundary, so the row granted access to something never selected.
Match the stored name as written and keep stripping the prefix off the live name
only. This is the only producer that rewrote allowlist values; every other one
stores what the admin typed, so the tools/list filter, the tools/call permission
check, the REST listing and the Responses API path are all corrected without
touching them.
A row that stores an already-prefixed name no longer resolves. Such a row names
a tool that does not exist on the server, and the dashboard has never written
one; it was only ever accepted because of the guess this removes.
The savings readers (extract_compression_saved_tokens, feeding
compression_saved_tokens on the daily spend tables) key exclusively on
tokens_saved in the guardrail_response stats, but the Headroom guardrail
builds those stats as a filtered pass-through of the compression service
response and the live service omits tokens_saved. Every compressed request
recorded 0 saved tokens on the Cost Optimization dashboard.
Derive tokens_saved = tokens_before - tokens_after when the key is absent
and both operands are numeric; a service-sent value still wins. The two
sibling writers (compresr, native compression interception) already derive
it the same way.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On passthrough requests the shared guardrail plumbing still dispatches
headroom's pre_call apply_guardrail, but the passthrough translation hands it
only `texts` and no `structured_messages`, so it early-returns a no-op. The
@log_guardrail_information decorator then synthesized an "allow"/"success"
StandardLoggingGuardrailInformation entry, and the unified hook added the
guardrail to applied_guardrails, so spend logs reported the compression
guardrail as succeeded even though nothing ran.
Add a records_own_guardrail_information flag for guardrails that log their own
execution (headroom). The decorator skips the synthetic success entry for them,
and the unified hook lists such a guardrail in applied_guardrails only when it
actually recorded a run. A guardrail that owns its logging must record every
outcome it runs, so headroom now records a guardrail_failed_to_respond entry on
the fail_open path (compression attempted, service unreachable, request
forwarded uncompressed) instead of leaving it unlogged; fail_closed is still
recorded by the decorator's error path, and a genuine no-op stays not_run.
The End User filter listed every row of LiteLLM_EndUserTable, which is both
unscoped and the wrong source. Team admins and internal users can open the
Logs page, and their log view is already restricted to their own requests
plus the teams they administer, but the filter dropdown offered them every
end user on the proxy.
Team attribution only exists on spend logs, so /customer/aliases now reads
LiteLLM_SpendLogs and applies the same scoping /spend/logs/ui does: a proxy
admin sees the whole window, everyone else sees ("user" = caller OR team_id
IN permitted_teams), reusing _get_permitted_team_ids_for_spend_logs so the
two paths cannot drift. A caller with neither matches FALSE rather than
falling through to unscoped, and a failed team lookup degrades to
own-rows-only.
Querying spend logs safely is the other half. start_date/end_date are now
required, so the query always has the indexed startTime bound, and the
inner scan is capped at MAX_SPENDLOG_ROWS_TO_SCAN_FOR_FILTERS rows ordered
by startTime DESC. DISTINCT therefore runs over a bounded row set instead
of the whole table the way /global/all_end_users does.
Also adds /customer/aliases to spend_tracking_routes. Without it RouteChecks
rejects INTERNAL_USER and INTERNAL_USER_VIEW_ONLY before the handler runs,
which would have made the scoping above dead code; a test pins the route to
the same access tier as /spend/logs/ui.
The dropdown now shows the end users present in the window the table is
showing, so the filter list matches what it filters. formatLogsWindow is
shared with the logs query so the two windows cannot diverge.
The guardrail-information writer picked its metadata bucket with a hand-rolled
precedence that preferred a caller-supplied `metadata` field, while every reader
resolves the bucket through `get_metadata_variable_name_from_kwargs`, which
prefers `litellm_metadata`. The two rules agree only when the caller sends no
`metadata` of its own. Routes in `LITELLM_METADATA_ROUTES` seed `litellm_metadata`,
so on /v1/messages and /v1/responses a caller that sends `metadata` sent the entry
to a dict nothing reads; the spend log then reported `guardrail_status: not_run`
with no `guardrail_information` even though the guardrail ran and the
`x-litellm-applied-guardrails` header was present.
Give the resolver one owner. `get_or_create_metadata_bucket` moves from the proxy
layer into core_helpers next to the resolver it calls, so `litellm/integrations`
can reach it without a proxy dependency, and the byte-identical duplicate of
`get_metadata_variable_name_from_kwargs` in callback_utils is deleted. The writer
now shares that owner with `add_guardrail_to_applied_guardrails_header`, so the
response header and the spend log can no longer disagree.
Two readers had to move with it or the fix would be a no-op on the affected
routes. `_sync_guardrail_info_to_logging_obj`, which bridges request_data into the
spend-log payload for passthrough routes, picked the first truthy bucket, so a
non-empty caller `metadata` short-circuited it. The otel failure-path span reader
`_emit_guardrail_spans_from_request_data` read a hard-coded `metadata` key, which
also dropped the span whenever the entry lived in `litellm_metadata`.
Model Armor already resolved the bucket for its file-scan results but wrote its
text-scan and post-call results, and read them back in `_process_response`,
through a hard-coded `metadata` key; on a seeded route that split the record so a
file scan's evidence never reached the logger. All four Model Armor sites now use
the shared resolver. The unified guardrail hook seeds `litellm_metadata` on every
route, so the OpenAI moderation entry lands there too; spend-log output is
unchanged because `merge_litellm_metadata` reads both buckets.
Enterprise MCP users mint virtual keys against tool access groups rather than
explicit server ids. Nothing exercised that end to end.
Registers the upstream MCP server tagged with a server-side access group
(mcp_access_groups), then mints one key granted that group and one granted a
different group. Asserts the granted key sees the tagged server's tools on
tools/list and the other key sees none, so access-group scoping can't leak the
server across the boundary.
Adds mcp_access_groups support to the e2e MCP client (server registration, key
generation, ObjectPermission) and the registry cell
mcp.list_tools.api_key.access_group_scoped.
The http_handler pair only received its full mutation verdict after the
first removal batch landed; these five tests pass unchanged when every
function they execute is mutated and the owning file killed none of
their scored mutants. The ssl tests excluded from mutation scoring are
untouched.
Opening Logs > Filters fetched the entire customer table through
/customer/list, which is an unbounded find_many that eagerly loads the
budget and object-permission relations for every row. On a proxy with
61k customers that is a 20 MB, 7.6 s response; the dropdown then built an
option per row and rendered all of them, since the combobox does not
virtualize. The result was a multi-second freeze every time the drawer
opened.
Adds GET /customer/aliases, a projection of user_id alone with page/size/
search, mirroring /key/aliases. The End User field now uses
PaginatedSearchSelect behind an infinite query, the same shape the Key
Alias and Model filters already use, so it fetches 50 rows at a time and
pushes the typed query to the server.
The response reports has_more instead of a total count. A total needs
COUNT(*) over the whole match set on every keystroke, which is the cost
this endpoint exists to avoid; ordering by the user_id primary key and
fetching one row past the page lets Postgres stop early and still tells
the client whether to request more.
LIKE metacharacters in the search term are escaped, because end-user ids
routinely contain underscores and an unescaped one silently widens the
match.
Drops the now-unused accessToken prop threaded from RequestLogsPanel
through RequestLogsTable into the filters.
Under otel_v2 a single MCP tool call surfaced in APM as two disconnected
traces joined only by a span link: the HTTP transport transaction
POST /{mcp_server_name}/mcp and the tools/call span carrying
error.type=MCPToolResultError. resolve_mcp_span_context parented the MCP
span to the W3C trace context the client propagates in params._meta
(SEP-414) and recorded the transport as a link, so with no traceparent
propagated (the common case today, including MCP Inspector) the span
started its own root trace.
Nest the MCP span under the transport span when nothing is propagated, so
the call stays in one trace; the propagated-context path is unchanged and
still parents to the remote context and links the transport per the OTel
GenAI MCP semconv.
The transport has to be resolved per message rather than read from the
request-root ContextVar. A stateful streamable-HTTP session runs every
message on the single task the session's initialize POST spawned, so that
ContextVar is frozen at initialize inside the handler: live capture on
staging showed the tools/call span linking the initialize POST rather than
the POST that carried it, and nesting on that anchor would hang every tool
call of a session off the first request's already-ended span. The gateway
now resolves the current request's span on the ASGI task and carries it to
the handler on the authenticated-user object, the same way per-request auth
already crosses that boundary.
* fix: handle explicit outputInfo: null in Vertex AI batch response
Vertex AI can return HTTP 200 for a create_batch/get_batch call with an
explicit "outputInfo": null body (the output directory is assigned
asynchronously and may not be populated yet at response time).
_get_output_file_id_from_vertex_ai_batch_response did:
response.get("outputInfo", OutputInfo()).get("gcsOutputDirectory", "")
dict.get(key, default) only substitutes default when the key is absent,
not when it is present but explicitly None, so this crashed with:
AttributeError: 'NoneType' object has no attribute 'get'
surfaced to callers as an opaque openai.InternalServerError 500 from
litellm.create_batch()/retrieve_batch() for any Vertex AI batch job,
regardless of whether the job ultimately succeeds.
Fixed by guarding with `response.get("outputInfo") or OutputInfo()`,
matching the existing null-safe pattern already used by the sibling
_get_input_file_id_from_vertex_ai_batch_response for inputConfig. The
existing outputConfig fallback branch (a few lines below) already
handles this case correctly once it's reachable - it just never was.
Added 2 regression tests covering outputInfo: null with and without an
outputConfig fallback available.
* test: drop explanatory comment from regression test
---------
Co-authored-by: htourinho-clgx <htourinho@cotality.com>
The gateway acts as an MCP client toward upstream MCP servers, and the MCP
authorization spec requires an MCP client to send the RFC 8707 resource
parameter on both the authorization request and every token request. The
gateway sent it on none of its upstream OAuth legs, so an authorization server
that requires resource indicators rejected the exchange with invalid_target
with no way to configure around it.
Authorization servers disagree irreconcilably and nothing advertises which
camp they are in, so this is a per-server opt-in rather than a default: most
providers ignore the parameter, some hard-reject it and carry audience in
scopes instead, and strict or MCP-native ones refuse to mint a correctly
scoped token without it. The new upstream_resource setting is unset by
default, which keeps today's requests byte-identical.
Both outbound OAuth stacks resolve the value from the server exactly once and
carry it structurally rather than attaching it per call site. In v1 every
plain-OAuth2 token leg builds its body through one helper that resolves the
resource in the same call as the mandatory client authentication; in v2 the
adapter, the single place an MCPServer becomes an outbound config, resolves it
onto the client_credentials config that the HTTP/SSE M2M path uses, and it
joins the config's mint identity so retargeting a live server refreshes the
token rather than serving the previous audience's. A leg cannot authenticate
without also naming the resource its sibling legs named, which is what an
attach-per-call-site approach kept getting wrong.
The setting is non-secret admin config sharing a blob with real secrets, and
the backend classifies which key is which rather than nulling the blob
wholesale or gating on its truthiness: redaction returns admin config to an
admin, session inheritance ignores it when deciding whether a real credential
was supplied and carries it onto the derived server, and the edit form renders
the same shared OAuth component as create so the field exists on both, an
emptied field submitting an explicit null that the credential merge drops.
