codecov/patch flagged the diff at 75.4% against the 77.1% target. The
uncovered regions were real gaps, not noise: the list/get/stop endpoints
had no tests at all, and neither did the judge's failure paths.
Lifecycle endpoints: list returns newest-first without results, get
aggregates verdicts for one job (404 on unknown), stop completes an
active job and returns its verdicts, stopping a finished job is a 400
that writes nothing, view-only admins can list but not stop.
Judge failure modes: a provider error or unparseable verdict returns
None, bumps failed_count, and never writes a verdict row.
Local coverage on the two flagged files: 79% -> 88%.
Co-Authored-By: Claude <noreply@anthropic.com>
Zero-estimate cap bypass (cursor): a key quiet during the estimate
lookback gets cost_estimate 0.0, which the spend cap treated the same
as 'no estimate' and left uncapped — a later traffic spike on exactly
that job would bill until ends_at. Only a NULL estimate (rows predating
estimates) is uncapped now; $0 still gets the $1 floor.
Concurrent-start race (cursor): the find_first-then-create check passes
on both sides of a race, giving one key two active jobs and double
judge spend. A partial unique index (api_key_id WHERE status IN
(pending, running)) — raw SQL in the unshipped migration, since
schema.prisma cannot express partial indexes — makes the DB the
arbiter; the losing create surfaces as the same 409 as the advisory
check. The old find_many('desc') + reversed() insertion in the logger
cache already prefers the newest job for any legacy duplicates.
Request-path DB read (greptile): an expired job snapshot awaited
find_many inside the success callback, so every N seconds one request
per pod paid a synchronous Prisma read. The lookup is now sync-only:
it serves the current snapshot and kicks a detached refresh task when
stale. Cost: a cold pod's first ~1 refresh-window of samples are
missed (acceptable for a sampled eval); stale-if-error semantics keep
a DB blip from disabling the feature.
Also from review: a collapsed previous-job row said 'no verdicts' for
jobs with thousands of verdicts, because the list endpoint omits
results by design — it now says 'view results' when completed_count>0.
Co-Authored-By: Claude <noreply@anthropic.com>
The per-key TTL cache did one indexed find_first per distinct key hash
per 30s — fine at small scale, but on a proxy serving 10k active keys
that is hundreds of small reads per second across pods, all to discover
that almost every key has no job.
Cache the entire active-job set instead: one find_many per pod per TTL
(the set is admin-started and capped at one job per key, so it is
single-digit rows), served to every key as an in-memory dict hit. DB
load is now flat and constant in the number of keys, idle or active.
On a DB blip the stale snapshot is kept and the next TTL retries, so a
blip degrades freshness rather than disabling the feature. Concurrent
requests share one refresh behind a lock instead of stampeding.
Adds @@index([status]) for the status-only find_many, folded into the
unshipped shadow eval migration.
Co-Authored-By: Claude <noreply@anthropic.com>
The start endpoint counted the key's trailing 7-day requests directly
against LiteLLM_SpendLogs, which has no api_key index — on a busy proxy
that is a scan over every request in the window (potentially tens of
millions of rows) to answer one count, holding a multi-second query per
'Start shadow eval' click.
Read SUM(api_requests) from LiteLLM_DailyUserSpend instead: one indexed
row per key/day, the same table the usage dashboards already use for
this question. A regression test asserts the estimate queries the
rollup and never touches LiteLLM_SpendLogs.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds a "Shadow eval" button next to the Auto-router usage heading that
smooth-scrolls to the shadow eval section, so it's reachable without
scrolling past the benchmarks body first.
Also fixes a PR review comment (veria-ai): shadow and judge calls ran
outside the normal auth path, so they never went through
reserve_budget_for_request and could push an already-exhausted key or
team further over budget before their own spend was even recorded.
_key_or_team_is_over_budget reads the same cross-pod spend counters
that path reserves against (via the existing get_current_spend) and
skips the shadow/judge pair outright when the shadowed key or its team
is already at or over budget. This is a read-time check, not a
reservation — appropriate for a best-effort background measurement
task, not a billed user request — so it narrows the window rather than
closing it against concurrent bursts, which the response comment
explains.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover malformed-body rejection when auth error is recovered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): skip authorization for a request whose body never parsed
Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A shadow eval samples ongoing traffic, so a job without an end date keeps
billing judge calls until someone remembers to stop it — and the upfront
estimate silently priced exactly one week regardless. Jobs now take a
duration_days (1-30, default 7): the start endpoint stamps ends_at, the
estimate scales trailing volume to the requested window, and the logger
completes a job past its window through the same guarded update + cache
eviction path as the spend cap (generalized into _finalize_job). The
existing shadow eval migration is amended in place since it has not
shipped anywhere yet.
The start form no longer asks anyone to paste a key hash: the key is a
type-to-search combobox backed by /key/list alias substring search that
submits the token, the auto-router is a filter-as-you-type combobox fed
by the configured auto-router deployments, and duration is a select.
Active job cards show when the job will end.
The judge model field is now labelled as such, with guidance: judging
two answers blind needs solid comprehension and reliable JSON, not
frontier reasoning — a mid-tier model (Claude Sonnet / GPT-4o class) is
recommended, nano/mini-class judges give unreliable verdicts, and
frontier reasoning models add cost without changing outcomes. Same
guidance mirrored into the API field description.
Co-Authored-By: Claude <noreply@anthropic.com>
Shadow and judge calls were fired with no caller identity on their metadata.
The proxy's cost callback requires user_api_key/_team_id/... to log spend and
apply budget checks, and silently drops the entry without them — so an
admin-enabled eval billed real provider spend that landed on no key, no team,
and no budget counter, invisible to every limit the shadowed key is normally
subject to.
Extract the identity-forwarding rules the auto-router classifier already
implements into a shared litellm/litellm_core_utils/internal_call_metadata.py:
forward the caller's identity subset, strip the parent's budget reservation
(top-level and the copy nested in user_api_key_auth) so a sub-call can't
finalize a reservation that belongs to the parent, and stamp the sub-call's
origin. The classifier now uses this module instead of its own copy.
Wire both shadow eval call sites (_call_router_shadow, _call_judge) through
it, and add a per-job spend cap (job cost_actual >= max(3x the quoted
estimate, $0.50)) so a bad estimate or a traffic spike can't turn a quoted
eval into a runaway bill; a capped job self-completes and is evicted from
cache so it can't be resurrected by an in-flight request.
Also surface api_key_id/team_id on GetShadowEvalJobResponse so an admin
running several jobs can tell which key's traffic a given win rate belongs
to, and regenerate the dashboard's OpenAPI types for the new fields.
Co-Authored-By: Claude <noreply@anthropic.com>
Three fixes from the live end-to-end run:
- The judge ran with max_tokens=200, which truncated roughly 12% of
verdicts mid-JSON so they were lost to failed_count. Raise it to a
named JUDGE_MAX_OUTPUT_TOKENS=500 and price the upfront cost estimate
off the same constant, so the estimate can't silently drift from what
the judge is actually allowed to emit.
- The UI only ever rendered the newest job, so starting a new eval hid
the results of a populated older one. Prior jobs are now listed in a
collapsible 'Previous evaluations' card, each expandable to its own
per-tier results.
- Move the shadow eval section above the benchmarks body: pre-adoption
keys have no router sessions, so it was buried under an empty state.
It stays outside BenchmarksBody so it survives that early return.
Co-Authored-By: Claude <noreply@anthropic.com>
Since #35491, every Router joins the module-global _live_routers weak set at
construction, and every model cost map swap replays the deployments of every
member on top of the freshly adopted map. #36039 isolated the register_model
ledger half of that replay but not this half: under pytest-xdist, a Router
created by an earlier test in the same worker that was still referenced (or
simply not yet garbage collected) re-registered its deployments during
TestPriceDataReloadIntegration::test_distributed_reload_check_function, and
register_model hydrated the sparse mocked gpt-3.5-turbo entry into a full
ModelInfo dict, failing the exact-equality assert (reruns cannot help since
the polluting router survives in the worker process)
The autouse isolate_litellm_state fixture now snapshots _live_routers before
each test and restores its membership on teardown, so a test's routers stop
contributing to cost map rebuilds once the test ends. A canary pair in
test_conftest_isolation.py asserts the rollback
The lint job's strict-rule budget flagged 17 new violations. Rather than
raise the ceiling, this types the code properly:
- validate the judge verdict into a PairwiseVerdict pydantic model at the
parse boundary instead of dict[str, Any] + cast, which also removes the
defensive float()/str() coercion downstream
- replace the untyped job dict with a frozen ActiveShadowEvalJob dataclass
- validate the prisma job row into _ShadowEvalJobRow, replacing 11 no-op
'# type: ignore[attr-defined]' comments
- annotate the success hook and drop Any from the remaining signatures
- mark the genuine third-party dict shapes (prisma filters/payloads, SDK
message lists) with '# mutable-ok' reasons per the existing convention
The column was declared Float? with no default, so every row started NULL.
The verdict writer increments it, and NULL + x is NULL in Postgres, meaning
judge spend never accumulated and the UI always showed no spend.
Makes the column non-null with a default of 0 across all three schema copies
and the migration, and tightens the response model to a plain float.
- Remove unused imports (datetime, timezone, ModelResponse) flagged by ruff.
- Update test_cost_tracking_adds_two_callbacks_when_prisma_set to expect 2
callbacks on litellm.callbacks (not 1): ShadowEvalLogger now registers
alongside _ProxyDBLogger in cost_tracking(). Test name already said 'two',
now it actually tests for the correct count.
- Format ShadowEvalSection.tsx/.test.tsx per prettier.
Tests: 94/94 passing (lifecycle, shadow-eval, auto-router endpoints).
Lint: ruff + prettier all clean.
Co-Authored-By: Claude <noreply@anthropic.com>
require_managed_files was only checked on upload, so raw provider ids still
reached the batch, fine-tuning and vector store file routes. Ownership rows
exist only for managed ids, so those requests were forwarded under shared
credentials with no tenant check: knowing another tenant's id was enough to
read, run against, cancel or delete their object.
Generalise the file-id guard to validate_managed_id_requirement(resource_id,
resource_kind) and call it on batch create/retrieve/cancel, fine-tuning
create/retrieve/cancel (training_file and validation_file both) and the shared
vector store file id resolver. Behaviour is unchanged when the setting is off.
A budget rule that graduates into a config's hard-fail select list rightly
leaves the budget file, but the ratchet guard read any disappearance as a
silently raised ceiling. Teach it the pairing between ruff-strict-budget.json
and ruff.toml: a dropped rule is excused only when the paired config's
lint.extend-select (minus lint.ignore) now hard-fails it, so deleting a rule
without graduating it still trips the guard.
Every strict-gate rule whose budget ceiling was already 0 moves into the base
config's lint.extend-select, so editors and ruff check --fix surface the
diagnostics directly and the budget file shrinks to rules with real debt.
Graduates stay in ruff-strict.toml's select so the strict RUF100 pass keeps
policing their stale noqa directives, and base external entries they made
redundant (FURB, I001, RUF010, RUF022, RUF023, RUF051) are dropped so base
RUF100 polices those directly. UP037 had two violations hidden behind a star
import; importing Literal explicitly fixes them so UP037 can graduate too.
New drift tests pin the invariants: every strict-selected rule is budgeted or
hard-failed by base, every base-owned rule stays visible to exactly one
RUF100 pass, and graduated rules fail the normal ruff run.
HTTPHandler.post and AsyncHTTPHandler.post call raise_for_status before returning, so the status_code != 200 branches after the create and cancel POSTs could never run. Non-2xx already surfaces as httpx.HTTPStatusError from inside the client. The checks after GETs stay: the get helpers return without raising. Tests that faked a non-raising POST response are replaced by HTTPStatusError propagation coverage.
Resolves two real conflicts (the PR's actual base branch is
litellm_internal_staging, not main):
- litellm/types/management_endpoints/auto_router_endpoints.py: kept both
the Mapping and Literal imports, both used by pre-existing types.
- tests/e2e/proxy_client.py: kept upstream's more detailed create_model()
docstring covering multi-replica propagation.
Everything else auto-merged cleanly. Regenerated the OpenAPI schema
(schema.d.ts) to pick up upstream's tier_turns addition to the auto-router
benchmarks response.
Co-Authored-By: Claude <noreply@anthropic.com>
- Forward the original request's non-default params (temperature, tools,
response_format, etc.) to the shadow router call. Previously only model and
messages were sent, so a request with tools or a non-zero temperature was
judged against a shadow response generated under totally different sampling
settings -- an unfair, biased comparison. stream and metadata are still
stripped: the shadow call needs the full text back and must not leak the
caller's own metadata.
- Fix unbounded background task backlog: asyncio.create_task() fired
unconditionally and only the task body waited on a semaphore, so a traffic
spike queued unlimited tasks (each holding a copy of messages/response)
before any of them ran. Now the in-flight count is checked and incremented
before scheduling; over capacity, the sample is dropped instead of queued.
- Fix stopped jobs being silently reactivated: the verdict-write counter
update unconditionally set status='running', so a pipeline that started
before stop_shadow_eval_job() marked the job 'completed' could overwrite
that back to 'running' after the fact. Now it's a conditional update_many
scoped to status in (pending, running), so a completed job can never
transition back.
- Strip comments/docstrings added during the previous fix pass per
CLAUDE.md's no-new-comments rule (flagged by review bot).
Tests: 5 new regression tests (param forwarding, stream/metadata stripping,
backlog cap drops samples under saturation, backlog cap schedules+decrements
under capacity, stop-race status guard). 31/31 passing.
Co-Authored-By: Claude <noreply@anthropic.com>
- Registry check now scans all pre-routing strategy registries (auto, complexity,
adaptive, quality), not just auto_routers. Fixes 400 on adoption for
complexity-router or adaptive-router users.
- Split auth into _require_admin_viewer (GET) and _require_admin_writer
(start/stop); view-only admins can no longer initiate paid work (judge calls).
- request_count UPDATE buffering: in-memory counter flushed every 10s instead
of one UPDATE per request. High-traffic keys now cost one DB op per flush
interval, not per request.
- Default judge model: anthropic/claude-sonnet-5 (was unmapped claude-3-5-sonnet).
Cost estimation now prices correctly; fallback is no longer needed.
- completion_cost error handling: try/except around litellm.completion_cost() so
unmapped judge models don't crash the verdict write.
- UI: ShadowEvalSection now always renders (pre-adoption keys have no router
sessions yet but still show the start form). Added judge_model parameter to
the start form. Fixed accessToken undefined in AutoRouterBenchmarksTab.
Tests:
- test_shadow_eval_logger.py (26 tests): sampling, verdict parsing, unmasking,
skip logic, metadata isolation.
- ShadowEvalSection.test.tsx (7 tests): form, active job display, per-tier
results, low-sample flagging, completed job handling.
- All existing auto-router endpoint tests (22) and component tests (96) pass.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for a08e4cf309, which landed with only the
test. Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* style(guardrails): sort the constants import into the first-party block
* refactor(guardrails): bring the Bedrock chunking path under the LIT lint budgets
Annotates never-rebound locals with Final, replaces the retry counter and the two
branch-assigned locals with single bindings, and moves the internal chunking chain
to Sequence parameters and tuple returns. Collections that reach the logged payload
stay lists on purpose: redact_nested_match_and_regex_keys only traverses dict and
list, so a tuple would carry PII past redaction. The remaining constructions are
contract-bound and carry inline reasons
* fix(guardrails): keep the pre-chunking contract for failures reported inside a 200
Reverts the 500 this branch introduced for an AWS 200 whose body carries an
Output.__type exception marker: the request proceeds as it did before chunking
existed. The logged status is now derived from the merged response instead of
being hardcoded to success, so that shape is still reported as
guardrail_failed_to_respond. The consolidated failure logger also goes back to
logging a dict rather than a bare string, matching both the pre-chunking code and
the InvokeGuardrailChecks path in this file
* docs(guardrails): correct the docstring for failures reported inside a 200 body
The raise was reverted, so the docstring no longer describes the code. Records that
the request proceeds by design and points at LIT-5338 for closing the fail-open path
behind the existing unreachable_fallback setting
---------
Co-authored-by: spencer-burridge <265588760+spencer-burridge@users.noreply.github.com>
The suite already waits for a new model or agent to become servable before
handing it back, but that wait returns on the first successful read. Every
request opens a fresh connection (e2e_http calls requests.* with no Session), so
a load-balanced Service routes each one independently: one successful read proves
one replica converged, and the caller's next request re-rolls and can land on a
replica that has not reloaded yet.
At replicaCount: 2 this surfaced as 30 failures on a SHA that is green at 1
replica -- 400 "Invalid model name passed", 404 "Guardrail not found", "no
healthy deployments for this model", and a /model/info listing that contained
one of two models created moments apart.
Add PROPAGATION_TIMEOUT (default 15s, override E2E_PROPAGATION_TIMEOUT) and
settle_propagation(), sized off the proxy's proxy_config_reload_interval_seconds
(30s by default, 7s on the e2e stack) plus margin, and settle after every
control-plane create whose object the suite then uses:
- ProxyClient.create_model and A2AClient.register_agent, after their existing
polls -- the poll still fails loudly if the object never appears at all
- GuardrailsClient.register, which had no barrier; create_content_filter_guardrail
and create_bedrock_guardrail now route through it instead of POSTing directly
- the guardrail creates in mcp_client and logging_client
- the vertex passthrough model, whose body cannot go through create_model
Left alone: the /model/new calls that assert a 403 or read back a status code,
since they never use the model.
* fix(otel): name the RPC system and upstream on MCP tool-call spans
An MCP tool-call span carried only gen_ai.*, mcp.* and litellm.* attributes. A
CLIENT span holding none of the http/db/messaging/rpc families is
unclassifiable, so Elastic APM indexed these spans as span.type=unknown with no
span.subtype at all, and its span-links API then rejected the whole trace with
"Missing required fields (span.subtype)".
MCP frames every message as JSON-RPC 2.0, so the tool-call span now names
rpc.system. It names server.address and server.port alongside it, derived from
the already-redacted mcp_server_resource origin: naming the RPC system makes a
consumer treat the span as a downstream dependency and key that dependency off
the server address, so emitting one without the other labels the dependency
":0".
The tools/list span is left alone. It reaches the callbacks with no upstream
identity, and a listing can span several upstreams, so it has no address to
attach and would produce exactly that ":0" node.
The wire is untouched: streamable MCP still returns HTTP 200 with isError: true.
* fix(otel): drop rpc.system when no MCP upstream address resolved
server.address and server.port come from mcp_server_resource, which is absent
whenever the tool name resolves to no registered server, is None for a stdio
transport that has no host to log, and parses to no host for an IPv6 origin the
redactor rebuilds without its brackets. rpc.system was stamped unconditionally,
so each of those paths emitted it alone and named the dependency ":0", the
outcome the address pair exists to prevent.
Gating the system attribute on a resolved address makes the pairing structural
rather than leaving it to the two extractors happening to agree.
* fix(otel): require a full MCP destination before naming the RPC system
The gate gave rpc.system a resolved address, but not a resolved port. A
host-bearing scheme outside the HTTP(S) default-port map resolves an address
alone, and mcp_servers[].url is not scheme-validated, so an origin like
mcp://host or ws://host reaches the mapper and names the dependency host:0
instead of the :0 the previous commit removed.
Gating on the complete pair closes it, and covers a port of 0 as well.
_upstream_address_port also gets a direct contract test, including the IPv6
origin the redactor rebuilds without brackets.
* fix(otel): do not raise when an MCP origin has an unparseable port
_redact_mcp_resource_url rebuilds the origin without its IPv6 brackets, so a
zone-scoped address leaves a truthy hostname behind that the host check admits:
http://[fe80::1%25eth0]:80 becomes http://fe80::1%25eth0:80, whose hostname is
fe80 and whose port raises ValueError. That propagated out of
MCPToolCallSpanData.from_standard_logging_payload and cost the span.
Reading both halves inside a guard degrades an unparseable origin to no address,
which is already how the mapper treats an unresolvable upstream, and matches the
guard the redactor puts around the same split. The scheme default port drops the
dict literal so the LIT002 ceiling stays put.
* feat(proxy): return per-group model provenance on /team/info
/team/info now carries access_group_details, one entry per resolved access
group with its id, name, and model list, so the UI can attribute each
inherited model to the group granting it. The batch resolver returns the
access group rows keyed by id instead of a stringly dict of lists, and the
team member budget helper returns a copy instead of mutating its parameter.
Type discipline and basedpyright budgets ratchet down accordingly.
* feat(ui): allow group-only teams and show model provenance on hover
Team create and edit no longer require a model selection: an empty
selection is saved as the no-default-models sentinel, never as a bare
empty list, since an empty team model list means unrestricted access.
The team info Models card now renders every badge with a hover tooltip
naming how the team got that model: directly, via named access groups,
or both, and group-granted badges stay visible when the direct list is
empty or a sentinel.
* refactor(proxy): dedupe access group ids and return copies instead of mutating
Duplicate access_group_ids no longer amplify the /team/info response: ids
collapse order-preserving before provenance is built, pinned by a regression
test. The resolver returns a model_copy rather than mutating its parameter,
and the team create call sends a new object instead of reassigning
formValues.models. Budgets ratchet down further with the mutation removal.
* fix(proxy): resolve entity access groups in the model listing endpoints
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): reuse the fetched team object when listing models
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover key-level access group resolution in model listing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch): restore snippet text in native web_search_tool_result blocks (LIT-5315)
The build_web_search_tool_result_block method copied url/title/page_age but
hardcoded encrypted_content to empty string, never reading SearchResult.snippet.
This left every native block content-free, forcing clients to web_fetch each
result to recover evidence—the reported symptom.
The Anthropic spec carries page text only in encrypted_content (an opaque
server-issued blob we cannot mint), so snippet is emitted as an additive key
alongside the spec fields. encrypted_content stays empty rather than holding
plaintext, which would assert encryption semantics that don't hold.
The anthropic SDK's BaseModel sets extra='allow', so the additive snippet key
survives SDK parsing. litellm has no typed model for web_search_result at all,
so nothing drops it internally. Turn-2 replay behavior is unaffected: the
empty encrypted_content already exists today.
Tests:
- Updated test_shape_with_results to assert snippet present
- Added test_snippet_carried_for_every_result to cover multi-result ordering
- Added test_missing_snippet_degrades_to_empty_string for edge case
- Mutation check: reverting source-only yields 3 test failures, restored to 117 passed
Fixes: LIT-5315
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(websearch): make synthesized web_search blocks replayable by native clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(websearch): flatten a resultless replayed search block so Bedrock accepts the next turn
The flatten added for LIT-5315 bails when the replayed web_search_tool_result
carries an empty content list, but that is exactly what the interceptor emits
when a search legitimately returns nothing and when a search raises. The block
survived into the outbound body, Bedrock rejected the tag, and the conversation
died on the following turn just as it did before the flatten existed.
An empty content list has no encrypted_content to respect and no evidence to
preserve, so it flattens safely, and its paired server_tool_use goes with it.
The rendered text now says so explicitly rather than emitting a bare header.
Adds the multi-turn replay coverage that existed nowhere: the outbound Bedrock
invoke body is asserted free of both block types, parametrized over the
results-present and resultless cases, and built from the interceptor's own
builder so the fixture cannot drift from what it emits.
Resolves LIT-5320
* test(websearch): pin flatten idempotency for the agentic-loop re-entry
The agentic loop re-enters the same /v1/messages entry point for its follow-up
call and hands it the original client history, so the flatten runs again over
already-flattened messages once per iteration. Bedrock always takes that path,
since its config reports web search as natively handled and the short-circuit
is skipped.
A pass that appended the rendered text instead of replacing the block would
duplicate the evidence on every iteration and re-ship the unsupported tag, and
no existing single-pass test sees it. Mutation checked: keeping the original
block alongside the rendered text fails this test on its own.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
