Adds 'agentcore' as a search provider backed by an AgentCore Gateway MCP web-search target, usable from litellm.search()/`/search` and as a websearch_interception backend. Supports SigV4 (AWS_IAM gateways) and bearer tokens (CUSTOM_JWT gateways) via a new BaseSearchConfig.sign_request hook.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
xAI already converts Responses usage to chat Usage so web_search_calls survive
cost tracking. The chat completions bridge then re-ran the Responses usage
transform and crashed on missing input_tokens. Pass through already-chat Usage
and chat-shaped dumps instead
Gate web search like OpenAI (output/annotations/web_search_requests).
xAI uses server_side_tool_usage_details only for per-call cost math, with
web_search_requests mirrored in llms/xai for existing gate compatibility.
Treat positive web_search_calls as a web-search signal in built-in tool
cost gating, and mirror counts onto prompt_tokens_details.web_search_requests
when attaching xAI tool usage details so charges are not skipped.
Use usage.server_side_tool_usage_details.web_search_calls at $5/1k calls
instead of legacy num_sources_used/web_search_requests. Preserve tool usage
details through Responses usage transform for accurate response cost.
A poll of a Vertex passthrough batch wrote nothing to the managed-object row,
so status and file_object stayed frozen at the create-time snapshot and
GET /v1/batches served a stale status and an empty output file id for the life
of the batch. Only the create may claim a batch, but every observation of one
may refresh its state.
store_unified_object_id takes create_if_missing, which the poll clears: it
refreshes status and file_object through update_many, and leaves a row that is
absent absent rather than creating one owned by the observer, since created_by
and team_id are written by whoever reaches the create branch. The update payload
is now shared with the upsert so it cannot drift into writing api_key,
request_tags, created_by or team_id.
The passthrough identity re-assertion that was previously part of this PR ships
separately in #36121, so this PR keeps only the batch attribution work.
The creating key owns user_api_key_alias only when it actually has one. Guarding
the overwrite on the presence of a key rather than on a resolved alias nulled the
field out for every key generated without key_alias, and for any key rotated or
deleted before its batch finished, losing the creating user's alias that the spend
row previously carried. The guard now matches the team-alias line below it.
* fix(otel): mark v2 server spans as failed for pre-call errors (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): authenticate malformed-body requests before rejecting them (LIT-4780)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): cover malformed-body rejection when auth error is recovered
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): skip authorization for a request whose body never parsed
Deferring the parse failure ran the full auth phase, including budget reservation, whose reserved amount is only released by the endpoint's post call path; the endpoint never runs, so malformed requests leaked reservations and locked a budgeted key out. Authorization now runs only when the body parsed, and a parse failure with a rejected key keeps returning the 400 it returned before.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: shivam <shivam@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Since #35491, every Router joins the module-global _live_routers weak set at
construction, and every model cost map swap replays the deployments of every
member on top of the freshly adopted map. #36039 isolated the register_model
ledger half of that replay but not this half: under pytest-xdist, a Router
created by an earlier test in the same worker that was still referenced (or
simply not yet garbage collected) re-registered its deployments during
TestPriceDataReloadIntegration::test_distributed_reload_check_function, and
register_model hydrated the sparse mocked gpt-3.5-turbo entry into a full
ModelInfo dict, failing the exact-equality assert (reruns cannot help since
the polluting router survives in the worker process)
The autouse isolate_litellm_state fixture now snapshots _live_routers before
each test and restores its membership on teardown, so a test's routers stop
contributing to cost map rebuilds once the test ends. A canary pair in
test_conftest_isolation.py asserts the rollback
require_managed_files was only checked on upload, so raw provider ids still
reached the batch, fine-tuning and vector store file routes. Ownership rows
exist only for managed ids, so those requests were forwarded under shared
credentials with no tenant check: knowing another tenant's id was enough to
read, run against, cancel or delete their object.
Generalise the file-id guard to validate_managed_id_requirement(resource_id,
resource_kind) and call it on batch create/retrieve/cancel, fine-tuning
create/retrieve/cancel (training_file and validation_file both) and the shared
vector store file id resolver. Behaviour is unchanged when the setting is off.
A budget rule that graduates into a config's hard-fail select list rightly
leaves the budget file, but the ratchet guard read any disappearance as a
silently raised ceiling. Teach it the pairing between ruff-strict-budget.json
and ruff.toml: a dropped rule is excused only when the paired config's
lint.extend-select (minus lint.ignore) now hard-fails it, so deleting a rule
without graduating it still trips the guard.
Every strict-gate rule whose budget ceiling was already 0 moves into the base
config's lint.extend-select, so editors and ruff check --fix surface the
diagnostics directly and the budget file shrinks to rules with real debt.
Graduates stay in ruff-strict.toml's select so the strict RUF100 pass keeps
policing their stale noqa directives, and base external entries they made
redundant (FURB, I001, RUF010, RUF022, RUF023, RUF051) are dropped so base
RUF100 polices those directly. UP037 had two violations hidden behind a star
import; importing Literal explicitly fixes them so UP037 can graduate too.
New drift tests pin the invariants: every strict-selected rule is budgeted or
hard-failed by base, every base-owned rule stays visible to exactly one
RUF100 pass, and graduated rules fail the normal ruff run.
HTTPHandler.post and AsyncHTTPHandler.post call raise_for_status before returning, so the status_code != 200 branches after the create and cancel POSTs could never run. Non-2xx already surfaces as httpx.HTTPStatusError from inside the client. The checks after GETs stay: the get helpers return without raising. Tests that faked a non-raising POST response are replaced by HTTPStatusError propagation coverage.
--use_v2_migration_resolver had no env var, and the Helm migrations job runs
prisma_migration.py, which calls run_server with a fixed argv. There was no
seam to pass the flag, so a Helm install could not reach the v2 resolver at
all. Reading it from the environment makes the existing flag configurable
from a deployment.
* feat(guardrails): chunk oversized Bedrock ApplyGuardrail requests instead of failing
AWS's ApplyGuardrail API rejects requests whose content exceeds the
account's per-request "maximum input size in text units" quota with a
400 ValidationException. That cap is account/region/policy-dependent
and cannot be predicted from config, so it can only be reacted to.
_make_apply_guardrail_request now tries the whole-content call first
(no behavior change for requests that already fit). On a too-large
ValidationException it bisects the flat content list and retries each
half sequentially, recursing until every piece fits or cannot be split
further, then merges the per-chunk responses (action, assessments,
outputs, usage) into one so callers cannot tell chunking happened. A
real guardrail block on any (sub-)chunk still raises immediately.
Contextual-grounding requests are never chunked: grounding scores the
response holistically against the whole reference source, so
fragmenting it would produce misleading scores.
Each chunk call also gets a small exponential backoff retry on AWS
ThrottlingException (429), since chunking increases the number of
per-second API calls and can trade a 400 for a 429.
All new state is local to a single request's call stack (no shared
cache, no cross-process coordination), so this is safe for
single-pod, multi-pod, and cache-less LiteLLM proxy deployments alike.
* fix(guardrails): address Bedrock ApplyGuardrail chunking review feedback
Fixes three issues flagged in review of the chunking fallback: a single
oversized content item couldn't be split (only list-length bisection was
supported), a chunked request that got recovered still logged a stray
failure telemetry entry alongside the real outcome, and flattening chunk
outputs without positional bookkeeping could misalign masked text onto
the wrong original message once a chunk had nothing to mask.
* test(guardrails): add regression test for multi-level Bedrock guardrail chunking
Confirms the too-large bisection recursion isn't capped at a single split:
a payload that is still oversized after the first halving keeps splitting
until every piece fits, converging on however many chunks it takes rather
than only ever producing two.
* fix(guardrails): hybrid bin-pack+bisection chunking, whitespace-safe splits
Rework Bedrock ApplyGuardrail chunking from pure reactive bisection to a
hybrid strategy: bin-pack content into fixed-budget batches up front as
the fast path, falling back to the existing recursive bisection only for
a batch AWS still rejects as too large. Avoids paying O(log n) round
trips on every oversized request when a single pass would do.
Also switch single-item text splitting from a raw character midpoint to
the nearest whitespace boundary, so a fragment never starts or ends
mid-word. Closes the accidental-severing case from review; the residual
gap (a multi-word denied phrase deliberately straddling the boundary) is
documented as an accepted limitation, since fixing it would require an
overlap window reconciled against masked output with no documented
length-preservation guarantee from AWS.
* chore(ui): regenerate dashboard API types
* fix(guardrails): don't retry an oversized Bedrock guardrail call as a throttle
AWS reports an ApplyGuardrail request that exceeds the per-request
text-unit cap as a ThrottlingException (429), not only as the documented
ValidationException (400). Verified against a live guardrail with an
active content-filter policy: a 3273-text-unit request comes back as
"Input text size (3273 text units) exceeds the maximum allowed (1000 text
units) for the content filter policy (Classic tier)".
The throttle retry keyed off status 429 alone, so every oversized chunk
burned the full backoff-retry budget - each attempt a billed AWS call
preceded by a sleep - before the bisection fallback got a chance, at every
level of the recursion. A size error is not transient; re-posting the same
content can never succeed. It now short-circuits straight to bisection.
Also rename _is_input_too_large_validation_error to
_is_input_too_large_error (it never keyed off the status code, and the
error is not always a ValidationException), correct the docstrings that
asserted a 400, and log at warning level when a split happens so the
recovery is visible without --detailed_debug.
* Revert "chore(ui): regenerate dashboard API types"
This reverts commit ebf8ba2fd57f13bccf7aa6c5dfcac41c74db1ed9.
* fix(guardrails): group all fragments of one item and stop double-logging
Two defects found in review, both invisible to the existing tests.
Fragment grouping assumed a split content item always produces exactly two
adjacent fragments. That holds for one bisection level but not two: an item
split twice yields four fragments, which were regrouped in fixed pairs into
two output entries for a single message. Since masking walks the merged
outputs by a running index across the original, unchunked message list, that
message was written back truncated to its first half and every later message
shifted. Fragments now carry the size of the group they belong to, so any
number of them collapse back into exactly one output entry.
Telemetry was also double-counted. AsyncHTTPHandler.post calls
raise_for_status(), so every non-200 from Bedrock reaches _sign_and_post's
error path, which logged guardrail_failed_to_respond before re-raising as an
HTTPException that the consolidating caller then logged again. A request
recovered by chunking reported one failure per rejected attempt plus a
success. The ApplyGuardrail path now opts out of that per-attempt logging,
since it owns consolidated per-request logging; the connection-level branch
still logs, as nothing else records it.
The existing tests missed both because their mocks return a non-200 response
object, while the real client raises. Added a helper that raises a genuine
httpx.HTTPStatusError so these paths are covered the way production hits
them, plus a case asserting an unrecoverable failure still logs exactly once
rather than zero times.
* refactor(guardrails): move Bedrock chunking rationale into docstrings
The chunking work explained itself with inline comment blocks, which this
repo's conventions do not want. Folded that reasoning into the docstrings of
the functions it describes and dropped the comments, including the
module-level constant blocks and the test-file banner.
No behavior change. The banner also claimed AWS rejects an oversized request
with a 400 ValidationException, which live testing disproved, so removing it
drops a stale claim as well as an internal ticket reference from a public repo.
* feat(guardrails): match AWS default chunk budget and make it configurable
ApplyGuardrail's default quota is 25 text units, roughly 25,000 characters,
per second. Chunking has to respect that throughput limit rather than just the
per-request size, otherwise splitting an oversized request trades a size error
for a throttle. The budget now defaults to 25,000 to match that default for
every user, up from an arbitrary 20,000.
Accounts with raised quotas can spend fewer calls by setting
chunk_budget_chars on the guardrail. A value AWS still rejects as too large is
bisected automatically, so an over-large setting costs an extra round trip
rather than failing the request.
* fix(guardrails): never split a Bedrock text into an empty fragment
_nearest_whitespace_split_index could return len(text) when the only space at
or after the midpoint was the final character, so the first fragment came back
identical to the text AWS had just rejected as too large and the second came
back empty. AWS rejects the unchanged fragment again, and each retry re-splits
it into the same fragment, so an oversized single item shaped like a long
unbroken token with one trailing space exhausted the stack with a
RecursionError instead of scanning or surfacing Bedrock's error.
Candidate boundaries that would leave either side empty are now discarded, and
the raw midpoint is used when none remain. The midpoint is always safe because
_split_bedrock_content only calls this for text of at least two characters.
* style(guardrails): move chunking rationale out of comments and into docstrings
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* fix(guardrails): raise 500 when Bedrock reports a failure inside a 200 body
Restores the source changes intended for a08e4cf309, which landed with only the
test. Also types the credentials parameter on the new chunking helpers and rebuilds
fragment grouping without mutating a list or rebinding an index
* style(guardrails): sort the constants import into the first-party block
* refactor(guardrails): bring the Bedrock chunking path under the LIT lint budgets
Annotates never-rebound locals with Final, replaces the retry counter and the two
branch-assigned locals with single bindings, and moves the internal chunking chain
to Sequence parameters and tuple returns. Collections that reach the logged payload
stay lists on purpose: redact_nested_match_and_regex_keys only traverses dict and
list, so a tuple would carry PII past redaction. The remaining constructions are
contract-bound and carry inline reasons
* fix(guardrails): keep the pre-chunking contract for failures reported inside a 200
Reverts the 500 this branch introduced for an AWS 200 whose body carries an
Output.__type exception marker: the request proceeds as it did before chunking
existed. The logged status is now derived from the merged response instead of
being hardcoded to success, so that shape is still reported as
guardrail_failed_to_respond. The consolidated failure logger also goes back to
logging a dict rather than a bare string, matching both the pre-chunking code and
the InvokeGuardrailChecks path in this file
* docs(guardrails): correct the docstring for failures reported inside a 200 body
The raise was reverted, so the docstring no longer describes the code. Records that
the request proceeds by design and points at LIT-5338 for closing the fail-open path
behind the existing unreachable_fallback setting
---------
Co-authored-by: spencer-burridge <265588760+spencer-burridge@users.noreply.github.com>
