Seven integration tests fail on main for reasons that are not product bugs. Two are order or race bugs in the tests, one reads rows other tests wrote, one passed on an error body, and three still expect behavior that #48644, #39864 and #39540 changed on purpose
* fix(proxy): stream a guardrail block on /v1/messages as Anthropic events
* fix(proxy): keep post-call guardrails off the streamed messages block
The streamed block is sent as-is, like the Responses endpoint's block, so post-call guardrails and streaming hooks never scan the proxy's own block message. Tests cover a post-call rail on the stream and the call id on the JSON block
* test(proxy): expect Anthropic events from a streamed messages guardrail block
The legacy guardrail suite pinned the old single bare data: line, which is the
malformed block this branch replaces. The test now checks the six Anthropic
stream events and the block text, so it fails if the bare line comes back
* test(proxy): block messages through a real pre-call guardrail in endpoint tests
The block tests replaced ProxyBaseLLMRequestProcessing.base_process_llm_request on the class to raise the block. They now send a real request through the endpoint with a pre-call guardrail that raises the passthrough block, and check that the x-litellm-call-id header carries the request's call id
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): update stale expectations to intended provider behavior
Each expectation moved because a merged PR changed behavior on purpose.
- chatgpt string input is wrapped as one user message (#41522)
- Gemini 2.5 tool media sits beside function_response (#48428)
- sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645)
- a body base_model is stripped without allow_client_pricing_override (#47639)
- ai_decide answers without usage return usage null (#48261)
- azure_ai forwards mapped client anthropic-beta values (#48033)
- the OpenCode scripted upstream mints a fresh id per request
* test(e2e-ui): type the guardrail name before deleting it
GuardrailDeleteModal requires the exact name before Delete enables since
#45893. All three guardrail delete flows now go through one helper.
* test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body
OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces,
so the gateway's traceresponse now ends in -03. The producer copies the server
span's own trace flags, so the tests now accept any two-digit flags with the
sampled bit set instead of pinning -01
#48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an
object" on purpose, so the [1] body case is renamed non_object_body and
expects that 400
* test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test
The outage test waited for "Exception while exporting Span batch" in the proxy
log, which is OpenTelemetry's wording and no longer appears with 1.45. The test
keeps asserting what the gateway guarantees, callers unaffected during the
outage, a live gateway, and every span after recovery delivered once with no
outage span duplicated
* test(guardrails): count the Responses disconnect spend rows #47611 added
Since #47611 a streamed /v1/responses disconnect writes its own spend row, so
the concurrent disconnect test saw 20 rows where it counted 10 by model group.
It now reads each row's call type and expects ten acompletion rows with the
outage verdict and ten aresponses rows with success. /v1/messages disconnects
still write no row on main, so they are left out of the count rather than
pinned as absent
The responses-httpx case of the single disconnect test no longer needs its
LIT-8603 skip and passes
* test(observability): size K4's timeout from the harness boot budget
K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to
58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of
30 runs hit pytest-timeout. The cap and the restart gate wait now come from
readiness_seconds and graceful_stop_seconds, like other multi-boot tests
* fix(proxy): abort the request with a 499 when the client disconnects during the body upload
Co-authored-by: Sinfony8838 <Sinfony8838@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): half-uploaded body on chat and vector store routes never reaches the provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): close the auth-time OTel server span before aborting a half-uploaded body
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Sinfony8838 <Sinfony8838@users.noreply.github.com>
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
68db022d57: `AssertionError: (('success',),)`).
The test now reads the spend row first, which is written once the proxy has settled the disconnect, and
releases the upstream after it. The proxy's behavior is unchanged; only the test's ordering is.
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* test(integration): stop GET /__observations from draining the upstream log
GET /__observations used to empty the queue, so a second read in a test
saw nothing and a "never reached upstream" check could pass whether or not
the request was forwarded. GET now returns every request since the last
reset and DELETE /__observations is the reset. An autouse fixture resets
the worker upstream before each test, which also covers tests that read
without the gateway fixture.
Discarded reads used as resets become DELETEs, helpers that accumulated
drained reads now read one snapshot, and tests with several phases reset
between them. The three vacuous negative checks (rag allowlist, guardrail
opt-out chaos, xAI batch output) now assert the absence on the same read
that shows a request which did reach the upstream.
* style(integration): space the observations unit test like its neighbours
* test(integration): give each owned proxy a private database and Redis by default
owned_proxy and owned_proxy_process now boot on a scratch database and a
per-proxy redis-server unless the caller opts into the worker's state with
state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips
DATABASE_URL. Under the replica profile the private database gets the
writer and reader roles so the reader path stays covered.
Gateway carries the database_url its proxy writes to, and the Scenario
cleanup checks and delete_key_if_present read from it, so they keep
checking something on a private database.
The 189 test files with owned proxy sites that depend on the worker's
database or Redis opt in with a shares_worker_state module marker, which a
module-scoped conftest fixture turns into the shared state. The 7 helper
modules with such sites opt in per call, since conftest fixtures and other
test files reach them.
* test(integration): reach session fixtures with the shared marker and keep the worker retry default
The module-scoped fixture that applied shares_worker_state ran after
session-scoped fixtures, so proxies those booted went private in marked
files. The marker is now applied around each marked item's setup, call
and teardown, which covers fixtures of every scope.
Owned proxies on the worker database inherited the worker gateway's
router_settings.num_retries of 0 from it. A private proxy now gets the same
default in its config copy unless the config sets num_retries itself.
Five providers files read the owned proxy's spend rows or response cache
through the worker's database or Redis and opt into sharing.
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit d3fafc3da2.
* test(integration): cover chat_completions endpoint contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): pin per-call prompts and both n=2 chat choices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover mixed path-model auth, string guardrail rejection and typed text-completion forms
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): drop the prompt-spellings cache workaround and cover Azure SDK text completions on the deployments alias
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): give chaos and timing tests a harness API and gate raw signals
Chaos tests killed and suspended proxy workers with raw os.kill and psutil
calls and bounded elapsed time with fixed literals, so they raced the uvicorn
worker healthcheck and failed on slow CI boxes
OwnedProxy now offers kill_worker and replacing_worker, which return only once
a replacement answers readiness with its own pid, and suspended, which is
refused unless the proxy was launched with a lenient worker healthcheck.
timing.py bounds durations against a measured control or the injected delay.
run.py measures the proxy import time once and derives readiness deadlines and
the per-test timeout from it, writing boot-calibration.json
check_test_quality.py adds TQ010 (raw process signals) and TQ011 (elapsed time
asserted under a numeric literal) for tests/integration outside _support, and
every call site is migrated so both report zero with no suppressions
* test(integration): steer the assistants wire burst onto both workers
The mixed-status burst assumed the OS would spread 48 connections across both workers and failed when one worker took them all. Each half is now sent while the other worker is suspended, and the test asserts an exact 24 and 24 split
* test(integration): restore product-enforced timing bounds lost in the migration
Some TQ011 migrations dropped or loosened a bound whose time is the behaviour
under test. The websocket session cap only kept its floor, so a cap closing at
300 s would have passed, and several timeouts were bounded by the injected stall
instead of the configured product value
assert_enforced_within bounds an elapsed time by the configured value plus
explicit slack, with an optional floor. The websocket cap, the MCP validation
timeout, the guardrail litellm_params timeout, the prisma command timeouts, the
baseline projection lock timeout and the key metadata statement timeout use it.
The device code refusal keeps its 15 s margin, and the langfuse retried prompt
fetch is now bounded by a cold fetch of a steady prompt as well as by the
ignored retry-after
* fix(tests): wait for the supervisor to reap killed workers and fix real-kill migrations
The lenient worker healthcheck was the whole case budget, so uvicorn's supervisor sat
in a ping on the killed worker for up to that long before respawning it. It now uses
the graceful stop budget, and a worker kill returns only after the victim is reaped,
so pooled connections no longer race the dead worker's socket.
Several tests on main picked the multiprocessing resource tracker as their victim and
never killed a worker. Now that they kill a real one, presidio counts streams cut by
the death, delete_model waits for the config reload that removes a deleted model from
a replacement worker, and the wedged coordination Redis test kills before the wedge
and waits for every parked publish to fail before resuming Redis.
The redis cluster test imports the harness through the tests package so it collects
in the unit job, and the per-model fallback rig exposes the owned proxy it now needs.
* test(observability): only require S3 deliveries the killed worker could not have buffered
A real SIGKILL drops the victim's unflushed batch, so requests it already answered can't be expected to land. The test now requires every request served after the replacement to land, some of the surviving worker's burst share to land, and nothing unsent to land
---------
Co-authored-by: Yuneng Jiang <yuneng@berri.ai>
* feat(guardrails): split logging_only_scope direction from input-failure policy
logging_only_scope is now direction-only (input, output, unset for both
directions). The new logging_only_continue_on_input_failure flag controls
whether a flagged or raising logging_only input scan still lets the
response be scanned. "both" stays accepted everywhere as a deprecated
alias for unset scope plus the flag set to true, normalized at callback
init while stored rows keep the written value.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): narrow continue-on-input-failure update type and stay under guardrail_info max-lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): keep custom-code guardrail controls readable with logging_only
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): scope continue-on-input-failure to the default direction and restore verdict assertion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): remove the both value from logging_only_scope
logging_only_scope only shipped its both value in dev prereleases, so it is
dropped instead of kept as a deprecated alias. Leaving the scope unset and
setting logging_only_continue_on_input_failure gives the same behavior. A
stored both value is now treated like any other invalid scope: it is logged
and ignored on load, and rejected with 422 on create or update.
* fix(guardrails): tolerate an invalid logging_only_scope in the v1 guardrail list
A config.yaml guardrail whose logging_only_scope is no longer valid, such as
the removed both value, made GET /guardrails/list return 500 for every
guardrail. The v1 list now parses params with the same tolerant parser as
the v2 list, info and PATCH paths, so the bad value is logged and dropped.
* test(guardrails): drive continue-flag rejection tests through the real guardrail handler
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* feat(policy-engine): stream detect-only post_call pipeline steps live when buffering is off
* fix(policy-engine): scan live pipeline streams a provider error cuts short and discard rewrites per step
A live detect-only pipeline now scans the chunks the client received when the provider fails mid-stream, then re-raises the error. Each step's rewrite is discarded right after the step, so later steps scan the text the client actually received.
* fix(proxy): keep a guardrail verdict recorded after a stream failure on the failure spend row
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(guardrails): add logging_only_scope to observe one direction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(guardrails): remove unrelated test churn
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): allow native lifecycle logging-only scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate api types for logging_only_scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): remove callbacks when scope validation fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep output-only scans when request copy fails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): configure logging_only_scope on guardrails
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep guardrails enforcing when logging_only_scope is invalid at load
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): avoid inline guardrail test fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): support directional scope in PATCH and provider UI
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): format guardrail files for frontend lint
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): reset unsupported directional scope selections
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): normalize logging-only scope and sanitize warning logs
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): normalize scope in shared guardrail field
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(api): sync guardrail schema artifacts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid stored logging-only scopes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add logging_only_scope integration audit cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): fix scope test lint issues
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): extend K4 chaos test timeout
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): restore stored guardrail row verbatim on rejected patch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): satisfy collection lint budgets
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): validate masked params through TypeAdapter
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid stored params on reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep constructor coercion in tolerant params parser
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): edit logging_only_scope in the custom code guardrail modal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(ui): extract custom code logging-only scope control
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep no-op PUT and logging_only scan-failure semantics stable
Three regressions from the logging_only_scope feature, fixed while keeping
the input/output/both selection working:
1. PUT with byte-identical litellm_params no longer forces a teardown +
re-init of the live callback. reject_invalid_logging_only_scope now
only validates; re-init still happens exactly when params/name change.
Previously a description-only PUT re-appended the callback at the END
of litellm.callbacks, reordering guardrails: with a BLOCK guardrail
created before a MASK one, the mask started winning and blocked
requests started succeeding (400 -> PUT 200 -> 200). Invalid unchanged
scopes are still rejected with 422 without touching the live instance.
2. CustomGuardrail._scan_logged_call no longer swallows input-scan
exceptions per branch: a raising or BLOCKED input scan aborts the
logging_only hook again (one policy call, one verdict) for guardrails
that never selected a logging_only_scope. Explicit input/output/both
selection keeps selecting which scans run.
3. A failed PATCH now rolls the DB row back to exactly the stored raw
litellm_params (a legacy 4-key row stays 4 keys) instead of expanding
it to a full LitellmParams dump; pinned with a test. This matches the
PUT rollback shape and is the faithful rollback.
* refactor(guardrails): type directional-scope provider list as a tuple
* fix(lint): stay within the basedpyright budget
- drop a Final annotation assigned inside the validation loop
(reportGeneralTypeIssues over ceiling by one)
- rename the PR-introduced _configured_event_hooks to public
configured_event_hooks; its cross-module import added the two
reportPrivateUsage errors that pushed the rule over its ceiling
* fix(guardrails): keep the response verdict for explicit logging_only_scope=both on input-scan failure
Greptile P1: an explicitly configured both-direction observer asked for a
verdict on each direction, so a failed request scan must not silently drop
the response verdict. The implicit default (logging_only_scope None) keeps
the abort semantics of a logging_only hook whose scan raised, which is the
base behavior the earlier fix restored.
Also drops two test comments that restated their assertions (Greptile P2).
* test(guardrails): align logging_only_scope integration rows with abort-on-input-failure and encrypted params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep the response scan when the request copy fails for explicit both scope
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): run end-of-stream post_call scan when the client disconnects mid-stream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): close the guardrail stream chain in async_data_generator on client disconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): leave the raw upstream response to the shielded finalizer on client disconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep disconnect cleanup going when a streaming callback cleanup raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): assert the refund through a recorder instead of the mock
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): inspect tool calls released before a disconnect under incremental_diff and record a failed scan marker
The incremental_diff transform stream now scans tool calls it already released when the client disconnects, and a disconnect scan whose translation raises after the guardrail recorded success also records guardrail_failed_to_respond
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): pin that a text-only disconnect scan is not handed a tool_calls finish
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover disconnect scans on every streaming endpoint and client, plus outage, worker-kill and cache-hit cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): prove the cache-hit twin is served from the cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): type the disconnect-close streams so basedpyright stops reporting unknown arguments
* fix(guardrails): give the guardrail metadata cast a reason so the type discipline gate accepts it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan released Messages and Responses tool calls on disconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(guardrails): format the disconnect scan unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(guardrails): drop mutable-ok markers that no longer suppress a rule
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan released Responses output after a finished item and end only in-flight Chat choices
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): type the disconnect scan test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): type request_data in the disconnect scan helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): type request_data in the disconnect scan test doubles
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): pin that chat streams with no tool call in flight end as released
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): scan only released chunks on disconnect and skip it once a block owns the verdict
The disconnect scan now uses the chunks actually yielded to the client, copies them before scanning,
skips when a mid-stream block or HTTP error already settled the verdict, and the iterator wrapper
only closes hooks that are async generators so plain async iterator hooks keep working
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): pin that a delivered guardrail error or final chunk settles the disconnect verdict
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): close any hook iterator that exposes aclose when the stream ends early
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): accept a synchronous aclose on custom streaming hook iterators
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): swallow callback aclose errors at end of stream
A custom callback whose async_post_call_streaming_iterator_hook returns a
non-generator async iterator with a raising aclose() failed the finished
stream: content plus usage reached the client and then the stream surfaced
an error SSE with no [DONE], or aborted a post_call pipeline's buffering
loop into a 500 with an empty body. Wrap the aclose invocation in
_wrap_streaming_iterator_with_enrichment in try/except and log a warning
naming the callback and the cleanup error, matching close_guarded_stream
and _close_guarded_layers. Iteration-time hook exceptions still propagate.
* fix(proxy): log only the error type when a callback aclose raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): scan and mask top-level instructions with guardrails
The Responses guardrail translation handler put a non-empty top-level instructions field into structured_messages as a system row but never into the flat texts list, so guardrails that scan texts skipped it, flat-text masking could not rewrite it, and PANW latest-only selection failed its alignment guard whenever instructions were present.
Seed texts with the instructions row, carry that offset into the flat-text write-back so a rewritten row lands on data["instructions"], and account for the leading row in the PANW Responses alignment.
Resolves LIT-8931
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): reject empty guardrail rewrites instead of forwarding raw input
An explicit texts=[] answer from a guardrail now fails the count check and
raises UnappliableRequestRewrite like any other misaligned rewrite; only a
missing texts key means no rewrite. Types the out-param as dict[str, object]
and adds integration coverage for instructions blocking, masking, empty
instructions, tool loops, latest-only and concurrent workers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): type the texts-replacing guardrail helper explicitly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): honor skip_system_message_in_guardrail for instructions and system input items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): cover skip_system_message_in_guardrail on the live proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep skipped rows through full-coverage rewrites and align latest-only with skip_system
Trust a guardrail's structured_messages_cover_full_request claim only when it
returns as many rows as the full normalized request, otherwise merge the scoped
rows back so skipped instructions and system items survive the write-back.
Make PANW's Responses reasoning alignment skip-aware so latest-only still picks
the latest user turn when system content is excluded from texts.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): annotate new guardrail tests with return types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): treat an empty guardrail texts answer as no rewrite like chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): type the guardrail test doubles explicitly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): apply experimental_use_latest_role_message_only to every request shape
Explicit true/false now applies to chat completions, Anthropic /v1/messages and /v1/responses alike; unset keeps latest-only for Anthropic and full history otherwise. Text indices are mapped back to their source message by value instead of by count, so Responses instructions, function_call_output and reasoning items no longer derail the alignment and silently rescan the whole history
Co-authored-by: scthornton <scthornton@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(panw_prisma_airs): type latest-message helpers against AllMessageValues
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): require forward and reverse text attribution to agree
A Responses function_call_output whose text equals the latest user turn could claim that turn's slot in a forward-only walk and demote the latest-only scan to an earlier message. Walk both directions and fall back to the full role-filter scan when they disagree
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): pick the latest human turn from messages, not from aligned texts
An image-only latest user turn no longer promotes an earlier user turn into the
latest-only scan; it scans nothing on the request side, as the Anthropic path did
before. A latest user/developer message whose text never reached texts (a trailing
Responses reasoning item) falls back to the role-filter scan instead of narrowing.
Types the test helpers, drops the narrating docstrings and adds regressions for both
shapes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): log when latest-only selection leaves nothing to scan
An image-only latest user turn with experimental_use_latest_role_message_only=true intentionally yields zero scanner calls. Emit a debug line naming the call_id so operators can tell this apart from the guardrail not firing.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(panw_prisma_airs): keep Responses reasoning items out of latest-turn selection
The Responses translation handler gives reasoning input items the default user
role, so a reasoning item with text content after the latest prompt was picked
as the latest human turn and the real prompt went unscanned under
experimental_use_latest_role_message_only. Map reasoning items back to their
texts positions from the raw input and exclude them; fall back to the
role-filter scan when the raw items do not account for every text
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: scthornton <scthornton@gmail.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): stream guardrail pre-call block as SSE with a typed output item
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): import blocked usage helper from the guardrail utils module
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): drop narrating docstrings and poll without rebinding
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cover pre-call guardrail block on /v1/responses stream and json
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for responses guardrail block contract
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): observe upstream on the recorded chat route for responses denial cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tidy responses denial audit cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): wait for worker count to recover after SIGKILL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): require a replacement worker after SIGKILL
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): type the blocked response test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng <yucheng@berri.ai>
* test(integration): streamed Bedrock Messages usage cost equals the recorded spend (Pylon #6667)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): echoed cost-map model info is not persisted as deployment overrides (Pylon #6844)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): reset sweep runs on one pod per tick while replicas share the lease (Pylon #6521)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bedrock post-call guardrail scans streamed Anthropic Messages tool use without 500 (Pylon #6503)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): realtime cached audio tokens bill at the audio cache-read rate (Pylon #6704)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): legacy GET /spend/logs returns at most the 10000 most recent rows and flags truncation (Pylon #6752)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): itemize Responses API cache write tokens as cache creation cost (Pylon #6454)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): migration entrypoint deploys pending migrations before proxy startup (Pylon #6649)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): opted-in team keys stop at the owner's personal budget (Pylon #6641)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): guardrail information stays in the spend log when the caller sends metadata on /v1/messages (Pylon #6614)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): key model allowlist is enforced on Bedrock passthrough routes (Pylon #6419)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): JWT mapped key backfills a null user email from token claims (Pylon #6266)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): scheduled budget reset recovers from a transient DB transport failure (Pylon #6582)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): team key lists models granted through a team access group (Pylon #6044)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): JWT subject without team claim lands in the configured default team (Pylon #5895)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): end-user spend lands for a key without user_id when the auth cache is Redis (Pylon #6021)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): stale-low redis counter still blocks team member over budget (Pylon #5824)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): prompt-carrying spend rows are written in byte-bounded statements (Pylon #6083)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): logs UI session_total_spend sums every round of a multi-round session (Pylon #5928)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): config.yaml guardrails are served by the guardrail usage detail and overview (Pylon #5813)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): plain chat request skips the object permission lookup (Pylon #5965)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): register july accounting regression contracts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): isolate cost map override clear on owned proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): make reset lease claim and db relay refusal deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): poll pg_stat settle, bound unbanned relay refusals, clear reset lease on teardown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bound relay refusals so the budget sweep can reconnect
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): saving echoed model_info never persists cost map pricing as a deployment override (Pylon #6870)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): budget_duration change on /budget/update recomputes budget_reset_at (Pylon #6913)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): count_tokens on a budgeted key reserves no budget and a later completion still succeeds (Pylon #6966)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): /cost/estimate reports configured prices for a deployment absent from the cost map (Pylon #7014)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): cache the team member default budget in Redis as JSON (Pylon #7180)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): aggregated team daily activity reports whole-range team spend in one page (Pylon #7224)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): failed daily user rollup commits are retried so spend report and daily activity agree (Pylon #7268)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): CLI session token without org_id is charged to and capped by the team organization budget (Pylon #7291)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): gemini passthrough success releases its budget reservation from the spend counter (Pylon #7295)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): batch retrieval spend row sums reasoning tokens and counts output and error file failures (Pylon #7341)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): uncostable batches retire from the cost poll page so newer batches are costed (Pylon #7342)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): charge a team member added without any budget on its membership row (Pylon #7363)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): failed dispatched requests keep estimated input tokens in spend logs (Pylon #7519)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bedrock passthrough converse guardrail ignores tool definitions (Pylon #7524)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): explicit null budget_duration on /team/new is not replaced by default_team_params (Pylon #7536)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): PATCH /organization/update with a null limit clears it (Pylon #7577)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): ultrafast service_tier bills ultrafast rates without leaking pricing fields upstream (Pylon #7587)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): keep the selected model in the response and spend log for an Azure Model Router alias (Pylon #7636)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): disconnected Bedrock /v1/messages stream still bills terminal usage (Pylon #7685)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): databricks cached prompt tokens bill at cache rates (Pylon #7738)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): completed batch spend row records reasoning tokens and error file failures (Pylon #7928)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): bill OCR annotation pages at annotation_cost_per_page (Pylon #7958)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): in-flight count tokens request reserves no key budget so a completion still reaches the provider (Pylon #7307)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): fail-closed key rejects known estimate over remaining budget before provider (Pylon #7691)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): streamed /v1/responses success callbacks keep provider response headers (Pylon #7775)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "test(integration): fail-closed key rejects known estimate over remaining budget before provider (Pylon #7691)"
This reverts commit 910348be7a.
* test(integration): reconcile contracts manifest for bundled regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): isolate proxy config writes in bundled regression tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): address review feedback on budget reset bounds and callback batch accumulation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): avoid rebinding the cache identity accumulator
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): assert forwarded messages per cache identity call
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): make budget reset and team default tests deterministic
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Adds integration contracts for MCP lifecycle, protocol errors and OAuth
configuration, A2A wire versions, the OpenAI consumer path, persisted
toolsets, callback delivery, guardrail effects, configured prices, the
filtered spend ledger, and a CircleCI-owned browser flow for project
detachment, with the ASGI, browser-state, client and MCP helpers they use.
Consolidates the eleven commits previously stacked on
litellm_integration_providers onto its rebased tip
