mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-30 06:43:27 +00:00
64 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mrinal-berri
|
07a82a287b
|
fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276)
* fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text The pass-through guardrail translation json-dumps the request body as the scan text when no request_fields are configured. The unified guardrail hook injects litellm_metadata into that body first, so the dump carried the caller's key metadata. Exclude it alongside metadata and litellm_logging_obj; vendors still receive it through request_data. * test(guardrails): type the pass-through scan-text tests per AGENTS.md |
||
|
devin-ai-integration[bot]
|
1418c1990f
|
feat(guardrails): add native Reco guardrail provider (pre_call), fail closed by default (#48243)
* feat(guardrails): add native Reco guardrail provider (pre_call) Registers guardrail: reco as a first-class vendor integration, built on the existing generic guardrail API wire contract rather than reimplementing the HTTP client and payload mapping. Maps reco_tenant_id to the X-Reco-Tenant-Id header and hardcodes fail_open/fail_on_error since those should not be customer-configurable. v1 only supports mode: pre_call; post_call and streaming redaction already work against the existing generic pilot and are deferred. Also overrides apply_guardrail to re-raise blocked-request errors with the guardrail's own configured name: the base class hardcodes a module-level name in that exception, which would otherwise mislabel every blocked Reco request in logs. * feat(guardrails): configurable Reco fail modes Expose unreachable_fallback and fail_on_error on the Reco guardrail and keep fail-open as the default. Values under optional_params win. Top-level litellm_params values apply only when set explicitly, since their defaults are fail-closed and would otherwise flip the default Also type RecoGuardrail's forwarded kwargs with an Unpack'd TypedDict instead of silencing ANN003 Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Cursor <cursoragent@cursor.com> * chore(guardrails): align Reco with main's CI gates Upstream moved unit tests from tests/test_litellm to tests/unit, so test_reco.py was invoked by no CI job. Move it next to the other guardrail tests Replace RecoGuardrail's forwarded kwargs with explicit keyword params, read optional_params through a pydantic adapter instead of untyped dict/getattr access, freeze RecoOptionalParams, and drop mutable-ok markers that suppress nothing, to stay within main's type-discipline and basedpyright budgets Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Cursor <cursoragent@cursor.com> * feat(guardrails): fail closed by default in the Reco guardrail Matches GenericGuardrailAPI and every other guardrail here: a Reco error or outage now blocks the request unless the admin sets fail_on_error: false. Tests cover the default, explicit fail-open, and top-level vs optional_params precedence Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): honor explicit fail-open on Reco when optional_params is a model model_dump(exclude_unset=True) so pydantic defaults in optional_params no longer override an explicit top-level fail_on_error. Also drop the duplicate log_guardrail_information decorator, call the parent by keyword, freeze RecoConfigModel, and type the tests Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep Reco config model assignable and the hook decorator the gate requires frozen=True on RecoConfigModel froze the shared LitellmParams mixin, which the guardrail registry assigns resolved secrets on, so every guardrail failed to load. Dropped it and added a test that LitellmParams stays assignable. Restored log_guardrail_information on the Reco override since tests/code_coverage_tests/check_guardrail_apply_decorator.py requires it on every hook. Test params typed as Mapping for LIT001 Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(guardrails): format the Reco test file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Yaron Bratspiess <yaronb@reco.ai> Co-authored-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
464932b1d1
|
refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789)
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1) * refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2) * refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts * fix: restore pass-through behavior on paths the typing pass tightened MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites * fix(runwayml): keep a malformed 2xx task body on the 500 path * fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report EdenAI's embedding and image generation parse dropped a non-object body from the error they raised, so the message no longer named what the provider sent. A helper now raises with the body in the message, and both paths have a regression test. `lite encryption migrate` exited 0 with no message when the proxy's report was not an object or its residual_legacy was not a number, where base crashed. It now fails with a message naming the value. The CompactifAI and Gemini image-edit non-object body tests accept TypeError or AttributeError, which both base and this branch raise, so they guard against a ValidationError instead of pinning one class. RunwayML's async parse gets the same non-object body test as the sync one. * fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed * test: annotate the new regression tests' locals with Final * test: annotate the remaining new test locals with Final |
||
|
Mateo Wang
|
a8113b0dd7
|
fix(responses)!: round-trip Anthropic compaction blocks as compaction items (#46754)
* fix(responses): round-trip Anthropic compaction blocks as compaction items /v1/responses on Anthropic routes now returns Claude's compaction block as a compaction output item (and streams it as output_item.added/done), and a compaction input item is replayed to Claude as its native compaction block, so the next turn reuses the summary instead of compacting again. Other routes receive the summary as assistant text. The Anthropic stream handler now keeps the summary text that arrives in compaction_delta. Fixes #41456 * fix(responses): replay compaction blocks natively only when the request compacts * fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem objects on replay, strip the encrypted_content affinity wrap before decoding, scan and rewrite compaction summaries in Responses guardrails, drop the history a summary replay replaces, send the summary as a user message so thinking ordering holds, and send the streamed compaction .done with the newest block. * fix(responses): count, route, guard, and redact compaction items consistently A compaction item now reads as its summary everywhere the proxy counts or routes a Responses request: pre-call context checks, the TPM limiter, the input token endpoint, the stream usage estimate, and the previous_response_id session rebuild all drop the history the item replaces. A Claude compaction item sent to a non-Anthropic model (including a fallback) becomes a user summary message instead of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform takes the summary path. Output guardrails see the summary of a returned compaction item, and logging redaction hides it. * fix(responses): start previous_response_id replay at a stored compaction and count reservations from it * fix(responses): keep a stored compaction summary whole in spend logs * fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a client-built item no longer reach the database whole. Message redaction also nulls the streamed compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning item that follows it. Adds the audit's integration tests for compaction items on the wire. * test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order. * fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes A failed /v1/responses or /v1/messages call keeps its proxy request headers in litellm_metadata, but the failure row read redaction headers from metadata only, so x-litellm-enable-message-redaction never redacted those rows. * fix(spend_logs): lift only the redaction opt-in header into failure rows The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main * test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes * fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary A previous_response_id rebuild that starts at a compaction now keeps the system and developer messages of every earlier turn. Token estimates for raw Responses input count a compaction item as its summary text instead of the serialized item. * fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id * fix(responses): count only message items with a string role as compaction instructions * test(responses): flatten the last two-for comprehension in the compaction wire test * fix(responses): replay from a compaction in stored Responses output and SDK compaction items A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths. * fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history * fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound. * fix(responses): replay a compacted session only through the referenced response A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item * fix(spend-logs): keep main's stored value for a prompt length limit under 2 At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where main stores the marker followed by the whole value * fix(spend-logs): keep main's stored tail for a negative prompt length limit At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start * fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages * test(responses): run the compaction wire tests on the worker database under isolated workers * fix(router): keep the encrypted content affinity pin out of provider requests * fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction * fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message * fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses * fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call * test(proxy): run the dispatched-call-type failure test through the real request setup * fix(proxy): count a failed Cursor stream's input from the newest compaction item The Cursor route maps to no call type, so the stream generator's failure hook fell back to the route and counted the whole history. Pass the processor's route_type. * fix(proxy): count a broken Cursor stream's prompt from the newest compaction item * fix(responses): keep a tool call a later output answers when trimming before a compaction item * fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes * perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time * test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests * fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item * fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item * fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content * test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks * fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once * fix(responses): count input_tokens after a compaction item the way inference sends it |
||
|
alberto
|
f768fb32ef
|
fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
4b1928387f
|
test(bedrock): detect Vault reads on the event-loop thread (#47597) | ||
|
Mateo Wang
|
f3513bca9e
|
fix(bedrock): resolve AWS region and secret lookups off the event loop (#47331)
* fix(bedrock): resolve AWS region and secret lookups off the event loop Bedrock request paths that resolve their region, endpoint or bearer token inside the async handler now do it in the AWS signing executor, so a slow secret manager read no longer stalls every other request on the worker * fix(bedrock): type the STS client kwargs and claude platform environment override Keeps the basedpyright counts at or below the merge base after the off-loop wraps, by typing the STS client kwargs as a TypedDict, the claude platform environment override with concrete dicts, and the realtime region call through the typed getter * test(bedrock): detect event loop stalls without wall-clock timing The Vault fake now answers a read only after the test loop runs a callback, so a read made on the loop thread is caught deterministically instead of by a sleep and a heartbeat threshold |
||
|
Charles
|
5a1fa1c879
|
fix(responses): keep strong refs to stream and websocket logging tasks (GC race) (#31074)
* fix(responses): keep strong refs to stream and websocket logging tasks * test(responses): patch _spawn_logging_task instead of asyncio.create_task The success-logging dispatch in _log_completed_response now goes through _spawn_logging_task, so tests that patched asyncio.create_task no longer observed it (and the sync ones hit get_running_loop with no loop). * test(responses): assert stream and websocket logging tasks survive garbage collection Adds regression tests that force a collection while the logging task waits on an otherwise unreferenced future, and types the new test parameters * test(responses): drive drain and GC tests through a real logging object with gated callbacks * test(responses): use asyncio.wait_for so the logging tests run on Python 3.10 * test(guardrails): type the spawn_logging_task tracking wrapper --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Lai Wei
|
bda4e60f9b
|
fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions (#44349)
* fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt Read chunk fields as typed attributes instead of untyped subscripts, which clears the reportUnknownArgumentType breach the lint gate flagged * fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): rebuild dict-form text completion chunks again stream_chunk_builder routes dict chunks holding TextChoices to the text completion builder, which now reads attributes. Convert them to TextCompletionResponse at the routing point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage * fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild * fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions * fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null * fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`. The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text` next to `delta`, so streamed /v1/responses calls to Together ended without response.completed. Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about the same as rebuilding an n=1 stream of the same size. * fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals * fix(streaming): rebuild text completion choices without an index at index 0 Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base * fix(streaming): rebuild text completion choices with a string index under that int A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string * test(streaming): drop a docstring that repeats its test * fix(streaming): fall back to choice 0 for an infinite text completion index A float infinity index made int() raise OverflowError, which the index coercion did not catch, so the rebuild failed where the merge base returned choice 0. --------- Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
9c958aff34
|
fix(logging): deliver completed batch success callbacks consistently (#47192)
* fix(batches): dispatch configured success callbacks during reconciliation * fix(batches): retain billed job claim when sync dispatch shuts down * fix(batches): finish reconciliation when logging executor is stopped * fix(logging): avoid duplicate custom hooks during async batch dispatch * fix: preserve sync-only custom callback delivery * fix: scope callback deduplication to per-call async recipients * refactor: type per-call async success callbacks * refactor: retain logging constructor callback compatibility * test: support current batch reconciliation metadata * fix(logging): honor per-logger redaction during sync success delivery * fix(logging): centralize completed success callback dispatch * fix(guardrails): avoid duplicate success callback dispatch --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f6ae25939e
|
feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965)
* feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries * fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5 A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow * refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass * chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py |
||
|
yucheng-berri
|
223800f916
|
fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024)
* fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130) Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main with the unit test moved to tests/unit after the test tree relocation * fix(presidio): label an overlapping run by its outermost detection, add integration coverage A run of overlapping detections now takes its entity type from the highest-scoring detection that no other detection in the run contains, matching Presidio's own anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER. The resolver returns typed detections instead of Any. * fix(presidio): drop the redundant comment in the numbered token loop * fix(presidio): bump a numbered token another part of the request already holds Each message and content part numbers its tokens from 1 into one request-wide pii_tokens map. Collapsing overlapping detections made an overlapping value's token well formed, so it collided with the same type in another part and both restored as the later value. A token that maps to a different value now moves to the next free number, and a token for the same value is reused * fix(presidio): give a clashing token a spare number instead of scanning for one Scanning upward from the clashing number probed every token an earlier part had claimed, so two parts with thousands of values of one type took seconds of synchronous work. No token in the map is numbered above the map's size, so numbers above both that size and the part's run count are always free * test(presidio): make the anonymizer double reject detections the analyzer never returned Pick the token in one conditional expression instead of reassigning it, and have the scripted anonymizer answer 500 when the proxy forwards spans that differ from the analyzer's, so the default-scope cells check what was sent --------- Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
46ae68021e
|
fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096)
* fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events When the proxy renders a complete Anthropic Messages response as an SSE stream for a client that asked to stream, the message_delta event carried only stop_reason and stop_sequence. A real Anthropic stream puts safeguard_results and stop_details inside message_delta.delta, so a client such as Claude Code auto mode lost the server-side classifier verdict and a refusal lost its stop_details. Both stream builders now copy those keys from the source response through one shared helper when they are present. * fix(guardrails): keep safeguard_results and stop_details when re-emitting a rewritten Anthropic stream A post-call guardrail that rewrites an Anthropic stream (presidio, bedrock, tool permission, model armor) now carries the upstream message_delta safeguard_results and stop_details into the re-emitted stream. The router fallback message_start no longer leaks those final fields early * fix(guardrails): keep stop_details and safeguard_results in rewritten Anthropic streams without explanations A guardrail that rewrites a /v1/messages stream now carries stop_details (type and category) and the dangerous_tool_use verdicts for the tool calls it still delivers, with every explanation dropped so a masked stream never echoes the unmasked text through those fields. * fix(guardrails): copy only upstream stop_details and forward-compatible safeguard_results into rewritten Anthropic streams A rewritten stream now takes stop_details and safeguard_results from the upstream message_delta alone, so a refusal the upstream sent without stop_details is never given one by the OpenAI round trip. Unknown safeguard types, statuses, reasons and extra fields pass through, a verdict follows a tool call only when its input is unchanged, and tool_permission keeps the explanations since it rewrites no text. * fix(guardrails): apply the safeguard_results and stop_details rules to policy pipeline stream rewrites * fix(router): deliver a refused stream built from a finished agentic turn instead of falling back Websearch, MCP and advisor streams replay a complete reply whose tools already ran, so a refusal fallback would run every tool again. Also give the recovered response's message_start a null stop_details as the API sends it, drop safeguard_results a masking rewrite cannot read, and add the audit cells for synthesized streams. * test(router): annotate the finished-turn refusal stream test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
ef1cb19351
|
fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307)
* test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
b39eadcb82
|
feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966)
* feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * feat(openai): honor per-deployment workload identity on non-chat routes Image generation, audio speech and transcription, moderation, text completions, files, batches, and fine-tuning built their OpenAI client from the env vars alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file failed with the api_key error on every route other than chat, Responses, and embeddings. Those handlers now build their client through the shared workload identity resolver, and the managed batch file-access credential extraction keeps the trio * fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys Non-chat handlers built a fresh SDK client per call, so files, batches, fine-tuning, text completion and sync moderation exchanged the OIDC token on every request. The builders now reuse the cached workload identity client per identity, base, timeout, retries and organization; static-key clients are still built fresh. A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no longer silently overrides a deployment's openai_identity_* params. An explicit distinct api_key on the deployment or call still wins. * fix(openai): keep shared workload identity clients on the process http session The workload identity client is cached and shared across requests, so it must never borrow the transport a single caller passed in. Static-key clients keep using the caller's http client; federated clients always ride the process-wide session. Also makes the http client helpers on BaseOpenAILLM public since the builder now needs them from outside the class. * test(openai): pin partial deployment identity beating the process key * test(openai): type the managed-object invoke callback * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals The httpx-based OpenAI handlers built their Authorization header from the deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file sent "Bearer None". They now resolve the bearer through resolve_openai_bearer_token, which exchanges the deployment's identity token the same way the SDK-based routes do. A deployment that carries only part of the identity trio now raises a 500 naming the missing field instead of silently falling back to the process key, and the SDK client builders take a factory for the static-key http client so the federated branch never touches litellm.client_session. * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * feat(openai): resolve per-deployment workload identity for the assistants routes OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * feat(openai): resolve per-deployment workload identity for image variations Build the image variations client through the shared workload identity builders instead of scraping locals(), and give the async path the async http session it was missing. Let a federation trio serve the model-less batch fallback, and hold the standard LRU size of exchanged tokens. * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * style(tests): keep two blank lines before the batch_cost_is_final block * test(batches): type the workload identity fallback test fixtures * fix(moderation): stop validating unrelated kwargs on the sync workload identity path * fix(auth): refuse server-owned workload identity keys in array form and inside a client router config * fix(openai): keep a deployment's own key ahead of workload identity on every route Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation, image generation, and assistants so a deployment key equal to the process key still wins. A partial identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and /v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments * fix(containers): pass deployment params to container file auth headers Container file routes built their auth headers without the deployment's litellm_params, so a keyless deployment with an OpenAI workload identity sent Bearer None on upload, list, retrieve, delete and content calls * fix(openai): keep process-wide key precedence and move token exchange off the event loop Restore the merge base order where an explicit or process-wide OpenAI key wins over federation, so exporting OPENAI_API_KEY keeps working the way it did before. Run the header-building validate_environment calls (image edit, videos, containers, vector stores, vector store files) and the decisions and System One request preparation in a worker thread, so a slow token exchange no longer stalls the loop. Federate realtime websocket sessions, live sessions, realtime client secrets and transcription sessions, and the OpenAI moderation guardrail when no key is set. * fix(openai): answer 401 for a partial identity trio and federate realtime health checks A deployment with part of the workload identity trio and no static key now fails as an auth error naming the missing params, so the router does not retry it as a server error. The realtime health check sends the exchanged bearer for a keyless federated OpenAI deployment, and API token counting reads the same static keys as every other route before minting. * fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config * fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting * test(integration): refuse caller-chosen identity params in user_config and bracketed form keys * fix(openai): send the honored secret manager key from SDK client builders A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key on image, assistants, files, batches, and other SDK-built routes instead of failing with no api_key, matching what the workload identity resolver already treats as a static key * test(openai): check the token exchange runs off the event loop instead of timing the loop --------- Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
0dac61a555
|
fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338)
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open GuardrailToolParam required a type key, so Gemini native tools (function_declarations, googleSearch) failed validation inside the hook and either returned a 500 or skipped the scan when fail_on_error was false. The type key is now optional and left out of the forwarded tool when absent, so an echoing guardrail server cannot write type: null into the provider request. structured_messages was re-validated against the chat completion message types, which silently emptied tool_result, tool_use, image and document content blocks to content: [] and dropped undeclared keys before the guardrail saw them. The request model now forwards rows as opaque mappings, which also retires the write-back shim that restored the original rows for echoed-back entries. The fail-open critical log line now carries error_type so a timeout, a refused connection and an upstream HTTP status can be told apart without the traceback. * fix(httpx): retry aiohttp server disconnects the way httpx does aiohttp raises ServerDisconnectedError when a kept-alive connection is closed by the peer before a response arrives, which is what a load balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces. The transport mapped it to httpx.ReadError, which the handler's connection-error retry does not cover, so the request failed instead of being resent on a fresh connection. It now maps to httpx.RemoteProtocolError, the error httpx itself raises for the same condition, so the existing retry fires. * test(httpx): build the stub server reply in one bytes literal * fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable The connection-error retry closed its fresh client before a streamed body was read, so an aiohttp disconnect now billed a second upstream request and still failed. It also skipped masking on a status error from the resend, resent PATCH and DELETE as POST, dropped multipart files and ignored the handler's ssl_verify. The retry now resends the request the first attempt built, on a client with the handler's TLS settings, closes that client when a streamed body is closed, and masks a status error the same way the first attempt does. * fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type * fix(guardrails): resend a generic guardrail check cut before any response A load balancer that closes an idle keepalive connection made the guardrail POST fail with "Server disconnected", because #46336 only resends POST on ConnectError. AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to False, that gives a RemoteProtocolError the same single fresh-client resend PUT and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a guardrail check is a pure verdict over the request. Every other POST keeps the #46336 rule The fail-open log line now puts error_type at the end so the pre-PR text stays contiguous for existing alert matchers test_rows_reach_the_guardrail_verbatim now reads the body off a real local guardrail server instead of mock call args * fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration A ServerDisconnectedError raised while reading a response body now stays httpx.ReadError, as it was before this PR. Only a disconnect before any response maps to RemoteProtocolError, which is the case the resend rule is about. Stream readers that end on ReadError, like the SAP stream iterator, keep their old behavior The loopback-server disconnect, TLS and guardrail resend tests move to tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is in-process only. Unit keeps in-process MockTransport checks for the idempotent POST resend, masking on the resend, streamed and buffered client closing, the guardrail resend, and the guardrail receiving rows verbatim * fix(guardrails): resend a reset guardrail check and keep typed echoes on texts An idempotent POST now resends once on any network error after connect (ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load balancer that answers a stale keepalive with a TCP reset rides it out too. That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes back to ReadError. This stops PUT and DELETE from stacking litellm's resend on top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil Guard from replaying its analyze POST on a disconnect Every resend now runs inside the method's own error mapping, so a timeout on the resend is a litellm.Timeout like the first attempt and a status error is masked once by the caller A guardrail server that parses rows with its own message types and echoes them back without the keys those types lack (annotations, tool_calls[].index) is read as an echo again, so its texts redaction reaches the LLM. A row only counts as an echo when no value changed and every key the chat completion types declare is kept The wire tests for the PUT/DELETE resend, its masking and its TLS settings now run on the httpx transport, where litellm's resend is the only retry, and the POST and guardrail tests cover both transports and both cut styles * test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head * fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement * refactor(guardrails): move the dropped-connection guardrail resend out of this PR The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line * test(guardrails): pick the closed guardrail port from the integration port allocator Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port * feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
f281f77bc0
|
fix(guardrails): record every content filter match in logging_only and offer the mode in the UI (#46673)
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test * fix(guardrails): record every content filter match in logging_only and offer the mode in the UI * style(tests): format the content filter logging_only integration test * refactor(guardrails): keep the logging_only predicate and scan override within the type gates * fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning * test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test * test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects * refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach * test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter * test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up * test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
627bcab42b
|
fix(guardrails): honor streaming settings on the Noma v2 guardrail (#46606)
* fix(guardrails): honor streaming settings on the Noma v2 guardrail The Noma v2 initializer now passes streaming_buffer_until_moderated, streaming_end_of_stream_only, streaming_sampling_rate and streaming_buffer_release_on_scan from the guardrail config to NomaV2Guardrail, so the unified streaming hook applies them. The constructor validates them, so a custom guardrail class built on NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults, and an invalid value is ignored with a warning naming the guardrail, so the guardrail still loads. Resolves LIT-9633 * test(guardrails): cover Noma v2 streaming settings across clients, failures and workers Adds integration cells for the custom-subclass loader, string and invalid values, unset and false keys, a Noma outage with and without block_failures, an upstream rejection, n=2 choices, pre_call and non-streaming guardrails, the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a PATCH while traffic flows, a burst while Noma is down, a slow Noma, and replaced workers. Resolves LIT-9633 * test(guardrails): pin the PATCH probes to one worker at a time Each probe suspends the other worker and opens a fresh connection, so the test checks that every worker applies the patched setting. Resolves LIT-9633 * test(guardrails): check sampled scans by count, not window position /v1/messages hands the streaming hook raw network reads, so where the default sampling windows fall depends on timing; the base leg caught one shifted window. Check the scan count and the final full-text scan instead. Resolves LIT-9633 --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
3151c47960
|
fix(guardrails): resolve decision models in caller team scope (#46085)
* fix(guardrails): resolve decision and judge models in caller team scope * fix(guardrails): scope guardrail model calls to the caller's team only where it owns the deployment Read the team only from the auth object the proxy attached to the request, never from caller JSON, and use it only when the router's route to the guardrail model for that team reaches a deployment a teamless caller cannot. Shared names, aliases, legacy team tags and wildcards keep resolving and billing as before. Decision calls pass the team in litellm_metadata only when it is used, so the logged model group stays intact. Adds integration cells for chat, responses and messages, forged team ids, a team deployment that shares a name with a shared model or wildcard, and billing, plus unit cases for legacy team tags and aliases. * fix(guardrails): route team guardrail calls to the team deployment without charging the team Guardrail calls on a team's own model now go to the deployment the router picks for that team, by its id, with no team on the call. Like every other guardrail call, they write no spend row and add nothing to the team's or the key's spend * fix(guardrails): pin the team deployment only for the judge guardrail, not shadow eval judge calls * fix(guardrails): keep the teamless route when only a team wildcard covers the guardrail model * refactor(guardrails): scope the team model fix to the decision model guardrail only * fix(guardrails): send decision calls on a team's own model with the team, and keep team wildcards out The decision call on a team's own model now carries the caller's team instead of being pinned to a deployment id, so it is charged to that team and goes through the team's normal routing: an access window that reserves the deployment for the team lets the call through, and pre-call checks see the input and skip a deployment too small for it. A team deployment listed under a wildcard public name, such as team_public_model_name openai/*, no longer captures a guardrail model the team never named, whatever model it points at. Adds unit and integration cells for the reserved deployment, mixed context windows under enable_pre_call_checks, a team wildcard name over a fixed model, and the owner team's spend. * fix(guardrails): log the owner team's name on decision calls charged to it The decision call on a team's own model now carries the team alias beside the team id, so the spend row and the Logs page name the team the call was charged to. * fix(decision_model): use the caller's team only for names nothing else serves A team model named like a shared deployment, a shared wildcard or an SDK public name took over the guardrail's decision calls for that team, so a team admin could point a proxy-wide guardrail at their own endpoint. The team is now used only when a caller without a team reaches no model under the name. * refactor(decision_model): drop the team route subset check the served check covers Once nothing serves the name without a team, every deployment a caller without a team can reach is a wildcard, so the subset check only ever rejected an empty team route. Pin team wildcards over a name nothing else serves. * fix(decision_model): use the caller's team only when judge_target finds nothing without a team The served check rebuilt the router's rules and missed a shared wildcard whose provider comes from custom_llm_provider, so a team model under the guardrail's name took over that team's decision calls. Ask judge_target, the same lookup the teamless dispatch makes, instead --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
52fc37888b
|
fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns (#46444)
* fix(responses bridge): keep defer_loading on function tools sent to the Responses API Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): give the defer_loading casts a cast-ok reason Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): build defer_loading outside the tool loop so Final stays valid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): round-trip tool_search_call and tool_search_output items between turns Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(responses bridge): read replayed tool-search and namespace fields through typed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep tool_search items on a turn with no message or tool call Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep every reasoning item on a search-only turn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): leave content unset on a search-only turn so replay adds no empty message Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): replay tool search items only with tool_search, count replayed tools in the allowlist, keep namespace off guardrails * fix(responses bridge): honor drop_params for defer_loading and route tool_search to Responses Under drop_params, defer_loading is dropped when no tool_search tool is sent, so the 400 it would cause upstream goes away as before. A non-bool value answers 400 without drop_params and is dropped with it. A null tool-level value falls back to the nested function value. A request carrying a tool_search tool now bridges to Responses on OpenAI and Azure endpoints even with reasoning_effort none, since chat completions rejects the tool_search type. * fix(responses bridge): replay tool search items in output order and count replayed tools in tool policy Store the ordered output prefix through the last tool search item, reasoning included, so a second reasoning item no longer makes OpenAI reject the replay. The shared chat-to-Responses converter now replays tool search items and namespace only when the bridge opts in, so Noma, the complexity router and guardrail payloads keep their shape. The tool policy guardrail and the allowlist count replayed tools only when the request can send them. * fix(responses bridge): forward defer_loading only beside a tool_search tool defer_loading only means something with tool_search, and backends without it reject the key (xAI answers 403). A true value with no tool_search tool now answers 400 naming drop_params and is dropped under drop_params; a false value there is a no-op and is dropped. * fix(tool_loop): keep tool search items and namespace between rounds * fix(responses bridge): check replayed tool search loads when a deployment adds tool_search * fix(chat): drop litellm response fields before strict OpenAI-compatible providers A chat reply from the Responses bridge carries provider_specific_fields on the message and on each tool call, plus reasoning_items. Groq, Fireworks, Mistral and Together reject unknown fields, so sending that reply back to them as history failed with a 400 or 422. Strip them in the shared helper and use it in those providers. * fix(caching): keep reasoning_items and tool search items on response cache hits A cached bridge response lost reasoning_items on a non-streaming hit and lost both reasoning_items and provider_specific_fields (tool_search_items) on a streamed hit, so the next turn could not replay the search or the reasoning around it * test(integration): cover tool search item replay on the chat to Responses bridge Proxy, SDK, authorization and chaos cells for returning and replaying tool_search_call and tool_search_output items and the tool call namespace * test(integration): add response cache twins for the tool search guardrail cells Each guardrail cell now has a twin where the answer is already in the Redis response cache: a key opted out of global guardrails caches a replay that loads a policy-blocked tool, the same replay without guardrails is cached before block mode sees it, rewrite mode is not served the unrewritten conversation's cached answer, and an allowed replay is served from the cache. * fix(responses bridge): forward defer_loading beside a client-executed tool_search The bridge now replays a client tool_search_call and the client's tool_search_output through provider_specific_fields.tool_search_items, so a deferred function beside a client tool_search keeps defer_loading like it does beside a hosted one * test(responses bridge): pin the client tool_search turn and its replay, streamed and not * fix(responses bridge): strip nested namespace for strict providers and keep streamed reasoning_items on text replies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses bridge): keep dropping defer_loading beside a client-executed tool_search Forwarding defer_loading beside a client tool_search made GPT stop with a search call the chat client has to answer by hand, so a plain question that used to get tool calls got none. Keep main's behavior of dropping it there. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
airia-open-source
|
220bbfc0ce
|
feat(guardrails): add Airia guardrail integration (#40784)
* feat(guardrails): add Airia guardrail integration Adds `airia` as a built-in guardrail provider. The hook sends the prompt under `pre_call` and the model's response under `post_call` to the Airia AI Gateway, which answers allow, block, or redacted content; the only configuration is `api_base`, `api_key`, and an optional `timeout`. `during_call` is not offered: it runs concurrently with the model call, so a block could land after the prompt has already reached the provider. A BLOCKED verdict, or any action this version does not recognise, raises with `blocked_content=True`; a transport error, a non-2xx, or a response body that is not a JSON object raises with `blocked_content=False`, so callers can tell "could not evaluate" from "evaluated and blocked" while both still fail closed. On GUARDRAIL_INTERVENED a copy of the inputs is returned with every rewritten field substituted; an intervention carrying no applicable rewrite, a rewrite of the wrong shape, or a rewrite whose item count does not match what was sent (which would misalign the positional write-back downstream) blocks instead of letting the original through. Streamed responses are moderated whole and then emitted redacted: the hook opts into the unified hook's `incremental_diff` mode (the default `block_only` drops rewrites) with `streaming_end_of_stream_only`, so a redaction can never span transform rounds and underflow. Covered by a test that drives the real streaming iterator hook end to end and checks the assembled client output, not just the guardrail's own config attributes. The HTTP client is dependency-injected (an `async_handler` constructor parameter, defaulting to a real client) rather than mutated post-construction, so tests inject a fake client instead of monkeypatching an instance method. Known limitation, not fixable from this guardrail alone: LiteLLM's unified guardrail framework yields a tool-call chunk to the client immediately during `incremental_diff` streaming, inspecting the assembled tool calls against the guardrail's verdict only once the stream ends (`_run_incremental_transform_stream` in `unified_guardrail.py`). A blocked tool call can therefore reach an executing client before the terminal error arrives. Switching to `block_only` with `streaming_buffer_until_moderated` does not close this without a regression: that flag has no effect under `incremental_diff` (the mode branches away before it is read), and `block_only` itself replays the original, unredacted chunks on release, silently dropping any REDACT verdict on streamed text. Relatedly, LiteLLM's OpenAI chat completions guardrail translation skips `post_call` guardrails entirely for a response with no text content, so a tool-call-only response bypasses this guardrail (and every other post-call guardrail) even outside streaming. Both are shared-framework behavior, not specific to Airia; fixing either is out of scope for an isolated single-guardrail contribution. * fix(guardrails): fail closed on an Airia rewrite for a field that was not sent Bugbot finding on #40784. An INTERVENED verdict carrying `structured_messages: []` with `texts` omitted passed `_rewritten` unchanged: an empty list has the same length as an omitted field, so it counted as an applied rewrite while the `texts` that actually carried the content went through untouched. post_call sends only `texts`, so this was reachable. A rewrite for a field the request did not send is now a mismatch, same as a wrong length, and the verdict blocks. Regression test covers both the empty and the non-empty case. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(guardrails): fail closed on streamed non-chat routes the Airia rewrite cannot reach Veria finding on #40784. incremental_diff, the mode that carries a rewrite into the stream, exists only for OpenAI chat completions; UnifiedLLMGuardrails sends every other streamed route (/v1/responses, /v1/messages) to block_only, which streams the original text live and drops rewrites. So with post_call on, a streamed Responses or Anthropic call could receive text that Airia then blocked or redacted. Two changes, both confined to this guardrail: - streaming_buffer_until_moderated is set, so on the block_only fallback the framework holds every chunk until the end-of-stream verdict instead of releasing it live. The flag is never read on the incremental_diff path, so streamed chat completions are unchanged. - A REDACT verdict on a streamed response whose route cannot carry a rewrite is enforced as a block, since block_only would otherwise replay the original. The route check mirrors _resolve_transform_call_type so the two cannot disagree; an unresolvable route fails closed. Request-side rewrites and non-streamed calls are untouched. The end-to-end test drives the real UnifiedLLMGuardrails iterator over an Anthropic /v1/messages SSE stream and asserts not one original chunk is released for either verdict. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(guardrails): keep the Airia stream test's pytest.raises block to one statement CI's test-tree ruff run flags PT012 on the non-chat streaming test: the `async for` that collected released chunks sat inside `pytest.raises`, so a raise from the loop body would be indistinguishable from the one the test is about. The loop now lives in a `_drain_into` helper and the `with` block awaits it alone. Same assertion, same partial-leak capture. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * refactor(guardrails): keep the Airia guardrail within the per-rule basedpyright ceilings scripts/type_check_gate.py now gates every rule on its merge-base count (#40193), and the guardrail added sixteen strict-rule errors it previously absorbed under the global budget. - The parsed gateway response goes through `_as_object` once, so `action`, `blocked_reason` and the rewrite fields are read from `dict[str, object]` rather than `dict[Unknown, Unknown]`; `blocked_reason` is only used as the message when it is a non-empty string. - `get_config_model` returns the concrete `type[AiriaGuardrailConfigModel]`, and the config model carries the same pyright ignore as its siblings for the unparameterised base. - The remaining untyped upstream members (the decorator, the client factory, `super().__init__`'s kwargs, the handler's `post`) are ignored per line with the reason, as the Wingback guardrail does. The registry reads `guardrail["guardrail_name"]`, a required TypedDict key. - Tests reach the injected mock and the handler timeout through typed helpers instead of the real handler's partially typed attributes. No behaviour change: 46/46 tests, ruff default and strict, and both lint gates pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(guardrails): clear the Airia guardrail's type-discipline gate violations Drop the mutable-ok comments that suppress nothing (LIT013), give the two deferred-import noqa lines a reason (LIT003), and annotate the parsed JSON object as Mapping instead of dict (LIT001). --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Tobias Mayr
|
d03923e6a9
|
fix(responses): propagate cache_control through responses->chat completion bridge (#32511)
* fix(responses): propagate cache_control through responses->chat completion bridge User/assistant messages, tool call outputs, and function calls built from Responses API input items were dropping cache_control, breaking prompt caching for Bedrock/Anthropic when routed through the litellm_completion transform. Also propagate cache_control onto the last content element of user/assistant messages, since Anthropic/Bedrock read it from the content block rather than (or in addition to) the message level. Adds the missing cache_control field to ChatCompletionToolMessage and ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe. * chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control Adding cache_control to the ChatCompletionToolMessage TypedDict changed the proxy's OpenAPI spec; regenerated via `npm run gen:api`. * fix(responses): don't overwrite existing content-block cache_control; add list-content test Per Greptile review on the PR: - The message-level cache_control was unconditionally overwriting the last content block's cache_control when that block already carried its own (preserved by _transform_responses_api_content_to_chat_completion_content). Now only propagates onto the last block when it doesn't already have one. - Added the missing test coverage for the content-block propagation path with list content, which no existing test exercised (all used plain string content, so the isinstance(content, list) branch was never hit). * style(responses): extract cache_control into a named variable, trim comment Per style review: the message-building branch read cache_control 3 different ways in 5 lines (.get() twice, then [] indexing once), unlike the sibling tool-call-output/function-call functions this PR touches, which each extract it into a named variable up front (output_cache_control, function_call_cache_control). Matched that convention here too. Also trimmed the 3-line inline comment to one line, matching the file's usual comment length for similarly subtle behavior. * chore: refresh lazy OpenAPI snapshot * fix: type cache control propagation * fix: place cache control type suppressions * style: format cache control type conversion * refactor(responses): build bridged cache_control messages in one shot * fix(responses): keep cache_control marks on bridged tool calls and within the provider cap A function_call item's mark now rides on the tool call the bridge builds, an assistant text item merged behind a function_call keeps its own mark, a developer item keeps its mark when it is mapped to a system message, and a previous_response_id replay yields its stale marks to the newest ones so the provider cap holds. The Responses layer counts the marks on function_call and function_call_output items before it places an ordinal injection point, and the chat pass replaces that count with its own instead of adding to it. * fix(responses): count each cache_control mark once and keep the bridge within the provider's limits List content carries an item-level cache_control on its last block only, so the provider sees one cache point and the breakpoint census counts one. Marked function tools and SDK-typed input items are handled by the Responses-layer reserve, assistant text merged into a function call turn keeps its mark on its own block, and the previous_response_id replay budget drops marks per turn, newest kept, instead of stripping every older turn once one overshoots. The request's own marks now fall under the same budget, newest kept, so five marked items no longer answer the provider's 400. A cache_control value that is not an ephemeral mark with an optional ttl is left off the bridged message instead of reaching the provider as sent, and a developer message maps to the system role with its content, name, and cache_control only, so unknown keys never reach a strict provider. * fix(responses): send item cache_control only to cache-point targets and follow drop_params Item marks reach the provider only when the target turns them into cache points, and every other target keeps main's behavior of stripping them. Without drop_params a mark the provider would refuse reaches it as sent, so the client gets the same 400 as on /v1/chat/completions. With drop_params a malformed mark is dropped and only the newest 4 are kept. The shared input-to-messages helper and the developer mapping in base_utils go back to main, and a marked developer item reaches a cache-point target as a system message in the bridge. * fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers * fix(responses): validate replayed marks and count input tokens without item marks * fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks * test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets * fix(responses): keep item marks off inspection paths and match each chat route's keep rule Only the request the bridge sends gets item marks now. Guardrails, token counting, rate limiting, and input_tokens read the input with no item marks, as on main, so an off-spec mark cannot fail a guardrail call. Tool message and tool call marks are typed inside the bridge, leaving the shared chat types and guardrail schemas as on main. Databricks keeps marks only for Claude, Snowflake uses its chat route's Claude check, and every Anthropic deployment keeps them whatever its model name. The injection census skips SDK-typed tools, and the owned-proxy test expects the marked messages on /v1/responses. * test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census * fix(responses): leave a non-list input untouched when stripping item marks * test(responses): cover string, list, object and tuple input with and without item marks kept Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter * test(cache): derive owned-proxy cell timeouts from the harness boot budget The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
493ed616bb
|
fix(config): treat blank env values as unset so the proxy neither crash-loops nor silently drops a guardrail (#46341)
* fix(config): treat blank numeric env values as unset instead of crashing at import
A numeric knob exported as an empty string, which is what a Helm value: "" or an
emptied .env line produces, raised ValueError inside litellm/constants.py and the
proxy crash-looped before serving traffic. Every int(os.getenv(...)) and
float(os.getenv(...)) in constants.py now goes through get_env_int / get_env_float,
which treat blank as unset and fall back to the default with a warning on garbage.
* fix(guardrails): treat blank litellm_params values as unset so the guardrail still loads
A guardrail whose litellm_params read os.environ/X resolves X to "" when the
variable is exported but empty. Pydantic rejected "" for bool and typed fields
such as default_on and fail_on_error, the boot log reported the guardrail as
skipped and the proxy came up without it. A before-validator on BaseLitellmParams
now drops blank string values so each field takes its default, and the
default_on shortcut in LitellmParams treats a blank value as False.
* fix(config): warn once about an unparsable numeric env value so logging cannot recurse
The LiteLLM loggers' filter reads MAX_STRING_LENGTH_STDOUT_LOG and MAX_BASE64_LENGTH_STDOUT_LOG through get_env_int on every record. Warning about a garbage value from inside that read re-entered the filter, so every log call raised RecursionError. The value is now marked as reported before the warning is logged, which also stops per-record readers from repeating it. get_env_int_or_none and the new get_env_float_or_none share the same parser
* fix(config): treat blank values as unset for the remaining import-time numeric knobs
LITELLM_MAX_STREAMING_DURATION_SECONDS, REALTIME_WEBSOCKET_MAX_MESSAGE_SIZE_BYTES, a whitespace CLI_JWT_EXPIRATION_HOURS, LITELLM_CONFIG_PARAM_CACHE_TTL_SECONDS, the agent health check timeouts and the Prisma reconnect knobs still crashed the proxy at boot when blank. A blank or unparsable REQUEST_TIMEOUT no longer counts as explicitly set, so chat completion() keeps its 600s fallback
* fix(guardrails): map blank litellm_params to the field default so PATCH can still clear a value
Dropping blank keys before validation made PATCH /guardrails/{id} with api_base "" a silent no-op. Blank values of optional fields now resolve to the field default and stay in the set fields, so the PATCH clears the stored value. Required fields and unknown keys keep the value as given
* test: cover proxy numeric knobs in-process and expect no warning for a blank Straiker api_version
The three proxy import-time numeric knobs move into litellm/constants.py so the existing in-process blank-env test covers them, replacing the subprocess test. A blank api_version now means unset, so the Straiker test asserts the key-prefix version with no warning.
* fix(guardrails): keep LitellmParams kwargs typed without adding argument-type errors
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
|
||
|
Mateo Wang
|
0ebec123fd
|
chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy modules with one pydantic validation at the point the payload enters the module, so every value read afterwards carries a real type. reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright, ruff-strict or type-discipline rule is higher repo-wide or in any single file, and no cast, ignore or suppression marker was added. * chore(typing): replace 47 Any annotations that every caller already satisfies with object Each annotation here is a parameter or return type whose value is only stored, compared, logged or forwarded, so object describes it exactly and no function body changes. A site was kept only if no basedpyright, ruff-strict or type-discipline rule rose in any file that references it. reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655. * chore(typing): type upstream responses and loose annotations across providers, proxy and integrations Second pass of the Any reduction. Decoded upstream JSON is validated once at the provider boundary and read through typed views afterwards, annotations that every caller already satisfies move from Any to a concrete type, and the nine search transformations validate each result item on its own so error text no longer carries the item index. No cast, no type: ignore, no suppression marker and no budget file edit * ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard |
||
|
yuneng-jiang
|
1587faeeb9
|
test(unit): wait on the work itself instead of fixed sleeps (#46261)
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files Replace sleeps that raced background logging, cache writes, refreshes and thread work with either awaiting that work directly (logging worker flush, the batch logger's flush_queue, the tasks the call spawned, thread joins) or an Event the test's own fake sets when it runs. Each wait keeps a 5s hang guard that never decides pass or fail. Tests that need a fake clock, real sockets, or measure latency are left unchanged, and no production code moves * test(unit): await the tasks each completion spawns instead of yielding once The logging tests yielded once with asyncio.sleep(0) so the task that enqueues the success log could run before the worker flush. That relied on the enqueue happening before the task's first await. Snapshot the running tasks around each acompletion call and await the ones it spawned, so the guarantee holds however many steps sit before the enqueue |
||
|
yucheng-berri
|
c3ca8a5140
|
feat(guardrails): add Wingback guardrail integration (#46136)
* feat(guardrails): add Wingback guardrail integration
Mirrored from #38685 at
|
||
|
moyai-devin-berriai[bot]
|
0c04d0bdb7
|
fix(guardrails): honor decision model streaming buffering (#46086)
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
02b5a395d8
|
feat(zscaler_ai_guard): send the request user to AI Guard (#46094)
* Add user field to AI Guard execute-policy payload for dashboard attribution Send the top-level OpenAI request `user` field in the execute-policy JSON payload so AI Guard can surface usernames on its dashboard. This is independent of the existing user_api_key_* headers which remain unchanged. The upstream guardrail framework already passes the original request_data (containing the `user` field) through to response-path guardrail calls, so this works for both IN and OUT directions including streaming. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Apply suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Add type annotations and tests for user field in execute-policy payload - Add Final annotation to user_for_payload - Add str | None type to user parameter in make_zscaler_ai_guard_api_call - Add 3 tests: user included when present, omitted when absent, and user in/out of JSON body sent to AI Guard Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test(zscaler_ai_guard): inject the http client in user field tests Pass an AsyncHTTPHandler into ZscalerAIGuard so the user field tests read the real execute-policy body off a MockTransport instead of patching class methods, and drop the comments that restated the lookup * fix(zscaler_ai_guard): forward only non-empty string users Read the request user through a typed helper and type the execute-policy call params so basedpyright stays under the merge-base count, and skip non-string values the vendor rejects with a 500 * test(zscaler_ai_guard): drop the test-only http_client Run the user field tests against an in-process aiohttp server through api_base so the guardrail keeps its original constructor --------- Co-authored-by: Syamala Keerthana Chegu <schegu@zscaler.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1dbde0845e
|
test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)
* test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
888686d858
|
test(headroom): pin that SDK-echoed null tool_calls never reach /v1/compress (#39667)
* fix(headroom): drop null tool_calls before calling /v1/compress
headroom-ai 0.27.0 through 0.30.0 iterate msg.get("tool_calls", []) and fail
with 'NoneType' object is not iterable when an assistant row carries an explicit
tool_calls: null, which the fail-closed guardrail then surfaces as a 502. Strip
the null key from rows sent to the compression service; real tool_calls lists
are forwarded unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(headroom): pin that SDK-echoed null tool_calls never reach /v1/compress
Main now withholds every assistant row from /v1/compress (#39974), so the
null tool_calls strip added here never runs on a row headroom-ai 0.27-0.30
fails on. Drop the strip and pin the customer's history shape through the
public OpenAI chat translation handler instead.
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
|
||
|
yucheng-berri
|
c1e3d07e76
|
fix(guardrails): scan or refuse attachments in the bedrock guardrail (#43383)
* fix(guardrails): scan or refuse attachments in the bedrock guardrail * fix(guardrails): cover native pre_call and computer screenshots in the bedrock attachment scan * fix(guardrails): classify every attachment shape the bedrock guardrail can receive * test(guardrails): type the attachment-scanning test guardrail * fix(guardrails): keep base behavior for Bedrock subclasses and malformed messages - skip the attachment scan when a subclass overrides apply_guardrail - ignore non-string message role/type instead of raising - treat a null Converse document/video/audio block as absent - keep the base request-body debug log for text-only calls * fix(guardrails): let text documents, unpadded base64 images and request-override subclasses through as before * fix(guardrails): let Converse text documents through and refuse documents that carry images * fix(guardrails): scan images inside content documents instead of refusing them * fix(guardrails): scan text document content with the Bedrock guardrail * fix(guardrails): scan document title, context and nested documents * Expand nested content documents without recursion * Keep attachment scan within the type-discipline budget * Take the image format and pixel size from the image bytes * chore(guardrails): satisfy type-discipline gate after merging main * chore(guardrails): drop docstrings added by the attachment scan * fix(guardrails): scan Converse text documents and guardContent images A Converse document whose source is text (or a list of text parts) now goes through the Bedrock text scan instead of being refused as unscannable, so the pass-through request that worked before this PR keeps working. A Converse guardContent image is now scanned like any other image. Byte and S3 document sources stay unscannable. Adds the integration test that drives every scanned endpoint through a real proxy against a scripted Bedrock ApplyGuardrail stub, and retypes the touched hook signatures for the type gates. * test(guardrails): cover Bedrock checks mode in the attachment integration test Adds a checks-mode proxy to the rig: images are refused without calling Bedrock on every route, and a Converse text document is sent to InvokeGuardrailChecks. * fix(guardrails): scan a Converse document's name and context whatever its source A Converse PDF or S3 document let through by skip_unscannable_attachments now still has its name and context scanned as text, the same as the Anthropic document path. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
5f9dd38867
|
feat(guardrails): add decision model guardrail with Admin UI flow (#45663)
* feat(guardrails): add decision_model guardrail for decisions-API models Adds a new guardrail provider that scores request and response text with a decisions-API model (e.g. Jev) against named predicate checks. Each check carries its own threshold and action (block or log); a flagged block check rejects the request with a 400 naming the flagged checks, and call failures honor unreachable_fallback (fail_open or fail_closed). Ships four preset checks for the Admin UI (prompt injection, jailbreak, system prompt extraction, data exfiltration) served from /guardrails/ui/add_guardrail_settings, plus a Decision Model step in the add-guardrail form filtered to evaluation-mode models. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): fail closed on verdict errors and drop comments in decision_model * feat(guardrails): cap decision_model input with head and tail elision * fix(guardrails): list only block checks in decision_model 400s and tighten inputs * feat(guardrails): honor timeout, cap checks at MAX_DECISION_QUESTIONS, and reject duplicate check names * fix(guardrails): decorate decision_model apply_guardrail with log_guardrail_information * fix(guardrails): suppress untyped-decorator pyright error on log_guardrail_information import * feat(guardrails): screen each text in overlapping chunks with a shared decisions-call semaphore Replace head/tail elision with per-text screening: each non-empty text is sent to the decision model on its own, split into overlapping chunks when it exceeds max_input_chars, and all calls run under a per-event-loop semaphore shared across requests (max_concurrent_decision_calls, default 8). Check verdicts aggregate the max probability across calls; a missing predicate answer on a Block check blocks under fail_closed and only warns under fail_open, while log-only checks are never flagged for it. The 400 detail now lists flagged check names only. The provider combobox also matches Decision Model on jev, typesafe, prompt injection, jailbreak, and decision * fix(ui): align decision model custom check rows with preset rows * fix(ui): move decision model custom check remove button to the row end * style(ui): run prettier on decision model check files * fix(ui): inline decision model evaluation-mode filter after lens module removal * fix(guardrails): keep decision model detections when a chunk call fails * fix(ui): require an enabled decision model check before create * fix(guardrails): screen decision model texts sequentially like presidio * feat(guardrails): replace decision model presets with provider picker and per-question test * fix(ui): render decision model question test as a full-width row * feat(ui): add run-history test section to decision model form * fix(ui): prevent threshold overlap and show test shortcut * feat(guardrails): default decision model threshold to 0.7 * test(ui): expect decision model default threshold 0.7 in characterization payload * feat(guardrails): screen tool call arguments with the decision model * refactor(guardrails): freeze the decision model config model * feat(guardrails): list only evaluation-mode deployments in the decision model picker * feat(guardrails): hide the decision model questions table until a question exists * style(ui): format DecisionModelFields * style(ui): format decisionModelQuestion tests * refactor(guardrails): type the tool call function extraction * refactor(guardrails): build tool call texts without a mutable accumulator * refactor(ui): share isDecisionMode for the decision model picker * fix(guardrails): screen prior Responses function call arguments in the decision model guardrail The Responses input handler leaves function_call arguments out of texts and tool_calls, so a prompt injection in a prior call's arguments passed unscreened. Assistant tool calls from structured messages are now screened too, deduplicated against the request's tool calls * test(integration): send instructions with the decision model guardrail check Checks require instructions, so creating the guardrail returned 422 and the test never sent a chat request * fix(ui): score the decision model test preview the way the guardrail does An unanswered block question now shows Block, disabled questions no longer count, and a run is hidden once the model or a question's text changes since it was scored * fix(ui): clear the decision model error on pick and list questions under Add a question Picking a model after a failed Create left "Select a decision model" on screen, because the form only validates on submit. The field now revalidates when it already shows an error. The Questions table and its heading appear only once a question exists, below the Add a question box * test(ui): move the decision model form tests into an integration test file They render the whole AddGuardrailForm, so they belong in a .integration.test.tsx file. Plain fields are now filled with fireEvent.change, and the two existing form test files go back to main apart from the modelHubCall mock the form now needs * test(guardrails): build the decision model test params without mutating a local * fix(guardrails): reject list checks outside decision_model and honor streaming scan settings - LitellmParams now rejects a list `checks` for any guardrail other than decision_model, so a Bedrock guardrail with list checks gets 422 again instead of being stored and failing every request - The decision model guardrail reads streaming_end_of_stream_only and streaming_sampling_rate, so a streamed post_call response can be scored once at end of stream instead of every 5 chunks - max_concurrent_decision_calls is described as a per-worker limit * fix(ui): match guardrail providers like the default picker filter - The provider search uses the Base UI collator filter on the label and the aliases, so searches such as presidio_pii or llm_as_a_judge match again, as they did before the alias filter - Model groups from /model_group/info are parsed defensively and after the providers are populated, so a malformed response cannot empty the provider list * fix(ui): edit decision model questions in place and show passing test chips in green Questions start as an Add question button. Each click adds a blank question card with its name, text, action and threshold, editable until the guardrail is created, mirroring the per-model budget editor. Create and Next reject a blank or duplicate enabled question. Test run verdicts and chips use the success tone for Pass, the error tone for Block and the warning tone for a flagged Log only question * fix(ui): drop the per-question enable checkbox from the decision model form A question is either in the list or removed with its X, so every listed question is sent and scored. The card now labels Name and Question the way it labels Action and Threshold * fix(ui): give the question name field the full card width * fix(ui): show the provider logo on the selected decision provider * fix(ui): show the full threshold slider track on decision model questions * feat(ui): add decision model cards to the guardrail garden Adds a Decision Model section to the guardrail garden with TypeSafe Jev, Perplexity, Microsoft, OpenAI and Databricks cards. Each card opens the create form scoped to that provider's decision models. The create form drops the Decision Provider picker and lists every decision model on the proxy with its provider logo, prompts to add a model when none is available, and says that test runs are billed and logged. * fix(ui): shorten the decision model test hint to one line * fix(ui): prefill the decision model from the garden card's provider * chore(ui): regenerate the dashboard API types after merging main --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
nate-berri
|
7f5abfffb4
|
refactor(anthropic): move passthrough SSE stream assembly into litellm/llms (#45744)
Part of LIT-9423 |
||
|
devin-ai-integration[bot]
|
8a103e26ab
|
test: move routing, caching and callback tests out of local_testing (#45761)
* test: move routing, caching and callback tests out of local_testing Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion * test: cover least-busy routing and tighten migrated router, cache and pass-through tests Add router-level least-busy tests for the three legacy rows that had no destination, move completion fallback and model alias tests to their mirror files, move router batch tests next to the router, and strengthen tests whose mutants survived (specific deployment, batch pair grouping, custom cache key, router cache ttl). Add a unit test for configured pass-through Authorization forwarding. * test: tighten migrated callback, proxy config and cache ttl tests Assert the full StandardLoggingPayload key set and the wire user on callback tests, add an async embedding failure callback test, drive the prometheus redis failure through a real RedisCache, use the real salt key in proxy config tests, pin the redis ttl value and rename fallback test helpers * test: fix semantic cache assertions and order-dependent router and callback tests Count semantic cache embeddings on the route, drain async cache writes before the second lookup, echo the requested model from the callback provider mock so response_cost does not depend on model_cost entries other tests register, and move the two-key authorization assertion to the auth failure test it belongs to * test: cover model_list fan-out and the usage-based routing tpm limit lost in the move * test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard * test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file * ci: run the remaining router-keyword legacy tests on one node * test: refuse the redis write in-process instead of resolving an invalid host --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
yucheng-berri
|
7ff5010674
|
feat(guardrails): split logging_only_scope direction from input-failure policy (#45709)
* feat(guardrails): split logging_only_scope direction from input-failure policy logging_only_scope is now direction-only (input, output, unset for both directions). The new logging_only_continue_on_input_failure flag controls whether a flagged or raising logging_only input scan still lets the response be scanned. "both" stays accepted everywhere as a deprecated alias for unset scope plus the flag set to true, normalized at callback init while stored rows keep the written value. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dashboard): narrow continue-on-input-failure update type and stay under guardrail_info max-lines Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dashboard): keep custom-code guardrail controls readable with logging_only Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dashboard): scope continue-on-input-failure to the default direction and restore verdict assertion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(guardrails): remove the both value from logging_only_scope logging_only_scope only shipped its both value in dev prereleases, so it is dropped instead of kept as a deprecated alias. Leaving the scope unset and setting logging_only_continue_on_input_failure gives the same behavior. A stored both value is now treated like any other invalid scope: it is logged and ignored on load, and rejected with 422 on create or update. * fix(guardrails): tolerate an invalid logging_only_scope in the v1 guardrail list A config.yaml guardrail whose logging_only_scope is no longer valid, such as the removed both value, made GET /guardrails/list return 500 for every guardrail. The v1 list now parses params with the same tolerant parser as the v2 list, info and PATCH paths, so the bad value is logged and dropped. * test(guardrails): drive continue-flag rejection tests through the real guardrail handler --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Arjun Aravind
|
96ac1ec282
|
fix(guardrails): avoid blocking the event loop during HiddenLayer JWT refresh (#45437)
* fix(guardrails): avoid blocking the event loop during HiddenLayer JWT refresh * test(guardrails): mock HiddenLayer detection requests with respx |
||
|
devin-ai-integration[bot]
|
13928bdabe
|
fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream (#45433)
* fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream * test(websearch_interception): type the conversion helpers and drive the replay test through mock_response * test(websearch_interception): pin the replay test's values as Final and record follow-ups immutably * fix(proxy): strip every agentic-loop control key from client bodies, the web search keys included * fix(interception): move stream_options off forced non-stream calls for code interpreter and headroom, and keep adapters from replaying nested calls * test: annotate new interception test locals with Final * fix(responses): drop params a deployment hook removed on sync Responses calls * test: drive the sync Responses stream_options regression through a mock transport and real Azure config * test: type the agentic loop recorder and capture the sent request on an object * test(integration): cover interception stream_options on chat, responses, messages and the SDK * test(integration): assert the follow-up error text and release a held burst on failure * test(integration): pin the follow-up failure contract and wait for the replacement worker before teardown --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
b1d0e45f8d
|
feat(policy-engine): stream detect-only post_call pipeline steps live when buffering is off (#45516)
* feat(policy-engine): stream detect-only post_call pipeline steps live when buffering is off * fix(policy-engine): scan live pipeline streams a provider error cuts short and discard rewrites per step A live detect-only pipeline now scans the chunks the client received when the provider fails mid-stream, then re-raises the error. Each step's rewrite is discarded right after the step, so later steps scan the text the client actually received. * fix(proxy): keep a guardrail verdict recorded after a stream failure on the failure spend row --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
5e1c5c0bd1
|
test: move offline proxy auth, hook, spend and pass-through tests to tests/unit (#45568)
* test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit Relocate 53 legacy tests that pass with no network or keys into the tests/unit files that mirror the code they exercise, drop 2 llm_guard tests already covered by test_llm_guard_call_type_aliases, and delete the emptied legacy files. * test: freeze the reset clock and run the real reset methods in the moved budget reset tests Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await the service-hook tasks the job spawns instead of sleeping. Drive the per-user failure with a malformed budget_duration row from the prisma mock so the real _reset_budget_for_key/user/team run instead of patched replacements. * test: use fixed timestamps in the moved pass-through and spend log tests * test: mock the moderation HTTP call and type the moved moderation and spend log tests Serve the OpenAI moderation response through an httpx MockTransport so the real async_make_request runs, annotate the moved tests' locals with Final, and pass get_logging_payload typed kwargs instead of an untyped input_args dict. * test: drive the moved proxy tests through boundaries and annotate their locals with Final Serve GCS downloads and the Router moderation call through httpx mocks with fake Google credentials, let the real pass-through success handler fail on the mocked upstream response, and fold the Vertex live route endpoint check into the existing route test. Annotate moved locals with Final, build the budget reset rows without rebinding, and type the remaining fakes without **kwargs |
||
|
yucheng-berri
|
51fea825cc
|
feat(guardrails): support logging_only mode for the Akto guardrail (#45461)
* feat(guardrails): support logging_only mode for the Akto guardrail * test(guardrails): wait for the spend row before reading Akto calls and cover mixed modes * test(guardrails): cover Akto logging_only on MCP tool calls * test(guardrails): type the Akto logging_only unit tests and inject the HTTP handler * test(guardrails): cover failing Akto replies, provider failures and a mixed outage burst under logging_only * test(guardrails): cover an unreachable Akto with fail_open under logging_only |
||
|
devin-ai-integration[bot]
|
85a3869dfd
|
feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes (#43801)
* feat(guardrails): run each mode only on streaming, non-streaming, or both
Add stream_scope so a rail can target streaming inference, non-streaming inference, or both per pre, during, and post mode
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor stream_scope in pipelines and dashboard types
Pipeline steps skipped the stream_scope filter, direct construction ignored mixed-case maps, and schema.d.ts was stale.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): skip unmatched stream_scope steps instead of allowing
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): keep stored stream_scope keys for modes not on screen
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): format stream_scope helpers and update fork MCP unit tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): keep hang cancellation tests from timing out during setup
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor path-defined streaming for stream_scope
Passthrough routes like Gemini streamGenerateContent decide streaming from the URL, so stamp that onto hook data before guardrails run.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): copy AnthropicModelCapabilities instead of cloning
Clippy treats clone-on-Copy as an error, which failed rust-lint on the messages request tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): trust only server stream classification
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): keep streaming marker through deepcopy
scan_raw_request snapshots copy each field, so a plain object() marker would lose identity and skip streaming-only rails.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cost-map): drop duplicate perceptron-mk1.5 row
Two main cost-map PRs both added the OpenRouter model, so the merge left a second key that CI rejects.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): expect native transcription 429 as RustUpstreamError
HTTP status errors from native routes map through route_error_to_pyerr, so the wheel SIGINT child was dying on an outdated RuntimeError check and never reached the hang probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): follow Google Interactions OpenAPI without hardcoded names
The live spec dropped CreateModelInteractionParams and renamed the item path to {interactionsId}. Misc CI failed because the compliance tests still looked those names up as literals.
* test(guardrails): reproduce stream_scope bedrock and passthrough field gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover invalid stored stream_scope reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify bedrock stream actions and keep caller is_streaming_request
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): make stream_scope_allows public and drop mutable builds
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify pass-through and Bedrock stream scopes
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden stream classification and validation
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add stream scope integration audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid mutating passthrough custom body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): run stream scope audit without enterprise license
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): keep stored scope restart cell on one worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): scope stream scope audit sink assertions to the rail under test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert logging_only scope absence behind an ordered barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert one logging_only scan per phase after the barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): set request scope on pass-through stream fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid YAML stream_scope in v1 guardrails list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: merge main into litellm_guardrail_stream_scope_fixes
Update pass-through pre-call test callbacks for main's endpoint_type argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add request paths to pass-through fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover websocket pass-through stream scope and LIT-9050 outage spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): remove unused type discipline suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): script the vertex live upstream in the websocket stream scope test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep stream marker json-serializable and restore pass-through helper names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): allow required Bedrock action re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): strip the stream marker from pass-through payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): drop stream marker by value in scans and snapshots, plain-tuple stream scope state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): render tag-scoped guardrail modes read-only in the custom code editor
A guardrail whose litellm_params.mode is the tag-scoped dict {tags, default}
crashed the Custom Code editor on open: normalizeMode wrapped the dict into
the mode array and StreamScopeFields rendered it as a React child (error #31,
whole dashboard unmounted). Treat a non-string non-array mode as no editable
modes, show formatGuardrailMode(mode) in a disabled input (read-only, matching
the guardrail info view), and keep mode/stream_scope out of the update payload
for such guardrails.
* fix(ui): resolve merge fallout in guardrails components
Deduplicate toModeArray import after the merge, and move the read-only
guardrail details block back into GuardrailReadOnlyDetails (now rendering
the shared mode/logging-only rows plus the stream-scope detail) so
guardrail_info.tsx stays under the 800-line lint budget. Guardrails UI
suite: 298 passed.
* fix(ui): drop duplicate toModeArray import reintroduced by merge
* chore(pass-through): document the deliberate in-place marker strip as mutable-ok
The clear/update on _parsed_body is load-bearing: rebinding to a fresh
mapping instead breaks 75 pass-through tests because the marker-free body
must propagate through the caller's request dict so downstream guardrail
scans and snapshots never observe the server streaming marker.
* fix(guardrails): move stream_scope after timeout in CustomGuardrail init
Inserting stream_scope before the existing timeout parameter shifted the
positional slot of timeout, so positional callers constructed with their
timeout bound to stream_scope (ValueError) and timeout silently None.
Restores the base parameter order; keyword callers are unaffected.
* fix(guardrails): typing pass for the lint gates
stream_scope leaves the declared constructor parameters (restoring the
base positional surface; keyword construction unchanged), unknown config
values crossing the new stream-scope code paths get typed locals or
cast-ok boundaries, and the passthrough payload literals are annotated.
All three lint gates pass against current main; the guardrail suites are
unchanged (236+5019 passing; one known anyio-driver failure pre-existing
on base).
* style: sort cast imports for the ruff gate
* chore(pass-through): drop unused BEDROCK_STREAMING_ACTIONS re-export
The streaming check now uses is_bedrock_streaming_endpoint; nothing in
the repo imports the name from this module.
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: gabriele <gabriele@berri.ai>
|
||
|
devin-ai-integration[bot]
|
d5173e3d70
|
test: move 76 live top-level tests to offline unit and integration coverage (#45352)
* test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
0e26edfdb9
|
test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288)
* test: make legacy live tests in spend, batches, openai endpoints and audio dirs offline (partial)
* test: migrate wave-1b live tests offline (guardrails, images, ocr, search, openai endpoints)
* test: fix wave-1b review items, add responses/ocr integration tests and firecrawl unit test
* test: anthropic messages router/bedrock/openai-bridge unit tests for wave-1b nodes
* test: anthropic messages logging, prompt-caching and tool-search unit tests; drop migrated base nodes
* test: finish pass_through_unit_tests nodes, logging drain fix and mutations
* test: migrate anthropic passthrough tests to integration wire tests
* test: fix passthrough migration wire spend row lookup and wildcard config
* test: migrate hosted vllm and openai file passthrough tests offline
* test: move assemblyai and vertex passthrough nodes to in-process unit tests
* test: restore unlisted router node and fix logging worker drain in passthrough unit tests
* test: drop spend-row BUG skip and sharpen non-streaming skip reason for anthropic messages
* test: use public presidio alias after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: restore the batch and file unit tests the migration rewrote away
* test: restore full legacy intent in anthropic messages router unit tests
Drop the false BUG skip on non-streaming aanthropic_messages logging (success
callbacks do fire; the skipped body filtered on the wrong model), assert the
logged model_group, messages, cost and usage, cover streaming logging for both
Anthropic and Bedrock invoke, assert dict content blocks for Anthropic, Bedrock
invoke and the OpenAI bridge, add the Bedrock invoke leg of the router test,
fall back from a real 401, test system-prompt caching and streaming
message_start cache fields on converse and invoke, send the legacy tool-search
tools and beta header, remove the type: ignore and bare dict helper, and add
in-process native /anthropic passthrough spend logging tests
* test: fix passthrough migration wire tests and drop false native spend BUG skip
The native spend row test read rows by the shared master-key digest, so it
matched other tests' rows; read each row by its own message id instead and
assert tokens, total, spend, tags, provider, api_base and end_user on both the
non-streaming and streaming native routes. Merge the streaming test that never
checked spend, assert exact tags from litellm_metadata, stop rebinding a Final
in a loop, require cost > 0, and add the chat-completions bridge cost case the
legacy test covered
* test: harden the spend, OCR, image, search and guardrail migration replacements
The OCR spend tests now build fresh kwargs per case instead of mutating a
shared fixture, and every payload case asserts the exact logged spend. The
OCR wire test reads its spend row by request id and checks exact page
pricing. Image edit, Nova Canvas, DuckDuckGo, Firecrawl, Bedrock guardrail
and Presidio replacements now fake only the provider HTTP boundary (respx or
an in-process aiohttp server) and assert the outbound request, so the
DuckDuckGo limit, Azure base_model pricing and guardrail masking are proven
rather than assumed
* test: cover Exa and Perplexity search structure and max_results offline and retire the two base search methods
* test: drive batch and file replacements through the provider HTTP boundary and real logging callback
Replace monkeypatched litellm.afile_content and AsyncHTTPHandler doubles with respx routes,
read batch logging metadata from a registered success callback instead of get_logging_payload,
use the real managed-files hook for the GEN-2166 regression, assert outbound request bodies,
pin poller ownership explicitly in the migrated DB-sync tests, and require the scripted
upstream to be hit in the responses error-status wire tests
* test: assert file content download headers pass through the proxy
* test: point spend coverage references at the tests that replaced the retired spend job
* test: assert passthrough identity, spend and route dispatch from the code under test
The AssemblyAI non-admin test asserted metadata it wrote itself and leaked a
background poll to the real AssemblyAI host. It now drives assemblyai_proxy_route
with a real Request and waits for the success callback for its own transcript id.
The Vertex spend test matches its log by call id instead of taking the first event.
The OpenAI files wire test hit the native /{provider}/v1/files route; it now calls
/openai/files so the passthrough is what forwards the upload and delete.
* test: mock only the HTTP boundary in the migrated audio tests
Vertex TTS tests no longer replace _ensure_access_token or AsyncHTTPHandler.post;
the token comes from a mocked Google OAuth endpoint and the synthesize call from
respx. Speech tests assert the outbound body, the transcription cache test polls
for the cache write instead of relying on test ordering, and the model pass-through
test checks the multipart model field per model.
* test: wait on a logger event instead of polling the clock in anthropic messages unit tests
Recorders keep payloads in a rebound tuple and set an asyncio.Event; tests
await it with asyncio.wait_for instead of a sleep-and-deadline poll loop
* test: freeze module-level batch and file response fixtures as Final MappingProxyType
* test: fake the presidio analyzer with an in-memory aiohttp connector
The blocked-entity tests started an aiohttp TestServer, which binds a local
socket. They now hand the guardrail a ClientSession whose connector answers
/analyze and /anonymize in process, so no socket is opened and the outbound
analyze text and entities are still asserted
* test: type anthropic messages router test helpers with LiteLLM's Anthropic TypedDicts
Messages, cached system blocks and tool-search tools now use
AnthropicMessagesUserMessageParam, AnthropicMessagesTextParam,
AnthropicToolSearchToolRegex and AnthropicMessagesTool instead of bare
dict shapes; tools are converted to plain dicts only at the acreate call,
whose tools parameter is list[dict]
* test: type batch limiter helpers with TypedDicts and wait on the logging callback event instead of polling
* test: give the migrated OCR, image and presidio helpers precise types
OCR spend helpers take ReadOnly TypedDicts for kwargs and responses and use
LiteLLM's OCRResponse/OCRUsageInfo instead of local pydantic stand-ins; spend
metadata is validated with a TypeAdapter. The presidio fake uses LiteLLM's
PresidioAnalyzeRequest/ResponseItem types, and the image-edit logger validates
the logged payload instead of storing an untyped dict
* test: signal callback and cache events instead of polling
Recorders keep tuples and set an asyncio.Event, thread-safely, when the payload
for this test's transcript id or upstream URL arrives. The transcription cache
test waits on a Cache subclass that signals after async_add_cache. No clock
polling or sleeps remain in these tests.
* test: assert the batch limiter hook updates the caller's request in place
* test: tolerate model-list probes and read native passthrough rows by owned key
The router's OpenAI-compatible model-info refresh (litellm/router.py:10710)
sends GET /v1/models to configured openai api_bases, so the wire answers it
with an empty list and excludes it from the provider-call assertions. Native
/anthropic spend rows are now read by a per-request virtual key digest and
call_type, then the row's request_id is checked against the message id
* test: expect the OCR alias in the proxy response model
The proxy restamps every OpenAI-compatible response model to the name the
client requested (_override_openai_response_model), so /v1/ocr returns the
scenario alias. The upstream model is now checked on the drained request body
instead of inside the peer, where a failed assert never reached the test
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Rohan G
|
4417bf08ae
|
feat(guardrails): extend Akto guardrail to responses, MCP tools, attachments and masking (#44343)
* feat(guardrails): extend Akto guardrail to responses, MCP tools, attachments and masking - post_call now waits for Akto and blocks or masks the reply instead of only logging it - pre_mcp_call and post_mcp_call check MCP tool arguments and tool results - attached images, audio and files are sent to Akto's file check - streamed replies are checked every streaming_sampling_rate chunks - context_source routes traffic to Akto's endpoint or agentic policies - tags carry user email, team alias and key alias for attribution * fix(guardrails): harden Akto MCP detection, keep AGENTIC default, block unmappable masking - MCP handling trusts the logger's call type, so request body keys can't skip the prompt check - context_source defaults to AGENTIC, as before - masking that also hits text we can't write back now blocks - move tests to tests/unit and rename attachments.py to akto_attachments.py - regenerate the OpenAPI snapshot and dashboard types * fix(guardrails): ignore a client-sent response in Akto output checks - a "response" field sent in the request body is no longer scanned in place of the model's reply - MCP tool calls in the reply are still checked in that case - split match or-patterns so CodeQL can follow the bound names - cover a masked payload that is not JSON and drop unused test imports * refactor(guardrails): read Akto attachment fields directly instead of pattern captures * fix(guardrails): check Akto attachments in both messages and input * fix(guardrails): check every Akto attachment source and keep more prompt text in scope - check all of a file or image block's sources (file_data, file_url, file_id), since providers pick different ones - send Anthropic search_result blocks to the file check as text - keep document title and context, and legacy functions, in the checked request - take the client IP from the proxy's requester_ip_address before client forwarding headers - read litellm_params identity only from server-side call details * fix(guardrails): never drop an Akto attachment the text check removed - optional metadata (filename, title, format, media type) that isn't a string is ignored instead of failing the block - an attachment block that still can't be read blocks the request - search_result text is checked once, as a file, instead of also in the text check * fix(guardrails): strip only what the Akto file check sends from the text check - the text check keeps every attachment field except the ones the file check sends - document title/context and search_result source/title go to the file check as text, since the /v1/messages text check drops them - accept every image shape LiteLLM forwards (image_url or url, string or object) and check each source - ignore blocks whose type is not a string instead of failing the file check * fix(guardrails): keep model-visible text in the Akto text check - document title/context, text documents and search_result stay in the text check, so no Akto backend skips them - on /v1/messages the text check reads the messages Anthropic receives, with the guardrail's skip/scan scoping applied - a client-sent "response" key can only add reply checks, never skip recording or MCP tool-call checks - a "messages" key on the Responses API can't replace its input in the text check - the recorded IP comes only from the proxy's requester_ip_address * fix(guardrails): keep AktoGuardrail positional args backward compatible * fix(guardrails): keep zero Akto timeouts working and record every stream check A guardrail_timeout of 0 used to fall back to the default; the new ge=1 made the config invalid, so the proxy dropped the guardrail. Zero settings now fall back to the defaults again. The end-of-stream check can be skipped when the last sampled check covered the reply, so mid-stream checks now record, like base. * fix(guardrails): use defaults for non-positive Akto timeouts and sampling rate A zero or negative guardrail_timeout, file_guardrail_timeout or streaming_sampling_rate used to reach the HTTP call or the stream cadence. They now fall back to the defaults, like unset values. |
||
|
devin-ai-integration[bot]
|
fa2c8984ba
|
test: move the unit half of 126 mixed legacy files into tests/unit (#45090)
* test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
127278f951
|
feat(guardrails): add logging_only_scope to observe input, output, or both (#43695)
* feat(guardrails): add logging_only_scope to observe one direction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(guardrails): remove unrelated test churn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): allow native lifecycle logging-only scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate api types for logging_only_scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): remove callbacks when scope validation fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep output-only scans when request copy fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): configure logging_only_scope on guardrails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep guardrails enforcing when logging_only_scope is invalid at load Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): avoid inline guardrail test fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): support directional scope in PATCH and provider UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): format guardrail files for frontend lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): reset unsupported directional scope selections Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): normalize logging-only scope and sanitize warning logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): normalize scope in shared guardrail field Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(api): sync guardrail schema artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): tolerate invalid stored logging-only scopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): add logging_only_scope integration audit cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): fix scope test lint issues Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): extend K4 chaos test timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): restore stored guardrail row verbatim on rejected patch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): satisfy collection lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): validate masked params through TypeAdapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): tolerate invalid stored params on reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep constructor coercion in tolerant params parser Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): edit logging_only_scope in the custom code guardrail modal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): extract custom code logging-only scope control Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep no-op PUT and logging_only scan-failure semantics stable Three regressions from the logging_only_scope feature, fixed while keeping the input/output/both selection working: 1. PUT with byte-identical litellm_params no longer forces a teardown + re-init of the live callback. reject_invalid_logging_only_scope now only validates; re-init still happens exactly when params/name change. Previously a description-only PUT re-appended the callback at the END of litellm.callbacks, reordering guardrails: with a BLOCK guardrail created before a MASK one, the mask started winning and blocked requests started succeeding (400 -> PUT 200 -> 200). Invalid unchanged scopes are still rejected with 422 without touching the live instance. 2. CustomGuardrail._scan_logged_call no longer swallows input-scan exceptions per branch: a raising or BLOCKED input scan aborts the logging_only hook again (one policy call, one verdict) for guardrails that never selected a logging_only_scope. Explicit input/output/both selection keeps selecting which scans run. 3. A failed PATCH now rolls the DB row back to exactly the stored raw litellm_params (a legacy 4-key row stays 4 keys) instead of expanding it to a full LitellmParams dump; pinned with a test. This matches the PUT rollback shape and is the faithful rollback. * refactor(guardrails): type directional-scope provider list as a tuple * fix(lint): stay within the basedpyright budget - drop a Final annotation assigned inside the validation loop (reportGeneralTypeIssues over ceiling by one) - rename the PR-introduced _configured_event_hooks to public configured_event_hooks; its cross-module import added the two reportPrivateUsage errors that pushed the rule over its ceiling * fix(guardrails): keep the response verdict for explicit logging_only_scope=both on input-scan failure Greptile P1: an explicitly configured both-direction observer asked for a verdict on each direction, so a failed request scan must not silently drop the response verdict. The implicit default (logging_only_scope None) keeps the abort semantics of a logging_only hook whose scan raised, which is the base behavior the earlier fix restored. Also drops two test comments that restated their assertions (Greptile P2). * test(guardrails): align logging_only_scope integration rows with abort-on-input-failure and encrypted params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep the response scan when the request copy fails for explicit both scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
joshua-berri
|
950da7c2ec
|
fix(mcp): keep worker MCP configurations consistent via a catalog revision (#42568)
* fix(mcp): refresh server catalog for each gateway operation * fix(mcp): reject configuration changes during scoped dispatch * fix(mcp): refresh shared catalog state for each operation * fix(mcp): refresh catalog before native alias routing * docs(mcp): clarify native route resolution order * test(mcp): align streaming fixture and generated API documentation * fix(mcp): preserve discovery published during catalog refresh * fix(mcp): coalesce queued catalog reads without a stale window * fix(mcp): reconcile concurrent route changes when publishing catalog * test(mcp): provide catalog scope in post-call hook fixtures * fix(mcp): retain valid live routes and handlers during refresh * fix(mcp): keep refreshed OpenAPI operation membership authoritative * test(mcp): preserve logging fixtures after catalog integration * fix(mcp): isolate transport test admission state and exhaust OAuth outcomes * refactor(mcp): narrow catalog consistency change to ticket scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(mcp): gate catalog refresh on a database revision marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): refresh waiters that observed a newer catalog revision Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): awaitable catalog revision doubles in prisma mocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): satisfy catalog refresh lint and type gates * test(mcp): provide catalog scope in toolset fixtures * test(mcp): exercise temporary OAuth through catalog operations * fix(mcp): share active catalog snapshots across discovery tasks * fix(mcp): retain discovered routes across cached catalog operations * fix(mcp): preserve local tool ownership across discovery * refactor(mcp): satisfy tightened immutability lint ceiling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): authorize local handlers by registered server ownership * fix(mcp): check registered handler freshness and isolate fixtures * fix(mcp): read catalog revisions and snapshots from the writer * fix(mcp): resolve access groups from the operation catalog * fix(mcp): preserve empty access group restrictions * fix(mcp): retain rediscovered routes across concurrent updates * fix(mcp): enforce registered ownership for local tool dispatch * fix(mcp): preserve issuer discovery across catalog snapshots --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |