95 commits

Author	SHA1	Message	Date
joshua-berri	906fa7ed28	feat(accounting): add opt-in SpendLogs ledger (#49856) ... * feat(accounting): add opt-in SpendLogs ledger delivery * fix(accounting): preserve missing organization daily spend parity * fix(accounting): preserve ledger receipts when replaying migration * fix(accounting): preserve callback windows and tune ledger batches * fix(accounting): preserve window charges and bound ledger statements * test(accounting): verify budget callback registration order * test(accounting): verify captured charges after repeated startup * feat(accounting): store durable payloads independently of spend logs * fix(accounting): preserve first delivery and trace ledger writes * fix(accounting): keep SDK imports independent of proxy tracing * fix(accounting): honor log retention during batch takeover * test(accounting): cover retained batch alerts and invalid settings * test(accounting): verify downgrade command against pending work * fix(accounting): drain retries without starving fresh deliveries * test(accounting): cover delivery projection, metrics and window key reads Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): read delivery metrics without the shared registry Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): charge a leftover $0 batch row when its claim fails A failed batch claim buffers the retrieve's row and reports it charged. At flush the row conflicted with the pre-upgrade $0 SpendLogs row, so its delivery was rolled back and the charge never reached the spend tables. The flush now takes that row over in the same transaction and keeps the delivery. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): trace the batch takeover at flush as a SpendLogs update Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep local spend logs when SPEND_LOGS_URL has no export client The export client is never created, so main always writes SpendLogs to the proxy's own database even with SPEND_LOGS_URL set. The ledger skipped local rows whenever the URL was set, which dropped those request logs. It now treats a flush as an export only when the client exists, and batch claims write local rows as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep every request's charge when its log row is shared, evicted or rejected - Key non-batch deliveries on a per-request delivery_id so a reused client call id is still charged once per request, while buffer replays stay deduplicated - Persist the redacted api_key instead of an opaque bearer token - Keep the accounting payload when the spend-log queue evicts a row or Postgres rejects it - Strip NUL from payloads and skip non-text tags so projection never parks a paid request Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a raw-SQL spend log row Postgres rejects instead of requeueing the batch A rejected row on the ledger insert path raises RawQueryError, which the exact DataError check missed, so the whole batch was requeued every flush. Data and integrity SQLSTATE classes now bisect like before, while missing tables and deadlocks keep retrying Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging a request whose tag is too long for the daily tag index A tag past Postgres's btree entry limit failed the DailyTagSpend upsert, and on the ledger path that rolled back the request's key, team and user charges with it. Such tags are now left out of the daily tag rollup only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store only guardrail billing fields in the accounting payload The ledger payload copied whole guardrail records, including responses and match details, into LiteLLM_SpendLogDelivery even when spend logs were disabled. It now keeps the id, name, usage and cost fields the projection reads. NUL stripping reuses safe_json_structure instead of a recursive walk Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): move spend payload stamping out of update_database Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(proxy): restore the modules the import-layering guard evicts The guard popped litellm.proxy.utils from sys.modules for good, so later tests on the same xdist worker patched a fresh module while calling functions bound to the old one, and spend queue tests failed depending on scheduling Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): keep update_database's payload stamping on its own local Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): type guardrail entries and use a public key redaction helper Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep charging requests whose end-user id is too long for an index An oversized end-user id failed the SpendLogs insert with SQLSTATE 54000, which was requeued forever, and failed the end-user upsert in the delivery, which parked the key and team charge with it. Class 54 rows are now isolated like other rejected rows, and such ids are left out of the end-user rollups only Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(proxy): type the spend log queue snapshot Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): leave out any projected row Postgres cannot index or store Caller-supplied ids and counters, such as a long tag or end-user id or an out-of-range compression count, failed the delivery transaction and rolled back the request's key, team and user charges. One check in project() now drops only the row that cannot be stored, replacing the per-field cutoffs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep the guardrail fields compression savings are counted from Slimming the guardrail records dropped the provider and the tokens_saved figure that Headroom compression savings are read from, and dropped a bare-dict record entirely. The payload now keeps both while still leaving out guardrail content Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): cap projected counters so daily totals cannot overflow Counters such as a caller-supplied compression count fit one by one but overflowed the bigint daily total once summed across requests, rolling back the key, team and user charges in the same transaction. Each counter is now capped at 2^40, far below any total that could overflow Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): let a buffered batch charge take over its zero-cost receipt A batch charge queued after a failed lookup hit the $0 receipt an earlier poll left behind, and ON CONFLICT dropped it. The buffered insert now runs the same receipt takeover as a live retrieve Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read a rescued batch row's spend from its accounting payload Rows rescued from an overflowing queue keep only the accounting payload, so the buffered takeover never saw their positive spend and left the $0 receipt uncharged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): read budget windows through the delivery transaction Window seed and existing-row reads used a separate connection while the delivery transaction held one, so with a one-connection pool they waited on their own transaction until it timed out. Inside an enclosing transaction they now share its connection Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): retry a transient failure while keeping a dropped row's charge The rescue swallowed every error, so a database blip right after a row was rejected lost the unpaid charge. Only a rejected row is still logged and skipped, anything else propagates so the flush retries Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): give delivery projection payloads their charged entities project() now reads entities only from accounting_entities, so the hand-built delivery payloads carry them like real stored payloads do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): own ledger rows by their payload and key batches on the batch id Drop the accounting_path stamp and its SpendLogs column. A buffered row that carries an accounting payload is ledger owned, one without it is legacy, and spend counters get a plain windows_from_ledger flag. Window seeding tells applied ledger rows apart by their delivery row. A ledger batch is plain transaction A keyed by the batch id, so a repeated completed retrieve conflicts and charges nothing. A $0 retrieve records no delivery, a batch a legacy retrieve already charged is not charged again, and the legacy batch path is unchanged from main apart from refusing a batch the ledger already owns. Batch takeover and its delta bookkeeping are removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(accounting): drop a projection test duplicated while rebasing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop previous_payload now that deliveries are never replaced Batch takeover was the only writer of previous_payload, so the worker applies each delivery's payload as stored and the column leaves the migration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): drop the ledger downgrade checklist script The use_spend_logs_ledger flag is the rollback path, so a separate downgrade gate is no longer needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): name the window seed reads so their Prisma span resolves Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): remove the unused accounting event types and protocols types.py, protocols.py and measurements.py were left from the old event scaffolding. Nothing outside the package imports them, so they go with their tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): park only deliveries whose own data fails A delivery that keeps failing on a database outage, lock timeout or deadlock stays pending and retries with capped backoff, so a recovered database applies it. Only a data error or an invalid payload parks the isolated row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate a spend log row the insert statement rejects The raw SpendLogs insert validates columns and JSON before Postgres sees the row, so a row that fails there is now bisected out and dropped alone while its charge is kept, instead of failing and replaying the whole flush. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): carry a job's model window on every completed poll in ledger mode The ledger keeps the poll whose delivery claims the job, which need not be the poll that won the limiter's in-memory charge. Every completed poll now carries the job's window, so the delivery that lands always has it, while the live model budget is still charged once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * refactor(accounting): leave model-budget windows on the existing window writer Model-budget windows are written by the model budget limiter rather than derived from a request's spend, so they stay on main's window queue like live budget counters do. This drops window capture from the accounting payload, the deferred limiter registration and the job-window capture, and the ledger keeps owning key and team budget windows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): keep a delivery pending when the database is unreachable Prisma reports a can't reach database server outage as a DataError, so the worker now checks for an unavailable database before treating a data error as the delivery's own fault. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): isolate only rows the raw insert cannot encode Catching every ValueError also swallowed unexpected insert errors that main re-raises. The raw insert now raises RejectedSpendLogRow for an unknown column or non-JSON value, and only that is isolated as a rejected row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(accounting): store zero savings when caller data makes them uncomputable A huge caller-supplied tokens_saved overflowed the savings math inside build_accounting_payload, which dropped the request's charge in ledger mode. Savings that cannot be computed are now recorded as zero and the charge lands. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-11-04 17:49:15 -08:00
Meryem Sakin	4a7fb30b0a	fix(guardrails)!: check Anthropic and Responses request tools in tool_permission pre_call (#41011) ... * fix(guardrails): check Anthropic-format request tools in tool_permission pre_call The pre_call hook only collected tools shaped as {"type": "function", "function": {...}}, so flat tools from /v1/messages ({"name", "input_schema"}) and /v1/responses ({"type": "function", "name"}) were never checked and deny rules silently passed. Reuse the shared anthropic_tool_names helper so every request tool name is checked, and handle Anthropic tool_choice {"type": "tool"} in both block and rewrite modes. Fixes #40583 * fix(guardrails): detect Responses API named tool_choice in tool_permission Responses API named choices put the name at the top level ({"type": "function", "name": ...}), so rewrite mode removed the denied tool but left a tool_choice pointing at it. * test(guardrails): run tool_permission pre_call tests through real default_on gating Build the guardrail as an always-on pre_call guardrail instead of patching should_run_guardrail. * test(guardrails): type the new tool_permission pre_call test parameters * fix(guardrails): keep the Responses API custom tool type distinct Anthropic's user-defined tools carry type "custom", so pre_call mapped that to "function" for tool_type rules. The Responses API has its own custom tool type, and mapping it let a custom tool slip past a tool_type deny rule when the default action is allow. Normalize only for the formats where custom means a plain function, and read tool names directly in the rewrite path, which never needed a type. * test(guardrails): cover flat tools through Messages and Responses Add blocked and permitted flat-tool cases through the running proxy, with endpoint coverage and metadata for the current e2e harness Require a guardrail policy verdict when checking denied tools so an unrelated provider error cannot satisfy the regression test Co-authored-by: Christopher Krah <christopher.krah@plexito.de> * fix(guardrails): check namespace and nameless tools and relax tool_choice in tool permission rewrite * fix(guardrails): check forced, grouped, loaded, and hosted tools by their real type * fix(guardrails): drop references to removed tools and count loaded tools in tool permission rewrite * fix(guardrails): check tools sent outside top-level tools in tool permission * fix(guardrails): judge namespace members by their bare name as a fallback and keep rewrites valid for Gemini, Bedrock and hosted tools A namespace member is checked by namespace__name first, and by its bare name when no rule names the joined one. Rewrite mode now prunes Gemini allowedFunctionNames and turns off forced calling when no function is left, drops a removed hosted tool from allowed_tools, reads Bedrock toolSpec and systemTool blocks by name, skips cachePoint markers, and leaves a tools field that is not a list for the provider to reject instead of failing with a 500 * fix(guardrails): let only name rules decide a namespace member's bare-name fallback * fix(guardrails): judge malformed tool types, empty Gemini name lists and pass-through Anthropic tools like the endpoints do A tool or tool_choice whose type is not a string now reads as untyped instead of crashing the guardrail with a 500, and a null type key is never read as a Gemini tool part. An empty allowedFunctionNames list no longer turns off the functions a rewrite kept. An Anthropic custom tool reads as a function by its input_schema on every route, so a pass-through route judges it the same as /v1/messages. A Responses WebSocket frame now forwards the extra_body tools a rewrite stripped * fix(guardrails): judge a named hosted tool by its type and keep a nested chat custom tool custom * fix(guardrails): keep judging Anthropic client tools and type-named server tools by their name * fix(guardrails): judge a named tool by its name alone only when the name is its type's own * fix(guardrails): judge the function a tool names beside a declaration list or a top-level name * fix(guardrails): judge Gemini hosted keys whatever the tool type and a namespace's function * fix(guardrails): judge legacy functions entries like tools and send Gemini a namespace's members * fix(guardrails): judge a namespace member by every name a provider reads and send Gemini its schema converted * fix(guardrails): judge a legacy functions entry as a function and relax a tool_choice in extra_body * fix(gemini): keep a namespace whole when inlining its members would repeat a function name * fix(guardrails): judge a namespace member by every bare name that shares its qualified name * fix(vertex_ai): declare a repeated namespace member name once instead of sending namespace names to Gemini * fix(guardrails): judge a tool_choice sent in extra_body like a top-level one A forced name only extra_body.tool_choice carried was never checked, so block mode let a denied name through to the provider * fix(guardrails): judge a Gemini namespace member returned by its bare name and force a named namespace's members * test: keep formatter churn out of the tool_permission and Gemini test files * fix(guardrails): judge extra_body.input tools, argument-aware alias rules, and exact forced Gemini functions Tools loaded in extra_body.input are now checked and stripped like top-level input. A returned namespace member is judged by the alias whose first name rule its arguments satisfy. A forced Gemini name that matches a top-level function keeps that function. Alias grouping and Gemini declaration dedupe now run in one pass instead of quadratic scans. * fix(guardrails): judge and rewrite every tool field sent inside extra_body extra_body is merged into the upstream body, so functions, function_call, mcp_servers and toolConfig nested there reached the model unjudged. Treat extra_body as a second request layer that is collected and rewritten the same way as the top-level request. * fix(guardrails): judge Gemini hosted keys by key and count only the tools a provider receives after extra_body * fix(guardrails): relax a tool_choice that names a dropped namespace member by its bare name Gemini and Anthropic chat send namespace members under their bare names, so rewrite mode now treats a dropped member's bare name, and an emptied namespace's own name, as removed when no kept tool still carries it * fix(gemini,guardrails): keep a top-level function over a same-named namespace member and leave malformed tool values to the provider Gemini and Vertex chat now declare a top-level function over a namespace member that shares its name, so a forced call gets the top-level schema. post_call judges a returned name a top-level tool carries as that tool, and pre_call no longer raises on an unhashable or non-string tool name, input item type, or function_call name. * fix(guardrails): judge a returned namespace member named like a hosted MCP server as that member * fix(gemini,guardrails): send a namespace as main did when another tool has a member's name Gemini chat inlines a namespace's members only when no other tool, namespace member, or legacy functions entry declares the same name, and otherwise sends the namespace by its own name as main did. A tool_choice naming such a namespace forces the namespace itself. The tool permission post_call reads top-level names the way Gemini declares them and judges a call the Responses bridge tags with a namespace as that member. * fix(gemini): send colliding namespace members as namespace__name functions and judge them in post_call When a namespace member shares a name with another tool, Gemini chat now gets that namespace's members as namespace__name functions instead of one function named after the namespace, so every function it can call is a name the rules judged. Declared names are counted once, and a forced member maps to each holder's qualified name. post_call judges a returned namespace__name as that member, by its qualified or bare name the way pre_call does, so a bare-name allowlist or denylist applies to it. * fix(gemini): keep generated member names unique and let a top-level function own its exact name A namespace that listed a member twice, or a generated namespace__name that repeated a top-level function, sent Gemini two declarations with one name, and Gemini answered 400 where main answered 200. A name a namespace lists twice is now one member, and a top-level function keeps its exact name. post_call now judges a returned name a top-level function declares as that function, never as the namespace member the name could also mean. * fix(gemini): qualify namespace members by the namespace's top-level name The guardrail judges a member as `name__member` from the namespace's top-level name. Gemini built the name from `function.name` when the tool also carried one, so a request could pass pre-call and reach Gemini under a name no rule judged. * fix(vertex_ai): force only the namespace members Gemini is sent A forced namespace or member on Gemini chat could still allow a generated name that a top-level function or an earlier namespace owns, so tool_choice forced an unrelated function. The forced names now come from the same collision checks the declarations use * fix(vertex_ai): reject a forced namespace Gemini gets no member of, and resolve forced names in one pass * fix(guardrails): judge a namespace member by a rule on its namespace's name --------- Co-authored-by: Christopher Krah <christopher.krah@plexito.de> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-04 20:34:34 +00:00
tessellark	f4a2e2b3af	fix(anthropic): send an https PDF image_url to Claude as a document (#49708) ... * fix(anthropic): send an https PDF image_url to Claude as a document A PDF link sent as image_url to Claude on Anthropic went out as an image block and Anthropic answered 400. A link whose path ends in .pdf is now downloaded like Vertex links are, and the served Content-Type picks the document or image block, so CDN links that serve a PNG from a .pdf path keep working. An explicit format of application/pdf sends a url document with no download. PDF media types match regardless of case or parameters, and count_tokens picks the same block as the request. * fix(anthropic): download https PDF links off the event loop The proxy downloaded an https link ending in .pdf synchronously while building the Anthropic request, so on an async call every other request on that worker waited for the download. Those links are now inlined before the request is built, the same way http links already are, and the /v1/messages guardrail write-back does the same. A non-string format on an image_url is ignored the way it already was inside the image_url mapping, instead of raising an AttributeError. * fix(anthropic): send a PDF link the proxy cannot download as a link and keep guardrail url blocks as sent * test(anthropic): expect the PNG media type the served bytes carry at a .jpg path * fix(anthropic): bound the PDF link download and keep query-string PDF names Give up on a .pdf link after NON_LLM_CONNECTION_TIMEOUT instead of the client's 6000 s default, fall back to main's whole-URL extension check so download?file=report.pdf still counts as a PDF, treat application/x-pdf as a PDF, and set the url document's application/pdf format only for the guardrail write-back so other /v1/messages providers send what main sends * fix(anthropic): probe a .pdf link and send it to Claude as a url document Instead of downloading a .pdf link and inlining it as base64, ask the host for its first kilobyte and send the link as a url document when it serves a PDF. Every other link goes out as the url image main sends, so robots.txt handling, caching and the wire format stay as they are on main. Also restore the guardrail adapter and the content type inference to main, keep the client's url document blocks (title, context, citations, cache_control) through the guardrail write-back, treat a PDF filename in the query string as a PDF link, probe links concurrently on the sync path, cap the async probe's read at the download limit, and catch only transport, URL and SSRF failures in the probe * fix(anthropic): read only the first kilobyte when probing a PDF link, and restore each url document from its own place The probe now streams the response and keeps the first 1 KB, so a host that ignores Range no longer fills memory with up to 20 parallel bodies. The guardrail write-back now matches url documents inside a tool_result and outside one separately, since the write-back moves tool results ahead of other content * fix(anthropic): remember each PDF link probe's verdict and cap its total time, and skip the probe on Bedrock invoke * fix(anthropic): keep a PDF probe's error status for a minute, not ten --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-03 07:05:15 +00:00
simonp9	61da6fcb4d	fix(mcp)!: answer 400 for a hosted MCP tool a translated route cannot run, drop it under drop_params (#48931) ... * fix(bedrock, gemini, deepseek, databricks, gigachat, anthropic): answer 400 for a hosted MCP tool a route cannot run, drop it under drop_params A `type: "mcp"` tool sent to a route that rebuilds tools for its provider (Bedrock Converse and Nova, Bedrock Invoke Claude, Vertex AI Claude, Vertex AI and Google AI Studio Gemini, DeepSeek, Databricks Claude, GigaChat) was dropped silently and the request still answered 200. These routes now answer UnsupportedParamsError (400) naming the model, route and server, and pointing to the LiteLLM MCP gateway. Under drop_params the tool is dropped with a warning, other tools are kept, and no empty tools field is sent. Anthropic, Azure AI Claude and Claude Platform on AWS keep mapping the tool to the MCP connector. Native Responses providers and raw OpenAI-compatible forwarders are unchanged. * fix(mcp): check hosted MCP tools once per request and keep base tool_choice handling Each route now runs one request-level step before its param mapping. When the drop leaves no tools, the request sends no tools, tool_choice or parallel_tool_calls. Converse and GigaChat go back to their base tool_choice and empty-tools handling. * refactor(deepseek): hold the sent tools in a tuple * fix(mcp): keep tool_choice for legacy functions when every hosted MCP tool drops * fix(mcp): answer 400 for MCP servers a route cannot run on Messages, Ollama and the Responses playground /v1/messages requests with mcp_servers on a chat-adapter or Responses-bridge route now answer 400 naming each server, or drop it with a warning under drop_params, instead of being forwarded or silently ignored. Ollama generate no longer pastes a hosted MCP tool (headers included) into the prompt, and ollama_chat filters hosted MCP tools the same way. The Admin UI Responses playground now sends litellm_proxy/mcp references like the other endpoints. * fix(ollama): leave JSON mode off when drop_params drops every hosted MCP tool Ollama generate set format json for any request with tools, even after the hosted MCP check dropped all of them, so a plain answer came back as JSON or as a call to a tool the model never had. The check now runs before the JSON mode switch, which stays on only while a tool is left for the prompt. * fix(messages): honor additional_drop_params for mcp_servers and read the beta map only for hosted MCP tools * fix(mcp_security): accept gateway server names and toolsets the gateway resolves The guardrail compared litellm_proxy/mcp/<name> refs against registry keys, which are server ids, so a saved server referenced by name, alias, or toolset name was blocked although the gateway routes it. It now resolves each ref the way the gateway does and still blocks unknown names. * fix(tool_fields): keep MCP drop warnings on one log line Server labels, model names, and routes come from the caller, so a line break in one could forge extra log lines. The drop warnings now strip CR and LF from each value. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 13:13:03 -08:00
Kaustubh Anil Jain	6f3251c7e3	fix(responses): serialize sequence_number on every bridged streaming event (#44944) ... * fix(responses): serialize sequence_number on every bridged streaming event * test(responses): cover sequence_number on bridged streaming events * fix(responses): read undeclared fields through dict accessors and number guardrail block continuations * fix(responses): scope dict reads that see undeclared fields to Responses stream events Moves the extras-aware __getitem__, get, __contains__ and items from the shared response base onto a new base for the 37 stream event classes, so usage, Gemini, Interactions and moderation objects keep reading __dict__ and Gemini responses keep _hidden_params out of .get * fix(router): renumber Responses stream events only after a fallback takes over A stream that never falls back now keeps its provider's own sequence numbers, repeats included, so a starting_after resume still matches them. After a fallback the wrapper continues above the highest number the client saw * fix(guardrails): keep Responses stream numbers unique when LLM Shield flushes held text The restorer's flush copied the last delta and kept its sequence_number, so a native stream repeated a number and a bridged stream sent the flush unnumbered. A flush now takes the done event's slot and later events move up by one, and a stream the provider never closed numbers its flush after the last event. Also pins the created_at in the streaming iterator test fixture instead of reading the clock. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-11-02 09:01:58 -08:00
yucheng-berri	0f4d5c6ad0	feat(ovalix)!: scan files, tool calls and tool results (#49309) ... * feat(ovalix): file and tool scanning, app discovery from the key alias Mirror of BerriAI/litellm#34132 at 5ffc268585, rebuilt on main. * fix(ovalix): keep inspecting requests whose key has no user The tracker allows a checkpoint call with an empty actor without inspecting it, so a key with no user (or a /v1/messages or /v1/responses request that carries its own metadata object) got no Ovalix enforcement. Fall back to "unknown" as the guardrail did before, so every request is inspected. * refactor(ovalix): check texts without mutation and test discovery through tracker requests - _check_texts builds its result from per-text verdicts checked newest first by _check_text - discovery and routing cache unit tests answer the tracker routes instead of replacing private methods - integration test adds streaming cells and a tracker outage during a concurrent burst * fix(ovalix): resolve tool result names in one pass Each tool result rescanned every earlier message for its call id, so a request with thousands of tool results spent seconds in the event loop. Declarations are now indexed once per request and each result takes the nearest earlier one by bisection * test(ovalix): annotate new test bindings and build the burst plan with product * fix(ovalix): drop app discovery from the key alias A config with no application_id no longer boots in discovery mode, where the [Name] prefix of the key alias picked the Ovalix application and so its policy. Anyone who could rename a key could move it to a weaker app, or to an unknown name the tracker created with its default policy. The guardrail now needs application_id at startup again, as on main, and every checkpoint call goes by the configured application and checkpoint ids. Removes enable_routing_cache and fail_if_no_application, the app_name_regex and resolve_application tracker calls, and the per-alias routing cache. Restores main's startup check that a requested pre_call or post_call hook has its checkpoint id. --------- Co-authored-by: Shalom Jamil <shalom@ovalix.io>	2026-10-31 17:01:12 -07:00
ishaan-j-berri	6105aaa63e	Configure full Bedrock guardrail assessments (#49217) (#49470)	2026-10-31 23:37:45 +00:00
ishaan-j-berri	8d26b8bba3	Return configured guardrail native findings (#49217) (#49460)	2026-10-31 23:37:45 +00:00
moe-berri	bba63c1934	fix(proxy): serve /a2a/{agent_id}/message/stream routes (#49471) ... * test(a2a): cover message stream route aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): serve /a2a/{agent_id}/message/stream routes Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): document A2A stream route schemas * fix(proxy): use immutable A2A schema mappings * fix(proxy): map v1 a2a stream route to call types and keep stream alias unbilled * test(a2a): use real guardrail registry and typed locals in stream route tests --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Rustaman Delvianova <96533738+Rustaman1280@users.noreply.github.com>	2026-10-31 16:11:51 -07:00
paveln83	febd340120	fix(guardrails)!: keep the upstream error when a /v1/messages stream ends in one (#49088) ... * fix(guardrails): keep the upstream error when a /v1/messages stream ends in one Raw-SSE guardrail stream hooks assembled the whole stream, and the Anthropic chunk parser raises on an error event, so a stream the upstream ended with event: error failed assembly. tool_permission then blocked it with a 400 that hid the upstream error, bedrock and model_armor emitted a could-not-assemble block, and presidio skipped masking. Assembly now leaves the error events out, so the delivered content is checked as usual and a clean stream passes through with its error frame. A rewritten stream ends with the upstream error instead of a message_delta and message_stop the upstream never sent. tool_permission also passes a stream of only error frames, which carries no tool call to check. * fix(guardrails): do not close a block the failed upstream left open when re-emitting a rewritten stream * fix(guardrails): check only the finished blocks of a stream cut off by an upstream error tool_permission and Presidio now assemble an error-ended stream through the error with an opt-in keyword, leaving out a tool call or thinking block the upstream cut off. Logging, OpenTelemetry, Bedrock, and Model Armor keep the old assembly. A rewrite drops the text block stop only when the upstream left its text block open. * fix(guardrails): keep a cut-off tool call from the client and stop each rewritten block before the next starts A failed stream holding a cut-off tool_use is now re-emitted through the rewrite, which leaves the call out and keeps the upstream error, so a denied call never reaches the client. The rewrite leaves its text block open only when it is the last block, and keeps a message_delta the upstream sent before failing. The new tool_permission tests no longer patch should_run_guardrail. * fix(guardrails): pass a cut-off stream with nothing denied through as sent, minus the cut-off tool call Re-emitting such a stream through the rewrite sent an empty text block and stopped it, and Claude Code does not retry a failed stream that shows a finished block, so it showed a mid-response error instead of retrying. Dropping only the cut-off call's events keeps the stream as the upstream sent it. * fix(guardrails): strip a cut-off tool call only from a stream the upstream failed Assembly leaves unfinished blocks out of the check only when the stream carries an upstream error event, so a stream with an unfinished but allowed tool call and no error frame now reaches the client as sent instead of losing a call the check already read * test(guardrails): type the parameters of the new stream error tests * fix(guardrails): skip SSE data lines that are not JSON objects when reading a stream A data line holding a bare JSON value such as 5 made the stream helpers call .get on an int, so tool_permission and Presidio answered 500 where main answered 400 or passed the stream through * fix(anthropic): skip SSE data lines that are not JSON objects when assembling a stream * test(guardrails): annotate the upstream error stream tests * fix(anthropic): read a stream's non-object lines and named error frames the same way everywhere Run the skip for a data line that is not a JSON object through extract_sse_data and the assembler loop, so the usage fallback keeps a stream's tokens and the chat handler keeps its contract. Presidio now holds a leading non-object frame until the first object frame, so the rest of the stream is still masked or blocked. An event: error frame now counts by its name, so a rewrite keeps it. * fix(guardrails): refuse an unreadable stream in Presidio and cut every kind of tool call A non-object JSON line with no Anthropic frame after it now gets a 400 from Presidio before any data byte, since passing it would send text main refused. A frame named error stays a normal event when its data is another Anthropic event. server_tool_use and mcp_tool_use cut off by an upstream error are dropped like tool_use, and a leading frame with empty data is relayed at once. * fix(guardrails): refuse what Presidio cannot read past a non-object line and keep non-string error types out of the frozenset Presidio now refuses a stream assembly cannot read when it holds a non-object data line, instead of relaying it unmasked. Non-string event types no longer crash a rewrite, and a CRLF stream that opens with a typed error is no longer read as one error, so its tool calls are still checked. * fix(guardrails): refuse a first frame whose type is a list or dict in Presidio Presidio passed PII at 200 where main answered 500 when the first read was a JSON object with a list or dict type, or a JSON array with a character cut at the end of the read. has_unreadable_json_event now reads both as data no reader of these streams can take, so the first-frame check refuses them. * test(guardrails): cover Presidio's refusal when a word or null line comes before a Gemini stream The first read then holds the unreadable line behind one Presidio can pass, so the check has to look at every frame in it. * fix(guardrails): treat a stream as error-only only when every data frame is an error A tool call split over data: lines beside one error frame read as error-only, so tool_permission passed it, and an SDK joins those lines and reads the call. The check now joins a frame's data lines and refuses anything but an error. * fix(guardrails): refuse a stream past an upstream error when a client could read a frame differently Run 8 of the live risk pass showed each stream reader imitating a client and missing a shape: a whitespace-led data line, an error key with no type, a split tool call after a plain message_start. An error frame now has to be plain, and assembly past an error returns None for any frame a client could read differently, so those streams keep main's refusal. * fix(guardrails): on a failed upstream, deliver only what tool_permission read A failed stream now reaches the client through the rewrite that already serves a denial, followed by the upstream error, even when nothing is denied. Raw frames could carry a tool call the check never read in shapes such as a reused block index, mixed line ends, a tool in message_start, interleaved deltas, or a form feed, and each refusal rule only moved the search to the next shape. The error frame goes out as its event name and first data line. An error is found at CR, LF and CRLF line ends, and the rewrite adds no finished block when it read no text, so Claude Code still retries the stream. * fix(guardrails): read an error frame's name as clients do and send the error alone when a rewrite cannot be built A data-less event: error frame with junk after the name is an unknown event to both SDKs, so a healthy stream stays raw. A finished tool call whose input is not JSON cannot be rebuilt, so a failed stream sends its upstream error alone, and a healthy stream still raises as before. * fix(guardrails): read an error frame as a client dispatches it and keep its JSON whole Only the last event line of a frame a blank line ended names an error, so a healthy stream with a later event line or an unterminated error is no failed stream. Any rebuild failure on a failed stream now sends the upstream error alone with a warning, a param rule denies an input nested too deep, and a raw line break inside a JSON error stays escaped so every reader gets it whole. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 02:56:20 -07:00
paveln83	27fbcd9895	fix(anthropic): map pause_turn and return refusal explanations on chat and responses (#49192) ... * fix(anthropic): map pause_turn and return refusal explanations on chat and responses pause_turn now maps to finish_reason length (Python and Rust) and keeps the raw stop_reason in provider_specific_fields. Responses reports a paused turn as incomplete with max_output_tokens and shows the unrun server search as an in_progress web_search_call instead of a client function_call. A refusal's stop_details.explanation now fills message.refusal and the final stream delta.refusal on chat, and becomes a refusal content part on Responses, streamed or not. * fix(anthropic): let a chat client resend a paused turn so Claude continues it * fix(anthropic): replay a paused chat turn without its thinking blocks, and drop its unrun search before a new message Chat cannot say where Claude wrote each thinking block in a paused web search turn, and Anthropic answers 400 when one moves but accepts the turn without them. A new message after a paused turn also 400s on a server_tool_use with no result, so that unrun search is dropped. * fix(responses): send the streamed refusal part's type and text in content_part.added * fix(anthropic): keep pause_turn and Claude refusal explanations out of the Messages adapter's text A paused Claude turn now ends on stop_reason pause_turn instead of max_tokens, streamed or not. A Claude refusal keeps its explanation in stop_details with Claude's category and no longer repeats it as a text block, so a guardrail mask on a streamed refusal is delivered. A paused turn sent back with list content drops its thinking blocks like string content does * fix(anthropic): replay a paused turn with one unrun search and no thinking on chat and Responses * fix(responses): report a paused Claude turn as completed so Codex and the Agents SDK keep going * fix(responses): mark a paused turn's message item completed * fix(responses): mark every item of a paused turn completed * fix(responses): keep a refusal's explanation when a refused turn is sent back A refusal part in a Responses input message now goes to the model as text, so the assistant turn is no longer dropped from the history. The integration test now expects a paused Claude turn on Responses to report completed, and the paused-turn test helper is typed * fix(responses): scan refusal parts in Responses guardrails --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-31 02:53:59 -07:00
Nate Armstrong	87d91d78b7	refactor(router): route the router family through RoutingContext and close the outside surface (#49118) ... Strategies, pre-call checks, cooldown and fallback helpers, and the auto, complexity and quality sub-routers now take a RoutingContext instead of the Router, and read the router only through registry(), settings(), health() and the context's forwarding methods. Router-free helpers move from router_utils to litellm_core_utils, outside callers of router state switch to entry points, and the surface allowlist shrinks to the endpoint methods plus the entry points	2026-10-30 19:18:54 -07:00
tin-berri	0d5cf9b2f7	fix(headroom): forward tenant-scoped session affinity (#49255)	2026-10-30 12:32:03 -07:00
Mateo Wang	cbfb1abdb3	fix(proxy)!: label shared-handler errors the way the route catch-alls do (#39555) ... * fix(proxy): name shared-handler errors by the status they answer handle_exception_on_proxy pinned type=internal_server_error on every exception it wrapped, so a management route rejecting a caller's own argument answered {"type":"internal_server_error","code":"400"} and an SDK branching on the type read its own bad request as a gateway outage. Derive the type from the status the response actually carries, keeping a type the exception already names. * test(proxy): pin a 500 for exceptions whose status is None * fix(proxy): keep internal_server_error for timeouts and 5xx, redact hosts in shared handler errors Provider 5xx and timeouts keep internal_server_error so retry loops still retry them, the shared handler redacts internal hosts from raw exception messages, and /key/list routes its HTTPException through the shared handler instead of its own copy of the old label. * fix(proxy): label shared-handler errors like the chat route, pick the status before the type, redact only gateway failures * fix(proxy): pass shared-handler error messages through as on main Redacting timeout and 5xx messages changed what every shared-handler route returns, which relabeling the error type does not need, so messages go back to the verbatim text main sends. * fix(proxy): redact only the status-less failure main never sent * fix(proxy): answer a generic 500 for a status-less failure and expect auth_error on the /user/new 403 A failure with no status now answers "Internal server error" instead of its own text, since that text can name a guardrail's private host and pattern redaction misses cluster DNS names and IPv6 addresses. The /user/new integration test now expects the auth_error type main's route catch-alls give a 403 the proxy raises * fix(proxy): answer only a 4xx or 5xx carried status and fall back to the route default otherwise A numeric-string status outside 400 to 599 made /v1/messages, images, rerank and the OpenAI pass-through answer HTTP 200 with an error body or drop the connection, and non-ASCII digits or a 5000-digit string slipped through isdecimal. error_status_code now accepts an int or a three-digit ASCII string in 400 to 599, the range the chat route already enforces * refactor(proxy): parse a three-digit string status without recursing	2026-10-30 06:58:05 -07:00
simonp9	594f862bf3	fix(streaming): rebuild each choice of an n>1 chat stream on its own (#49002) ... * fix(streaming): rebuild each choice of an n>1 chat stream on its own stream_chunk_builder folded every choice of an n>1 chat stream into choice 0, so post-call guardrails such as tool_permission re-emitted one choice holding every choice's tool calls, and spend logs and logging callbacks kept only the first choice's text. Each choice is now assembled from its own chunks, while usage, cost, and hidden params still read the whole stream. Cisco AI Defense compares each choice's streamed text on its own so safe n>1 streams still pass through unchanged. * fix(streaming): keep bridged reasoning summaries on choice 0 and infer an unreported choice finish reason * fix(streaming): match MCP follow-ups and bridged summary text to the rebuilt choices MCP auto-execute now answers only the first choice's tool calls, since its follow-up turn continues that choice alone. The Responses bridge streams a space before each reasoning summary part after the first, matching the non-streamed reply. build_base_response now takes the per-choice streams instead of having its choices replaced afterwards * fix(streaming): space every later reasoning summary part, deny per judged call, and rebuild in one text pass * fix(guardrails): key tool_permission denials for custom tool calls too, and type the stream rebuild's messages * fix(mcp): continue the first choice that called a tool when an n>1 reply auto-executes MCP tools * fix(noma): scan each choice of a multi-choice reply on its own * fix(noma): keep masked choices when a later choice's scan fails open on a stream --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 18:29:14 -07:00
Joshua Hoblitt	d8f162ea22	fix(guardrails): allow MCP call modes on the generic guardrail API (#48060) ... GenericGuardrailAPI.get_supported_event_hooks only listed pre_call, post_call and during_call, so event hook validation rejected the MCP modes and the guardrail was skipped at proxy startup instead of being attached to MCP tool calls. Add pre_mcp_call, during_mcp_call and post_mcp_call. during_mcp_call runs alongside the tool call, so it can fail the call but cannot rewrite its arguments.	2026-10-29 15:32:37 -07:00
yucheng-berri	a54204b583	feat(guardrails): scan sampled stream windows in the background (#48952) ... * feat(guardrails): scan sampled stream windows in the background (LIT-9338) streaming_scan_in_background runs each sampled post_call scan as a task so the stream keeps flowing; a block ends the stream before the next chunk. * fix(guardrails): hold the stream end for the background verdict and cancel the scan on upstream errors (LIT-9338) * fix(guardrails): wait for the cancelled background scan on upstream errors (LIT-9338) * test(guardrails): cover /v1/responses, a failing scan peer and a mixed burst for background scans (LIT-9338) * test(guardrails): build the burst calls without a nested comprehension (LIT-9338)	2026-10-29 12:19:39 -07:00
yucheng-berri	83d6779158	fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask (#48944) ... * fix(guardrails)!: make lakera_v2 on_flagged block reject PII-only detections, add mask on_flagged block now returns 400 when Lakera flags only PII, instead of masking it and forwarding the request. on_flagged mask keeps the old behavior: mask PII-only detections and block anything else. * test(guardrails): cover lakera_v2 on_flagged on every endpoint, hook and a Lakera outage * test(guardrails): type the lakera_v2 on_flagged unit test helpers	2026-10-29 11:48:37 -07:00
devin-ai-integration[bot]	a0a8ca97d9	fix(cost): finalize synchronous response cost once for header, spend log and rollups (#48993) ... * fix(cost): finalize synchronous response cost before accounting callbacks * fix(cost): keep serving and billing when cost finalization fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): read the finalized cost only off real logging objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): follow the renamed success dispatch and deferred sync callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): keep cache hits free and their headers finalized when logging is deferred Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): preserve completed synchronous charges after hook failures * fix(cost): retain completed usage across cancellation and log suppression * fix(cost): reconcile completed failures and retain cached usage metadata * fix(cost): retain finalized request tags in failure budget updates * fix(cost): honor disabled spend tracking for completed failures * fix(cost): preserve pass-through accounting across cancellation --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 11:46:49 -07:00
mrinal-berri	4a4c9cf7e3	feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block (#46219) ... * feat(guardrails): opt-in response_masked_data on PANW Prisma AIRS block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): integration cells for PANW include_masked_response_on_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): reject non-boolean include_masked_response_on_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(guardrails): update PANW tool-call comment for include_masked_response_on_block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): build PANW kwargs without mutation and clean up DB-created test guardrails Restore main's single-expression kwargs in the PANW initializer. The two DB-backed integration tests now create their guardrail inside the scenario and delete it on exit. * fix(guardrails): type raw PANW boolean setting before validation --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 10:45:00 -07:00
Nate Armstrong	ead221995e	refactor(router): move router callers onto the entry points and invert the proxy imports (#48847) ... Code outside the router now reaches it only through the entry points (registry, settings, health, select, apply, observe, register_plugin, retire) or the endpoint methods; the outside-member allowlist shrinks from 139 to 45. litellm/router.py no longer imports litellm.proxy: it asks the installed RouterHost (litellm/router_host.py) for vertex pass-through credentials, member auto-router authorization and auto-router prompt compression, and the proxy installs its host on import. Call sites where a model name can be missing keep the old answer instead of reaching an entry point with None.	2026-10-29 17:26:33 +00:00
devin-ai-integration[bot]	c45b264024	fix(proxy): persist tag spend and drain success callbacks within the shutdown budget (#48988) ... * fix: drain shutdown callbacks and persist pending tag spend * fix: catch shutdown timeouts on Python 3.10 * fix: share shutdown callback budget with HTTP draining * fix: type shutdown cancellation tasks and document its clock * test: retain callbacks after bounded worker cancellation * test(proxy): inspect the lifespan body after the SIGTERM wrapper split Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(utils): keep the success callback enqueue on its own task The synchronous enqueue spawned the logging worker loop inside the awaited call and ran callbacks that per-call loops used to drop, which hung the SDK callback tests on the autouse worker flush. The shutdown worker flush covers the queued callback on its own, so the one-tick task is restored Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging_worker): keep the queue's unfinished count exact when a full queue rejects a callback The aggressive clear ran the rejected callback and marked it done against the queue it was never put in, so flush() could return while a real callback was still running and the shutdown drain cancelled it early. The rejected callback is now re-enqueued once the clear made room, and only runs uncounted when the queue is still full Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(logging_worker): build the aggressive-clear run set immutably Final tuple instead of an appended list, with the re-enqueue in a _requeue helper, and Final annotations on the overflow regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): track overflow callbacks through shutdown * chore(logging): explain shared callback ownership * test(logging): isolate deferred callback completion --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-29 09:32:18 -07:00
paveln83	06cdee91da	fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails (#49035) ... * feat(guardrails): scan and rewrite streamed /v1/completions in post_call pipelines Gives OpenAITextCompletionHandler a process_output_streaming_response that assembles the buffered TextCompletionResponse chunks per choice index at end of stream, runs the guardrail over the assembled text, and writes text rewrites back into the first text-carrying chunk of each choice. Tests and live proof still pending. * fix(guardrails): scan and rewrite streamed /v1/completions under post_call guardrails Assemble each choice's streamed text once the stream ends, run the guardrail on it, write a mask back into the buffered chunks, and end the stream with the guardrail's error on a block. User key metadata now goes through the shared metadata bucket so applied guardrails reach spend logs. * test(guardrails): cover streamed /v1/completions under post_call guardrails * fix(guardrails): keep Gray Swan key metadata, scan a cut-short text stream, and pass non-chunk frames through * fix(guardrails): replace blocked text completion streams like chat, keep Gray Swan's vendor payload, scan text streams in the content filter * fix(guardrails): release an unfinished choice's held text and omit usage the upstream never sent The content filter's streaming hook held back the end of a choice until its finish_reason, which litellm sends only for the first choice of an n>1 stream, so choice 1 lost its last characters on text and chat streams. A closing chunk now releases every unfinished choice when the stream ends. A blocked text stream's closing chunk reported 0/0 usage when the upstream sent none; it now carries usage only when the upstream reported it. * fix(guardrails): send no usage on the content filter's end-of-stream chunk * fix(guardrails): keep the usage chunk last and log the masked text of streamed text completions * fix(guardrails): stream Responses API events through the content filter as they arrive * fix(guardrails): treat a null text completion choice index as choice zero when assembling streams --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 08:46:49 -07:00
Mateo Wang	be6d082bdc	fix(anthropic)!: send /v1/messages PDF documents to OpenAI-shaped providers as file parts (#47166) ... * fix(anthropic): send /v1/messages PDF documents to OpenAI-shaped providers as file parts * refactor(anthropic): hold the chat completions document change for a later commit * fix(anthropic): send /v1/messages PDFs as file parts and keep file parts on azure_ai * fix(snowflake): turn file parts into document blocks for Cortex Claude * fix(prompt_security): sanitize chat file parts that carry file_data * fix: answer 400 or drop PDFs for models that cannot read them Models whose model_info or cost map entry sets supports_pdf_input to false, and Ollama, now get the PDF dropped under drop_params and a 400 from litellm without it. GigaChat uploads file parts, SAP keeps sending them as image_url data URLs, azure_ai passes dict content through, and annotations other than url_citation move to provider_specific_fields instead of failing the response. * fix(router): scope supports_pdf_input to its deployment and keep provider annotation keys Mark the azure_ai chat models in the cost map that cannot read PDFs, so drop_params drops the PDF and litellm answers its own 400 without it. Keep url_citation annotations whole, move only typed provider annotations to provider_specific_fields, and split them the same way when a stream is rebuilt. * fix(vertex_ai): send PDF file parts to Vertex partner models as image_url Vertex AI's OpenAI-compatible partner endpoint rejects the chat `file` part, so PDFs from /v1/messages reach those models as the image_url data URL they took before. Mark native DeepSeek chat models as unable to read PDFs, and treat a file part as a PDF by its format or filename when it has no data URL. * fix(cost-map): leave native DeepSeek PDF support unset /v1/messages reaches native DeepSeek through its own Anthropic endpoint, so the marks never applied there and would only change chat completions, which this PR does not otherwise touch. * fix(main): drop a user turn that held only a PDF under drop_params Dropping the PDF from a PDF-only user turn left an empty content list, which Foundry and OpenAI-shaped providers reject with a 400, so drop_params still failed for that request. The emptied user turn is now removed. * fix: read OCI PDFs, gate deployment base models, and stop empty drops OCI generic chat sends file parts as image data URLs again, so OCI Gemini keeps reading PDFs from /v1/messages and chat file parts. The PDF gate now reads model_info.base_model, drops PDFs under drop_params for azure_ai deployments that never declared PDF support (the old flattening behavior), and answers 400 when dropping the PDF leaves no user message. The cost map marks the Vertex partner text models that a live probe showed answering without reading the PDF. * test(integration): cover the PDF input gate, messages PDF file parts and provider annotations Wire cells for the supports_pdf_input check on chat, chat streaming, /v1/messages and the Responses bridge, its model_info, base_model, cost map and Ollama sources, drop_params at deployment and request level, azure_ai file parts, sibling deployment isolation, Anthropic SDK PDFs reaching azure as titled file parts, Gemini and Bedrock document controls, Prompt Security sanitizing a /v1/messages PDF, and OpenRouter file annotations live, cached and streamed * test(integration): cover Snowflake Claude and SAP in the messages PDF wire cells * fix(azure_ai): flatten non-PDF file parts to text as before and cover the GigaChat PDF upload * fix(main): PDF check reads the deployment's top-level base_model * fix(main): keep roles alternating after drop_params removes a PDF-only turn When ensure_alternating_roles is set, re-run role normalization on the messages left after the PDF check drops a turn, so two assistant turns never end up next to each other * test: type the PDF and annotation tests the PR added * test: type the GigaChat, SAP and Snowflake PDF tests	2026-10-29 08:06:12 -07:00
Priyansh Nandwana	67ae3fb527	fix(websearch)!: keep the provider prefix on the agentic follow-up model (#38885) ... * fix(websearch): keep the provider prefix on the agentic follow-up model The agentic follow-up rebuilt its model string with "a slash means it is already qualified". Handlers are handed the provider-stripped model, and for a provider that routes through a sub-path the remainder still holds a slash, so bedrock/mantle/anthropic.claude-sonnet-5 arrived as mantle/anthropic.claude-sonnet-5 and stayed that way. The follow-up acompletion could not resolve a provider from it and raised "LLM Provider NOT provided". Prefix unless the model is already qualified with that same provider, in one shared helper used by both the handler and the websearch patch builder, which carried copies of the same heuristic. This also fixes openrouter, whose stripped model keeps a slash for the same reason, and the old startswith test that read openai as a prefix of openai_like. Fixes #38829 * test(websearch): cover the agentic follow-up call site The helper had unit tests but the call site did not, so the two changed lines in _execute_chat_completion_agentic_plan were uncovered. Drive the real follow-up with mock_response instead of patching litellm, which keeps the test on the behaviour: against the old code the sub-path case raises "LLM Provider NOT provided", and the ordinary-model case passes either way as a guard. * refactor(websearch): trim the comments on the provider-qualify helper * refactor(websearch): cut the helper and test docstrings to one line * fix(websearch): leave a hook-qualified follow-up model on its own provider A hook that patches in its own model owns that choice, so re-qualifying it sent a cross-provider follow-up to the original request's provider instead. Only a bare patched model and the request's provider-stripped model take the prefix now, and the sibling agentic-loop path resolves the model the same way. * fix(websearch): fall back to the request model when a patch carries no model An empty patched model dispatched a provider-only name instead of falling back, and a provider-stripped sub-path copied into the patch lost its prefix. A patched model now keeps its own provider only when its leading segment names one. * fix(websearch): keep the provider prefix on the responses follow-up model too The responses patch builder still used the old "a slash means already qualified" test, so a provider-stripped sub-path model reached the follow-up unqualified on that surface. It now resolves the model the same way the chat completion builder does. * fix(websearch): read custom_llm_provider without a member call on untyped kwargs * fix(websearch): type the chat patch builder kwargs instead of working around them * test(websearch): run the real search path and flatten the qualify helper tests * fix(websearch): route a hook's model on a JSON-configured provider and keep the request provider on an echoed model * fix(websearch): leave a follow-up model alone when its prefix already routes to the request provider * fix(websearch): send the agentic follow-up to the model the caller requested completion() and responses() record the caller's model before routing strips it, and the chat and Responses follow-up builders prefer that model. This keeps bedrock/converse_like/, bedrock/claude_platform/ and openrouter/openrouter/* deployments on their own route for the follow-up. * test(websearch): cover the recorded caller model in Headroom's retrieval follow-up * test(websearch): cover the recorded caller model in Compresr's retrieval follow-up Also merges the repeated provider-prefix check in core_helpers into one helper and drops a test docstring * test(websearch): drive the real search path through a mocked Perplexity HTTP route * refactor(websearch): read the provider from kwargs inside _caller_model * test(http-handler): type the agentic follow-up provider tests * refactor(websearch): record agentic loop params through a Logging method Moving the recording out of completion() keeps it under basedpyright's code-flow complexity limit after the merge with main. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-29 04:47:18 -07:00
simonp9	0f68b9919c	fix(guardrails): skip embeddings input in google_text_moderation (#48934) ... * fix(guardrails): skip embeddings input in google_text_moderation and openai_moderations Since v1.103.0 the proxy dispatches these legacy moderation callbacks on every route, and they read the embeddings input field as text, so benign embedding inputs were blocked. Return early for embedding call types, matching the Aim and Cato guardrails. * test(guardrails): type the embeddings moderation test parameters * fix(guardrails): keep openai_moderations moderating embeddings, skip only google_text_moderation Google's moderation scores topic categories like Finance and Health, so it blocks documents being indexed by subject. OpenAI's moderation scores harm categories only, so it keeps moderating embedding input as it has since v1.103.0 * fix(guardrails): skip Gigachat passthrough embeddings in google_text_moderation Treat an allm_passthrough_route call whose endpoint ends in embeddings as non-conversational, log the skip, return None from the early exit, and type the hook's data so the basedpyright gate does not grow * fix(guardrails): match only plain embeddings passthrough paths in google_text_moderation --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:05:03 -07:00
mika-lt	4e5f585c2a	fix(guardrails): scan Responses tool results in texts-mode guardrails (#48827) ... * fix(guardrails): scan Responses tool results in texts-mode guardrails * fix(guardrails): keep guardrail-added tools and non-text parts on Responses tool scans Tool result texts now come from the same conversion that builds the chat tool message, so items without a call_id, dict outputs, and outputs with no text parts stay aligned for Bedrock and PANW. A changed joined text replaces the first text part and keeps the other parts. scan_only_tool_results appends tools the guardrail returns with new names, like chat, and tool_result and computer_call_output items are scanned through the shared type list. * fix(guardrails): skip empty and screenshot tool outputs and keep text part types on Responses scans * fix(guardrails): keep Bedrock latest-turn masking aligned when tool outputs send no text * fix(guardrails): scan each Responses tool output text part on its own * fix(guardrails): split a Responses tool output's structured row when only textless outputs share its call_id * fix(guardrails): pair each Responses tool output with its own structured row --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-28 20:01:07 -07:00
yucheng-berri	c6ca22ebc5	fix(guardrails)!: make hide-secrets honor skip_tool_message_in_guardrail (#48689) ... * fix(guardrails): make hide-secrets honor skip_tool_message_in_guardrail With the flag on, per guardrail or in litellm_settings, hide-secrets leaves role: tool messages and Anthropic tool_result blocks as sent, as the unified guardrails do. Tool call arguments are still masked. * test(guardrails): type the skip-tool walker test and assert the whole request	2026-10-28 14:49:48 -07:00
yucheng-berri	15eca24fc2	fix(guardrails)!: redact secrets in tool traffic and system prompts (#47122) ... * fix(guardrails): redact secrets in tool traffic and Anthropic system * fix(guardrails): bound tool payload scans and stop nested tool_result recursion Scan each tool value beside its key instead of an indented dump, read tool_result content one level deep, keep non-finite tool arguments as raw text, and give read-only guardrails one fragment per tool payload * test(guardrails): cover read-only guardrail consumers of tool and system text * fix(guardrails): keep tool argument parsing, scanning and tests within bounds Treat tool arguments that decode to a lone surrogate as raw text, decode JSON-escaped detections back to the leaf value, quote unkeyed leaves without repeating a parent key per list item, drop cycle tracking that parsed JSON cannot reach, and use the CI license instead of a fake license server * fix(guardrails): scan list items beside a capped parent key List items keep their parent key in the scanned JSON line, capped to its last 64 characters, so a credential key over a list is still detected while a long key over many items no longer multiplies the scan text * fix(guardrails): cap the scanned key after JSON escaping A key of control characters escaped to six times its length after the cap, so every list item's scan line grew with it * fix(guardrails): collapse whitespace in the scanned key context Space-heavy keys made every inherited list item scan several times slower than a short key * fix(guardrails): build each key's scan context once per payload Repeating the collapse, escape and cut for every list item made a long key over many items cost key length times item count * fix(guardrails): cover Responses tool call items and instructions Responses carries the same tool call arguments and system prompt as chat and messages under function_call.arguments, custom_tool_call.input and instructions, so secrets there reached the provider unscanned * perf(guardrails): skip rescanning text hide-secrets already found clean Agents resend their whole conversation on every turn, so each turn rescanned every earlier tool result. Each guardrail now keeps a bounded set of digests of scan inputs that held no secret and skips those; text with a secret is always rescanned and no secret material is kept * fix(guardrails): size the clean-scan cache for many sessions and replace longest secrets first 4096 digests held about 20 long agent sessions per worker, so interleaved sessions evicted each other and every resend rescanned its history; 65536 digests (about 10 MB) hold a few hundred. Tool payload secrets are now replaced longest first after JSON unescaping, so a shorter overlapping match cannot leave part of a secret behind. The key-context test checks the work it bounds instead of wall-clock time, which CI could not meet * fix(guardrails): keep unmasked tool argument text and redact keyword matches cut inside an escape * fix(guardrails): write a tool visitor's whole payload when it changes more than string values * fix(guardrails): read message text before tool and system text so a vendor error on the new text cannot skip it * fix(guardrails): moderate tool and system text in its own call so a long payload cannot hide a flagged message * fix(guardrails): skip the empty moderation call when only tool or system text is present * fix(guardrails): scan long multi-line tool strings as their own lines so the scan stays linear * fix(guardrails): keep the key beside every line of a long multi-line tool string * fix(guardrails): confine tool-traffic coverage to hide-secrets Restore iter_message_text to its main behavior, so banned keywords, Azure Content Safety, IBM detectors and the OpenAI and Google moderation hooks read exactly what they read on main. Only walk_user_text, which hide-secrets alone calls, still reaches system, instructions and tool payloads. * test(guardrails): stop the tool-traffic integration tests from flaking on a busy host Use the shared per-case timeout for the rig client, and open a new connection per call in the worker-kill burst so a pooled keep-alive socket to the killed worker is never reused. * test(guardrails): drop the wall-clock check from the deep tool arguments unit test The integration test test_deeply_nested_tool_arguments_do_not_stall_the_worker already covers the timing; the unit test keeps the redaction check. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-27 18:11:03 -07:00
Mateo Wang	0a1dbc0fb4	refactor(mcp): drop the experimental label from the MCP server and client paths (#48513) ... * refactor(mcp): drop the experimental label from the MCP server and client paths Move litellm/proxy/_experimental/mcp_server to litellm/proxy/mcp_server and litellm/experimental_mcp_client to litellm/mcp_client, with the mirrored tests moved under tests/unit/proxy/mcp_server and tests/unit/mcp_client. Both old packages stay as shims that warn with DeprecationWarning, re-export the new package's public names and alias every submodule in sys.modules so deep imports keep resolving to the same objects. The OpenAPI docs gate now puts the merge base's enterprise package on PYTHONPATH so its base counts do not mix the checked-out enterprise code with the base litellm tree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(mcp): keep tests/unit/proxy/_experimental as a sharded root for the shim tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): alias namespace subpackages and bind attributes in the deprecated MCP shims pkgutil.walk_packages skipped mcp_server/auth because it has no __init__.py, so old imports of litellm.proxy._experimental.mcp_server.auth.* failed, and sys.modules entries alone left litellm.experimental_mcp_client.tools unreachable as an attribute. Walk the package on disk and setattr each alias on its old parent. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 17:47:50 +00:00
Mateo Wang	14193435b8	refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510) ... * refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages Move the unified /v1/messages code out of a directory called pass_through, since it is not the passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner model experimental_pass_through package becomes messages. The old packages stay as import shims that warn once and alias every submodule in sys.modules, so old deep imports still resolve Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): keep Final variables out of loops in the rename shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): match the warning text in the rename shim test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): resolve old pass_through imports lazily through a meta path finder Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): mark the alias loader spec restore for the type discipline gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): format the renamed package shim Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(anthropic): keep the rebind-ok marker on the spec restore line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(anthropic): carry the new package search path on alias specs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): declare the shim test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-27 10:08:25 -07:00
mrinal-berri	07a82a287b	fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276) ... * fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text The pass-through guardrail translation json-dumps the request body as the scan text when no request_fields are configured. The unified guardrail hook injects litellm_metadata into that body first, so the dump carried the caller's key metadata. Exclude it alongside metadata and litellm_logging_obj; vendors still receive it through request_data. * test(guardrails): type the pass-through scan-text tests per AGENTS.md	2026-10-26 12:07:58 -07:00
devin-ai-integration[bot]	1418c1990f	feat(guardrails): add native Reco guardrail provider (pre_call), fail closed by default (#48243) ... * feat(guardrails): add native Reco guardrail provider (pre_call) Registers guardrail: reco as a first-class vendor integration, built on the existing generic guardrail API wire contract rather than reimplementing the HTTP client and payload mapping. Maps reco_tenant_id to the X-Reco-Tenant-Id header and hardcodes fail_open/fail_on_error since those should not be customer-configurable. v1 only supports mode: pre_call; post_call and streaming redaction already work against the existing generic pilot and are deferred. Also overrides apply_guardrail to re-raise blocked-request errors with the guardrail's own configured name: the base class hardcodes a module-level name in that exception, which would otherwise mislabel every blocked Reco request in logs. * feat(guardrails): configurable Reco fail modes Expose unreachable_fallback and fail_on_error on the Reco guardrail and keep fail-open as the default. Values under optional_params win. Top-level litellm_params values apply only when set explicitly, since their defaults are fail-closed and would otherwise flip the default Also type RecoGuardrail's forwarded kwargs with an Unpack'd TypedDict instead of silencing ANN003 Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Cursor <cursoragent@cursor.com> * chore(guardrails): align Reco with main's CI gates Upstream moved unit tests from tests/test_litellm to tests/unit, so test_reco.py was invoked by no CI job. Move it next to the other guardrail tests Replace RecoGuardrail's forwarded kwargs with explicit keyword params, read optional_params through a pydantic adapter instead of untyped dict/getattr access, freeze RecoOptionalParams, and drop mutable-ok markers that suppress nothing, to stay within main's type-discipline and basedpyright budgets Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Cursor <cursoragent@cursor.com> * feat(guardrails): fail closed by default in the Reco guardrail Matches GenericGuardrailAPI and every other guardrail here: a Reco error or outage now blocks the request unless the admin sets fail_on_error: false. Tests cover the default, explicit fail-open, and top-level vs optional_params precedence Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): honor explicit fail-open on Reco when optional_params is a model model_dump(exclude_unset=True) so pydantic defaults in optional_params no longer override an explicit top-level fail_on_error. Also drop the duplicate log_guardrail_information decorator, call the parent by keyword, freeze RecoConfigModel, and type the tests Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep Reco config model assignable and the hook decorator the gate requires frozen=True on RecoConfigModel froze the shared LitellmParams mixin, which the guardrail registry assigns resolved secrets on, so every guardrail failed to load. Dropped it and added a test that LitellmParams stays assignable. Restored log_guardrail_information on the Reco override since tests/code_coverage_tests/check_guardrail_apply_decorator.py requires it on every hook. Test params typed as Mapping for LIT001 Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(guardrails): format the Reco test file Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Yaron Bratspiess <yaronb@reco.ai> Co-authored-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-24 15:33:17 -07:00
Mateo Wang	464932b1d1	refactor(typing): clear 1181 basedpyright Any errors across 431 files (#47789) ... * refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1) * refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2) * refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts * fix: restore pass-through behavior on paths the typing pass tightened MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites * fix(runwayml): keep a malformed 2xx task body on the 500 path * fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report EdenAI's embedding and image generation parse dropped a non-object body from the error they raised, so the message no longer named what the provider sent. A helper now raises with the body in the message, and both paths have a regression test. `lite encryption migrate` exited 0 with no message when the proxy's report was not an object or its residual_legacy was not a number, where base crashed. It now fails with a message naming the value. The CompactifAI and Gemini image-edit non-object body tests accept TypeError or AttributeError, which both base and this branch raise, so they guard against a ValidationError instead of pinning one class. RunwayML's async parse gets the same non-object body test as the sync one. * fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed * test: annotate the new regression tests' locals with Final * test: annotate the remaining new test locals with Final	2026-10-24 11:31:09 -07:00
Mateo Wang	a8113b0dd7	fix(responses)!: round-trip Anthropic compaction blocks as compaction items (#46754) ... * fix(responses): round-trip Anthropic compaction blocks as compaction items /v1/responses on Anthropic routes now returns Claude's compaction block as a compaction output item (and streams it as output_item.added/done), and a compaction input item is replayed to Claude as its native compaction block, so the next turn reuses the summary instead of compacting again. Other routes receive the summary as assistant text. The Anthropic stream handler now keeps the summary text that arrives in compaction_delta. Fixes #41456 * fix(responses): replay compaction blocks natively only when the request compacts * fix(responses): carry compaction through dict-form edits, SDK replay, affinity, and guardrails Read Anthropic's dict-form context_management, accept SDK ResponseCompactionItem objects on replay, strip the encrypted_content affinity wrap before decoding, scan and rewrite compaction summaries in Responses guardrails, drop the history a summary replay replaces, send the summary as a user message so thinking ordering holds, and send the streamed compaction .done with the newest block. * fix(responses): count, route, guard, and redact compaction items consistently A compaction item now reads as its summary everywhere the proxy counts or routes a Responses request: pre-call context checks, the TPM limiter, the input token endpoint, the stream usage estimate, and the previous_response_id session rebuild all drop the history the item replaces. A Claude compaction item sent to a non-Anthropic model (including a fallback) becomes a user summary message instead of an encrypted blob the provider rejects, and Claude on Bedrock Claude Platform takes the summary path. Output guardrails see the summary of a returned compaction item, and logging redaction hides it. * fix(responses): start previous_response_id replay at a stored compaction and count reservations from it * fix(responses): keep a stored compaction summary whole in spend logs * fix(responses): keep stored compaction items summary-only, redact compaction_delta, and finish compaction before reasoning A spend-log row now stores a top-level compaction input item as its summary alone, so extra keys in a client-built item no longer reach the database whole. Message redaction also nulls the streamed compaction_delta, and the streaming bridge sends the compaction item's done event before the reasoning item that follows it. Adds the audit's integration tests for compaction items on the wire. * test(responses): cover stored compaction items, compaction_delta redaction, and compaction done order Adds integration cells for the summary-only spend row of a caller-sent compaction item and its previous_response_id replay, a redacted chat stream row that holds no streamed compaction_delta, and a compaction then thinking stream whose done events follow output order. * fix(spend_logs): honor the redaction header on failure rows for litellm_metadata routes A failed /v1/responses or /v1/messages call keeps its proxy request headers in litellm_metadata, but the failure row read redaction headers from metadata only, so x-litellm-enable-message-redaction never redacted those rows. * fix(spend_logs): lift only the redaction opt-in header into failure rows The failure hook on litellm_metadata routes copied every real request header into the caller's own metadata dict, so the stored request body gained the headers, a caller's metadata.headers was replaced, and a disable header loosened global redaction on failed calls. It now copies only the enable headers into a new litellm_params metadata dict, and only when the caller sent no metadata.headers, so no failure row is less redacted than on main * test(responses): cover failure-row redaction headers and caller metadata on litellm_metadata routes * fix(responses): keep earlier-turn instructions after compaction and count compaction items by their summary A previous_response_id rebuild that starts at a compaction now keeps the system and developer messages of every earlier turn. Token estimates for raw Responses input count a compaction item as its summary text instead of the serialized item. * fix(responses): drop the stored history a caller-sent compaction item replaces on previous_response_id * fix(responses): count only message items with a string role as compaction instructions * test(responses): flatten the last two-for comprehension in the compaction wire test * fix(responses): replay from a compaction in stored Responses output and SDK compaction items A previous_response_id whose stored row is a Responses output holding a compaction item now starts its replay at that compaction, and the handler dumps SDK compaction objects before checking for a summary. Adds tests for the uncovered compaction paths. * fix(responses): keep the stored tool call a function_call_output answers when a compaction item replaces history * fix(responses): keep a stored tool call on every turn after a compaction item and bound the stored summary A previous_response_id rebuild that starts at a turn holding a compaction item now sends again each stored call a replayed tool output answers, so a retry or a later turn no longer loses the turn-1 call. The start is picked only from rows up to the referenced response. A spend row keeps a compaction summary up to the larger of 64K chars and MAX_STRING_LENGTH_PROMPT_IN_DB instead of with no bound. * fix(responses): replay a compacted session only through the referenced response A later sibling turn that sent a compaction item put its summary between a stored tool call and its tool output, so Claude answered 400 on a turn that continued the first of two retried turns, and on a retry sent without the item * fix(spend-logs): keep main's stored value for a prompt length limit under 2 At a MAX_STRING_LENGTH_PROMPT_IN_DB of 0 or 1 the shared truncation helper stored only the truncation marker, where main stores the marker followed by the whole value * fix(spend-logs): keep main's stored tail for a negative prompt length limit At a negative MAX_STRING_LENGTH_PROMPT_IN_DB the shared truncation helper stored one more character of each value than main, since it dropped main's clamp of the end slice to the limit minus the start * fix(responses): keep a compaction item when a guardrail rewrites its summary through structured_messages * test(responses): run the compaction wire tests on the worker database under isolated workers * fix(router): keep the encrypted content affinity pin out of provider requests * fix(proxy): inspect compaction summaries in guardrails, strip a client-sent affinity pin, and price failures from the newest compaction * fix(proxy): count a failed chat or /v1/messages call on its whole history, not from a compaction-typed message * fix(proxy): trim a failed call's input at the newest compaction only when the client called /v1/responses * fix(proxy): trim a failed call's input at the newest compaction when the proxy dispatched it as a Responses call * test(proxy): run the dispatched-call-type failure test through the real request setup * fix(proxy): count a failed Cursor stream's input from the newest compaction item The Cursor route maps to no call type, so the stream generator's failure hook fell back to the route and counted the whole history. Pass the processor's route_type. * fix(proxy): count a broken Cursor stream's prompt from the newest compaction item * fix(responses): keep a tool call a later output answers when trimming before a compaction item * fix(responses): move a compaction-split tool call's turn to right before its output on both compaction routes * perf(responses): look up moved turn items in a set so a long compaction-split turn trims in linear time * test(responses): measure the compaction-split turn move as a CodSpeed benchmark instead of a clock read in unit tests * fix(responses): move only calls still open at the compaction, and bill a disconnected stream from the newest compaction item * fix(responses): restore only the newest stored call when a reused call_id is answered after a compaction item * fix(spend logs): bound a Responses compaction item's summary like a chat block instead of cutting its encrypted_content * test(proxy): price a failed dispatched call through a real ProxyLogging instead of mocked hooks * fix(responses): finish a stream whose Claude compaction failed and group answered call ids by turn once * fix(responses): count input_tokens after a compaction item the way inference sends it	2026-10-21 08:50:19 +00:00
alberto	f768fb32ef	fix(streaming)!: send the include_usage chunk with an empty choices array (#47109) ... * fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-21 02:26:05 +00:00
Mateo Wang	4b1928387f	test(bedrock): detect Vault reads on the event-loop thread (#47597)	2026-10-20 10:45:04 -07:00
Mateo Wang	f3513bca9e	fix(bedrock): resolve AWS region and secret lookups off the event loop (#47331) ... * fix(bedrock): resolve AWS region and secret lookups off the event loop Bedrock request paths that resolve their region, endpoint or bearer token inside the async handler now do it in the AWS signing executor, so a slow secret manager read no longer stalls every other request on the worker * fix(bedrock): type the STS client kwargs and claude platform environment override Keeps the basedpyright counts at or below the merge base after the off-loop wraps, by typing the STS client kwargs as a TypedDict, the claude platform environment override with concrete dicts, and the realtime region call through the typed getter * test(bedrock): detect event loop stalls without wall-clock timing The Vault fake now answers a read only after the test loop runs a callback, so a read made on the loop thread is caught deterministically instead of by a sleep and a heartbeat threshold	2026-10-20 17:04:41 +00:00
Charles	5a1fa1c879	fix(responses): keep strong refs to stream and websocket logging tasks (GC race) (#31074) ... * fix(responses): keep strong refs to stream and websocket logging tasks * test(responses): patch _spawn_logging_task instead of asyncio.create_task The success-logging dispatch in _log_completed_response now goes through _spawn_logging_task, so tests that patched asyncio.create_task no longer observed it (and the sync ones hit get_running_loop with no loop). * test(responses): assert stream and websocket logging tasks survive garbage collection Adds regression tests that force a collection while the logging task waits on an otherwise unreferenced future, and types the new test parameters * test(responses): drive drain and GC tests through a real logging object with gated callbacks * test(responses): use asyncio.wait_for so the logging tests run on Python 3.10 * test(guardrails): type the spawn_logging_task tracking wrapper --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 09:26:25 +00:00
Lai Wei	bda4e60f9b	fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions (#44349) ... * fix(streaming): keep finish_reason and provider usage when rebuilding streamed text completions stream_chunk_builder_text_completion read finish_reason from chunks[-1] and recounted usage from messages. With include_usage the last chunk is a usage-only trailer, so finish_reason came back None (or raised IndexError when the trailer had no choices), and prompt_tokens was 0 because a text-completion prompt is not in messages. Scan for the last finish_reason and prefer the provider-reported usage, as the chat path already does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): type the text completion chunk builder so its scans add no basedpyright debt Read chunk fields as typed attributes instead of untyped subscripts, which clears the reportUnknownArgumentType breach the lint gate flagged * fix(streaming): keep the provider's full usage, including token details, when rebuilding text completions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): rebuild dict-form text completion chunks again stream_chunk_builder routes dict chunks holding TextChoices to the text completion builder, which now reads attributes. Convert them to TextCompletionResponse at the routing point. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(streaming): accept dict chunks in the text rebuild and derive its usage like the chat path Moves the dict to TextCompletionResponse conversion into stream_chunk_builder_text_completion so direct callers keep working, and builds usage through ChunkProcessor.calculate_usage over the chunks that report counts, so a trailer without total_tokens keeps its prompt and completion counts and the result is a fresh Usage * fix(streaming): read text completion usage through the shared usage extractor so usage dicts on typed chunks rebuild * fix(streaming): merge text completion logprobs across chunks and rebuild dict text chunks as text completions * fix(streaming): rebuild one text completion choice per index and keep null logprobs fields null * fix(streaming): keep dumped chat chunks with a choice-level text on the chat rebuild path Route a dict chunk to the text completion builder only when its choice has `text` and no `delta`. The Responses bridge snapshots chat chunks with model_dump(), and Together's chat choices carry `text` next to `delta`, so streamed /v1/responses calls to Together ended without response.completed. Group text completion choices by index in one sorted pass, so rebuilding an n>1 stream costs about the same as rebuilding an n=1 stream of the same size. * fix(streaming): keep text completion rebuilds working on mixed chunks and unreadable logprobs A chat chunk inside a text completion stream now converts to a text chunk that keeps its finish_reason and adds no text. A dict chunk whose logprobs fail validation is read again with logprobs dropped, and logprobs that are not a Logprobs object keep the last value sent. Each of these inputs raised or lost data at the previous commit, and each now rebuilds without raising Also adds a Purview test showing a multi-choice text stream is blocked like the non-streamed response, and Final annotations on the new test locals * fix(streaming): rebuild text completion choices without an index at index 0 Text completion choices whose index is None now rebuild under index 0, as on the merge base, where the sort by index raised at the previous commit. A chat chunk inside a text completion stream now keeps the usage it reports, and a choice whose logprobs mix a Logprobs object with another shape keeps the last value sent, as on the merge base * fix(streaming): rebuild text completion choices with a string index under that int A text completion choice whose index is a string now rebuilds under the int it names, and one whose index cannot be read as an int rebuilds at index 0, as on the merge base. A string index mixed with an int index raised TypeError at the previous commit, and a string-only index came back as a string * test(streaming): drop a docstring that repeats its test * fix(streaming): fall back to choice 0 for an infinite text completion index A float infinity index made int() raise OverflowError, which the index coercion did not catch, so the rebuild failed where the merge base returned choice 0. --------- Co-authored-by: Lai Wei <8022184+roywei@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-20 05:31:39 +00:00
joshua-berri	9c958aff34	fix(logging): deliver completed batch success callbacks consistently (#47192) ... * fix(batches): dispatch configured success callbacks during reconciliation * fix(batches): retain billed job claim when sync dispatch shuts down * fix(batches): finish reconciliation when logging executor is stopped * fix(logging): avoid duplicate custom hooks during async batch dispatch * fix: preserve sync-only custom callback delivery * fix: scope callback deduplication to per-call async recipients * refactor: type per-call async success callbacks * refactor: retain logging constructor callback compatibility * test: support current batch reconciliation metadata * fix(logging): honor per-logger redaction during sync success delivery * fix(logging): centralize completed success callback dispatch * fix(guardrails): avoid duplicate success callback dispatch --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>	2026-10-18 09:19:04 -07:00
ryan-crabbe-berri	f6ae25939e	feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965) ... * feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries * fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5 A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow * refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass * chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py	2026-10-17 12:22:41 -07:00
yucheng-berri	223800f916	fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024) ... * fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130) Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main with the unit test moved to tests/unit after the test tree relocation * fix(presidio): label an overlapping run by its outermost detection, add integration coverage A run of overlapping detections now takes its entity type from the highest-scoring detection that no other detection in the run contains, matching Presidio's own anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER. The resolver returns typed detections instead of Any. * fix(presidio): drop the redundant comment in the numbered token loop * fix(presidio): bump a numbered token another part of the request already holds Each message and content part numbers its tokens from 1 into one request-wide pii_tokens map. Collapsing overlapping detections made an overlapping value's token well formed, so it collided with the same type in another part and both restored as the later value. A token that maps to a different value now moves to the next free number, and a token for the same value is reused * fix(presidio): give a clashing token a spare number instead of scanning for one Scanning upward from the clashing number probed every token an earlier part had claimed, so two parts with thousands of values of one type took seconds of synchronous work. No token in the map is numbered above the map's size, so numbers above both that size and the part's run count are always free * test(presidio): make the anonymizer double reject detections the analyzer never returned Pick the token in one conditional expression instead of reassigning it, and have the scripted anonymizer answer 500 when the proxy forwards spans that differ from the analyzer's, so the default-scope cells check what was sent --------- Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 11:21:03 -07:00
devin-ai-integration[bot]	46ae68021e	fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096) ... * fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events When the proxy renders a complete Anthropic Messages response as an SSE stream for a client that asked to stream, the message_delta event carried only stop_reason and stop_sequence. A real Anthropic stream puts safeguard_results and stop_details inside message_delta.delta, so a client such as Claude Code auto mode lost the server-side classifier verdict and a refusal lost its stop_details. Both stream builders now copy those keys from the source response through one shared helper when they are present. * fix(guardrails): keep safeguard_results and stop_details when re-emitting a rewritten Anthropic stream A post-call guardrail that rewrites an Anthropic stream (presidio, bedrock, tool permission, model armor) now carries the upstream message_delta safeguard_results and stop_details into the re-emitted stream. The router fallback message_start no longer leaks those final fields early * fix(guardrails): keep stop_details and safeguard_results in rewritten Anthropic streams without explanations A guardrail that rewrites a /v1/messages stream now carries stop_details (type and category) and the dangerous_tool_use verdicts for the tool calls it still delivers, with every explanation dropped so a masked stream never echoes the unmasked text through those fields. * fix(guardrails): copy only upstream stop_details and forward-compatible safeguard_results into rewritten Anthropic streams A rewritten stream now takes stop_details and safeguard_results from the upstream message_delta alone, so a refusal the upstream sent without stop_details is never given one by the OpenAI round trip. Unknown safeguard types, statuses, reasons and extra fields pass through, a verdict follows a tool call only when its input is unchanged, and tool_permission keeps the explanations since it rewrites no text. * fix(guardrails): apply the safeguard_results and stop_details rules to policy pipeline stream rewrites * fix(router): deliver a refused stream built from a finished agentic turn instead of falling back Websearch, MCP and advisor streams replay a complete reply whose tools already ran, so a refusal fallback would run every tool again. Also give the recovered response's message_start a null stop_details as the API sends it, drop safeguard_results a masking rewrite cannot read, and add the audit cells for synthesized streams. * test(router): annotate the finished-turn refusal stream test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>	2026-10-17 16:48:37 +00:00
yucheng-berri	ef1cb19351	fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307) ... * test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-17 00:53:53 -07:00
Mateo Wang	b39eadcb82	feat(openai)!: honor per-deployment workload identity on non-chat routes (#39966) ... * feat(anthropic): workload identity federation and pluggable identity sources Backend half of #38818 (internal copy of the fork PR #38013), rebuilt as one commit on top of litellm_internal_staging without the dashboard changes. Deployments on anthropic/ without a static api_key can exchange an OIDC workload assertion for a short-lived sk-ant-oat01 token through a shared RFC 7523 JWT-bearer engine. The assertion comes from a mounted token file, an env token, a LiteLLM-signed issuer, or Keycloak, chosen per deployment, per named credential, or through ANTHROPIC_IDENTITY_SOURCE. The federation fields are server-owned: refused inline in request bodies and on POST /model/new, proxy-admin only on credentials, and the token exchange is pinned to api.anthropic.com unless LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS adds a host. GET /credentials/{name}/jwks exports the public key set of a LiteLLM-signed credential for the Claude Console. The OpenAI federation trio from #39613 rides along on the backend side with the same server-owned handling. Fixes #28607 Resolves LIT-6107 Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com> * fix(anthropic): let batch-result downloads mint from deployment params and accept host:port allowlist entries The files handler enabled workload identity on batch-result downloads but never received the deployment's litellm_params, so a deployment authenticating through a named credential could only mint from process-wide env vars. It now threads litellm_params through to the auth header the way the batch retrieve path already does. LITELLM_ANTHROPIC_WIF_ALLOWED_HOSTS entries written as host:port were read by urlsplit as a scheme, so the allowlist kept the raw entry while the exchange compared bare hostnames and refused the gateway. Entries are now parsed as network locations whether or not they carry a scheme. * fix(types): move the WIF kwargs key sets to a leaf module so the kwargs funnel imports without a cycle * test(anthropic): pin case-insensitive matching of WIF exchange-host allowlist entries * fix(anthropic): end workload identity federation errors without a period so the router suffix reads cleanly * fix(proxy): decrypt stored litellm_params before the WIF write gate * fix(proxy): hide WIF secret references from /health output * fix(proxy): keep the proxy error shape on credential endpoint refusals * fix(proxy): hide identity token file paths from /health output * feat(openai): honor per-deployment workload identity on non-chat routes Image generation, audio speech and transcription, moderation, text completions, files, batches, and fine-tuning built their OpenAI client from the env vars alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file failed with the api_key error on every route other than chat, Responses, and embeddings. Those handlers now build their client through the shared workload identity resolver, and the managed batch file-access credential extraction keeps the trio * fix(openai): cache workload identity clients and let deployment identity outrank process-wide keys Non-chat handlers built a fresh SDK client per call, so files, batches, fine-tuning, text completion and sync moderation exchanged the OIDC token on every request. The builders now reuse the cached workload identity client per identity, base, timeout, retries and organization; static-key clients are still built fresh. A process-wide OPENAI_API_KEY (or litellm.api_key / litellm.openai_key) no longer silently overrides a deployment's openai_identity_* params. An explicit distinct api_key on the deployment or call still wins. * fix(openai): keep shared workload identity clients on the process http session The workload identity client is cached and shared across requests, so it must never borrow the transport a single caller passed in. Static-key clients keep using the caller's http client; federated clients always ride the process-wide session. Also makes the http client helpers on BaseOpenAILLM public since the builder now needs them from outside the class. * test(openai): pin partial deployment identity beating the process key * test(openai): type the managed-object invoke callback * fix(anthropic): rename the federation workspace param so Bedrock's anthropic_workspace_id keeps working The Bedrock Claude Platform route already reads anthropic_workspace_id from optional_params, so banning that spelling as a server-owned federation parameter broke a pre-existing client capability. The federation field is now anthropic_federation_workspace_id (env ANTHROPIC_FEDERATION_WORKSPACE_ID), which restores the base branch's behavior for Bedrock callers, drops the Bedrock-specific hint from the refusal message, and deletes the unconditional ban constant that no longer had a reader * fix(auth): share one exchanged token across workers reading the same assertion Anthropic accepts each identity assertion exactly once, so two uvicorn workers reading the same token file both minting from it means the second exchange is denied with jti_reused. Minted tokens now land in a per-user 0700 cache directory guarded by a file lock, so workers on the same host reuse one exchange until the token expires or the assertion rotates. A 401 is only retried when the re-read assertion actually differs, and the denial hint explains jti_reused. LITELLM_TOKEN_EXCHANGE_CACHE_DIR moves the cache and an empty value disables it * feat(openai): honor deployment workload identity on image edits, vector stores, containers, videos, and evals The httpx-based OpenAI handlers built their Authorization header from the deployment api_key or OPENAI_API_KEY alone, so a keyless deployment carrying openai_identity_provider_id, openai_service_account_id, and openai_identity_token_file sent "Bearer None". They now resolve the bearer through resolve_openai_bearer_token, which exchanges the deployment's identity token the same way the SDK-based routes do. A deployment that carries only part of the identity trio now raises a 500 naming the missing field instead of silently falling back to the process key, and the SDK client builders take a factory for the static-key http client so the federated branch never touches litellm.client_session. * fix: keep anthropic federation from being shadowed or leaked An empty or whitespace-only ANTHROPIC_API_KEY counted as set, so a federated deployment sent an empty x-api-key on every call instead of minting a token. Blank values now read as unset, and a real static key on a federated deployment logs once that it outranks federation and nothing is being federated. The exchange-host allowlist matched hostnames only, so a second process on another port of an allowed host was trusted with the workload's identity token. An entry that names a port now trusts that port alone, while a bare host still trusts every port. The shared token store exists so the workers reading one projected token file do not each spend its single-use jti. A source that mints its own assertion per exchange shares nothing with another worker, so it no longer writes a live token to disk for a lookup that can never hit. * feat(openai): resolve per-deployment workload identity for the assistants routes OpenAIAssistantsAPI built its OpenAI and AsyncOpenAI clients from a locals() scrape, so a deployment carrying openai_identity_provider_id, openai_service_account_id and openai_identity_token_file never exchanged a token. Thread litellm_params through the class and build both clients with the shared workload identity builders. * fix: unlink a staged token file a failed write leaves behind The 401 denial hint now also says federation ignores ANTHROPIC_WORKSPACE_ID, which the Bedrock Claude platform provider already reads. * refactor: move anthropic jwks derivation behind a provider-owned tagged union * feat(openai): resolve per-deployment workload identity for image variations Build the image variations client through the shared workload identity builders instead of scraping locals(), and give the async path the async http session it was missing. Let a federation trio serve the model-less batch fallback, and hold the standard LRU size of exchanged tokens. * fix: unlink the staged token file when its write fails at close A buffered write only reaches the disk when the handle closes, so a full disk surfaces at close and left the staging file behind holding a usable token. * fix(anthropic): close the staging descriptor before writing the shared token file * fix(wif): judge federation writes by what they set, not what is stored The admin gate read the stored deployment, so a team admin lost edit, delete and Test Connection on any deployment carrying federation params. It now returns early unless the submitted fields touch the federation surface, and a Test Connection probe that points the deployment at its own api_base is still refused, with the 403 no longer wrapped into a 500 The rest of the same review pass: POST /model/new refuses only a blocking value of `blocked`, so a client that always sends `blocked: false` is not turned away; a request body can no longer pick which federated identity to mint as by naming a stored credential; an advisory refresh the executor refuses disarms the entry instead of wedging the identity until the follower timeout; the static-key shadow warning resolves its env fallback inside the cache instead of once per request; credential writes drop nulls before storing them; the token exchange validates the endpoint URL before reading an assertion and keeps refusing redirects across a client heal; /health hides every server-owned federation field from non-admins; and the async create_file and create_batch paths say which setting is missing when the provider resolves no URL * style(tests): keep two blank lines before the batch_cost_is_final block * test(batches): type the workload identity fallback test fixtures * fix(moderation): stop validating unrelated kwargs on the sync workload identity path * fix(auth): refuse server-owned workload identity keys in array form and inside a client router config * fix(openai): keep a deployment's own key ahead of workload identity on every route Thread the caller's own api_key into the identity resolver for embedding, transcription, moderation, image generation, and assistants so a deployment key equal to the process key still wins. A partial identity trio now falls back to a configured static key, unset max_retries keeps the SDK default, and /v1/decisions plus OpenAI count_tokens resolve the exchanged bearer for identity-only deployments * fix(containers): pass deployment params to container file auth headers Container file routes built their auth headers without the deployment's litellm_params, so a keyless deployment with an OpenAI workload identity sent Bearer None on upload, list, retrieve, delete and content calls * fix(openai): keep process-wide key precedence and move token exchange off the event loop Restore the merge base order where an explicit or process-wide OpenAI key wins over federation, so exporting OPENAI_API_KEY keeps working the way it did before. Run the header-building validate_environment calls (image edit, videos, containers, vector stores, vector store files) and the decisions and System One request preparation in a worker thread, so a slow token exchange no longer stalls the loop. Federate realtime websocket sessions, live sessions, realtime client secrets and transcription sessions, and the OpenAI moderation guardrail when no key is set. * fix(openai): answer 401 for a partial identity trio and federate realtime health checks A deployment with part of the workload identity trio and no static key now fails as an auth error naming the missing params, so the router does not retry it as a server error. The realtime health check sends the exchanged bearer for a keyless federated OpenAI deployment, and API token counting reads the same static keys as every other route before minting. * fix(openai): keep local token counting without an identity trio and federate the moderation guardrail from its own config * fix(openai): skip a partial-trio moderation guardrail and let vault keys win in token counting * test(integration): refuse caller-chosen identity params in user_config and bracketed form keys * fix(openai): send the honored secret manager key from SDK client builders A deployment whose OPENAI_API_KEY lives only in the secret manager now sends that key on image, assistants, files, batches, and other SDK-built routes instead of failing with no api_key, matching what the workload identity resolver already treats as a static key * test(openai): check the token exchange runs off the event loop instead of timing the loop --------- Co-authored-by: derhornspieler <15236687+derhornspieler@users.noreply.github.com>	2026-10-16 22:25:15 +00:00
ryan-crabbe-berri	0dac61a555	fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338) ... * fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open GuardrailToolParam required a type key, so Gemini native tools (function_declarations, googleSearch) failed validation inside the hook and either returned a 500 or skipped the scan when fail_on_error was false. The type key is now optional and left out of the forwarded tool when absent, so an echoing guardrail server cannot write type: null into the provider request. structured_messages was re-validated against the chat completion message types, which silently emptied tool_result, tool_use, image and document content blocks to content: [] and dropped undeclared keys before the guardrail saw them. The request model now forwards rows as opaque mappings, which also retires the write-back shim that restored the original rows for echoed-back entries. The fail-open critical log line now carries error_type so a timeout, a refused connection and an upstream HTTP status can be told apart without the traceback. * fix(httpx): retry aiohttp server disconnects the way httpx does aiohttp raises ServerDisconnectedError when a kept-alive connection is closed by the peer before a response arrives, which is what a load balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces. The transport mapped it to httpx.ReadError, which the handler's connection-error retry does not cover, so the request failed instead of being resent on a fresh connection. It now maps to httpx.RemoteProtocolError, the error httpx itself raises for the same condition, so the existing retry fires. * test(httpx): build the stub server reply in one bytes literal * fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable The connection-error retry closed its fresh client before a streamed body was read, so an aiohttp disconnect now billed a second upstream request and still failed. It also skipped masking on a status error from the resend, resent PATCH and DELETE as POST, dropped multipart files and ignored the handler's ssl_verify. The retry now resends the request the first attempt built, on a client with the handler's TLS settings, closes that client when a streamed body is closed, and masks a status error the same way the first attempt does. * fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type * fix(guardrails): resend a generic guardrail check cut before any response A load balancer that closes an idle keepalive connection made the guardrail POST fail with "Server disconnected", because #46336 only resends POST on ConnectError. AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to False, that gives a RemoteProtocolError the same single fresh-client resend PUT and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a guardrail check is a pure verdict over the request. Every other POST keeps the #46336 rule The fail-open log line now puts error_type at the end so the pre-PR text stays contiguous for existing alert matchers test_rows_reach_the_guardrail_verbatim now reads the body off a real local guardrail server instead of mock call args * fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration A ServerDisconnectedError raised while reading a response body now stays httpx.ReadError, as it was before this PR. Only a disconnect before any response maps to RemoteProtocolError, which is the case the resend rule is about. Stream readers that end on ReadError, like the SAP stream iterator, keep their old behavior The loopback-server disconnect, TLS and guardrail resend tests move to tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is in-process only. Unit keeps in-process MockTransport checks for the idempotent POST resend, masking on the resend, streamed and buffered client closing, the guardrail resend, and the guardrail receiving rows verbatim * fix(guardrails): resend a reset guardrail check and keep typed echoes on texts An idempotent POST now resends once on any network error after connect (ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load balancer that answers a stale keepalive with a TCP reset rides it out too. That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes back to ReadError. This stops PUT and DELETE from stacking litellm's resend on top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil Guard from replaying its analyze POST on a disconnect Every resend now runs inside the method's own error mapping, so a timeout on the resend is a litellm.Timeout like the first attempt and a status error is masked once by the caller A guardrail server that parses rows with its own message types and echoes them back without the keys those types lack (annotations, tool_calls[].index) is read as an echo again, so its texts redaction reaches the LLM. A row only counts as an echo when no value changed and every key the chat completion types declare is kept The wire tests for the PUT/DELETE resend, its masking and its TLS settings now run on the httpx transport, where litellm's resend is the only retry, and the POST and guardrail tests cover both transports and both cut styles * test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head * fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement * refactor(guardrails): move the dropped-connection guardrail resend out of this PR The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line * test(guardrails): pick the closed guardrail port from the integration port allocator Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port * feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-10-16 14:52:35 -07:00
yucheng-berri	f281f77bc0	fix(guardrails): record every content filter match in logging_only and offer the mode in the UI (#46673) ... * test(guardrails): pin content filter logging_only matches and UI modes in an integration test * fix(guardrails): record every content filter match in logging_only and offer the mode in the UI * style(tests): format the content filter logging_only integration test * refactor(guardrails): keep the logging_only predicate and scan override within the type gates * fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning * test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test * test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects * refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach * test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter * test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up * test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-16 00:37:03 -07:00
yucheng-berri	627bcab42b	fix(guardrails): honor streaming settings on the Noma v2 guardrail (#46606) ... * fix(guardrails): honor streaming settings on the Noma v2 guardrail The Noma v2 initializer now passes streaming_buffer_until_moderated, streaming_end_of_stream_only, streaming_sampling_rate and streaming_buffer_release_on_scan from the guardrail config to NomaV2Guardrail, so the unified streaming hook applies them. The constructor validates them, so a custom guardrail class built on NomaV2Guardrail gets the same checks. Unset keys keep the hook defaults, and an invalid value is ignored with a warning naming the guardrail, so the guardrail still loads. Resolves LIT-9633 * test(guardrails): cover Noma v2 streaming settings across clients, failures and workers Adds integration cells for the custom-subclass loader, string and invalid values, unset and false keys, a Noma outage with and without block_failures, an upstream rejection, n=2 choices, pre_call and non-streaming guardrails, the OpenAI and Anthropic SDKs (sync and async), stored-guardrail info, a PATCH while traffic flows, a burst while Noma is down, a slow Noma, and replaced workers. Resolves LIT-9633 * test(guardrails): pin the PATCH probes to one worker at a time Each probe suspends the other worker and opens a fresh connection, so the test checks that every worker applies the patched setting. Resolves LIT-9633 * test(guardrails): check sampled scans by count, not window position /v1/messages hands the streaming hook raw network reads, so where the default sampling windows fall depends on timing; the base leg caught one shifted window. Check the scan count and the final full-text scan instead. Resolves LIT-9633 --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-15 18:29:05 -07:00
moyai-devin-berriai[bot]	3151c47960	fix(guardrails): resolve decision models in caller team scope (#46085) ... * fix(guardrails): resolve decision and judge models in caller team scope * fix(guardrails): scope guardrail model calls to the caller's team only where it owns the deployment Read the team only from the auth object the proxy attached to the request, never from caller JSON, and use it only when the router's route to the guardrail model for that team reaches a deployment a teamless caller cannot. Shared names, aliases, legacy team tags and wildcards keep resolving and billing as before. Decision calls pass the team in litellm_metadata only when it is used, so the logged model group stays intact. Adds integration cells for chat, responses and messages, forged team ids, a team deployment that shares a name with a shared model or wildcard, and billing, plus unit cases for legacy team tags and aliases. * fix(guardrails): route team guardrail calls to the team deployment without charging the team Guardrail calls on a team's own model now go to the deployment the router picks for that team, by its id, with no team on the call. Like every other guardrail call, they write no spend row and add nothing to the team's or the key's spend * fix(guardrails): pin the team deployment only for the judge guardrail, not shadow eval judge calls * fix(guardrails): keep the teamless route when only a team wildcard covers the guardrail model * refactor(guardrails): scope the team model fix to the decision model guardrail only * fix(guardrails): send decision calls on a team's own model with the team, and keep team wildcards out The decision call on a team's own model now carries the caller's team instead of being pinned to a deployment id, so it is charged to that team and goes through the team's normal routing: an access window that reserves the deployment for the team lets the call through, and pre-call checks see the input and skip a deployment too small for it. A team deployment listed under a wildcard public name, such as team_public_model_name openai/*, no longer captures a guardrail model the team never named, whatever model it points at. Adds unit and integration cells for the reserved deployment, mixed context windows under enable_pre_call_checks, a team wildcard name over a fixed model, and the owner team's spend. * fix(guardrails): log the owner team's name on decision calls charged to it The decision call on a team's own model now carries the team alias beside the team id, so the spend row and the Logs page name the team the call was charged to. * fix(decision_model): use the caller's team only for names nothing else serves A team model named like a shared deployment, a shared wildcard or an SDK public name took over the guardrail's decision calls for that team, so a team admin could point a proxy-wide guardrail at their own endpoint. The team is now used only when a caller without a team reaches no model under the name. * refactor(decision_model): drop the team route subset check the served check covers Once nothing serves the name without a team, every deployment a caller without a team can reach is a wildcard, so the subset check only ever rejected an empty team route. Pin team wildcards over a name nothing else serves. * fix(decision_model): use the caller's team only when judge_target finds nothing without a team The served check rebuilt the router's rules and missed a shared wildcard whose provider comes from custom_llm_provider, so a team model under the guardrail's name took over that team's decision calls. Ask judge_target, the same lookup the teamless dispatch makes, instead --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>	2026-10-15 11:43:36 -07:00