mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-29 06:33:28 +00:00
128 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yucheng-berri
|
74392efbd4
|
fix(otel): label configured pass-through routes litellm.passthrough and read the operation from the upstream (#48675) | ||
|
devin-ai-integration[bot]
|
1171273511
|
fix(langfuse otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata on the OTEL v2 Langfuse mapper (#42697)
* fix(langfuse_otel): emit request metadata under langfuse.observation.metadata and langfuse.trace.metadata keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): name trace metadata identity keys after their request metadata fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): type metadata helper and drop null fields so they cannot clobber caller trace metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): annotate new request metadata tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): build rig config and wire batches immutably Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): skip empty trace identity values so teamless requests cannot erase attribution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): drain wire batches per poll instead of accumulating Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): cover null stripping and empty trace identities Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): set redaction flag directly instead of monkeypatching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): add integration audit matrix for langfuse metadata keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): tighten audit chaos and unauthenticated cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): restore contracts.json formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): close SDK clients, pin worker identity, and normalize response id matching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): gate the slow sink reply instead of sleeping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): emit request metadata and identity under langfuse.observation.metadata and langfuse.trace.metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel v2): satisfy type discipline gate for metadata export Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: drop prohibited covers markers from langfuse otel integration tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): put the mutable-ok suppression on the dict construction line the type-discipline gate reports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse otel): share request metadata export across callbacks * fix(langfuse otel): drop prompt variables, MCP tool I/O and vector store queries from exported request metadata * fix(langfuse_otel): keep span export alive when nested request metadata has non-string keys The content strip validated prompt_management_metadata, mcp_tool_call_metadata and vector_store_request_metadata items as str-keyed mappings, so an SDK caller passing one with a non-string key raised inside exported_request_metadata: OTEL v2 dropped the LLM span for every vendor and v1 langfuse_otel lost the request metadata and every attribute after it. Validate those nested mappings with object keys instead. Also hoists the langfuse_otel imports. * fix(langfuse_otel): send the request metadata blob to Langfuse destinations only - A requester_metadata that is not a mapping no longer raises. The v2 LLM span (every vendor) and the v1 Langfuse attributes survive it again. - Prompt-quoting routing_decision fields stay out of langfuse.observation.metadata. - An exporter or key/team destination for any other backend drops langfuse.observation.metadata and langfuse.trace.metadata.user_api_key_*, as on main. This covers a collector folded into the Langfuse logger. - The v2 Langfuse key-set pin in the arize family test gains the two new keys. * fix(langfuse_otel): export only caller identity under the Langfuse metadata keys - The Langfuse metadata keys carried the whole request metadata. On OTel v2 that sent request headers, caller metadata, user email, user agent and auth metadata to a Langfuse destination that never received them on main; on v1 it doubled each span's metadata payload, so big caller metadata doubled the export size - Both paths now export only the key alias, user, end user, team, team alias and org. On v2 that is the team plus the identity among the operator's `baggage_metadata_keys`, which v2 spans already carry as `litellm.metadata.*`. Everything else stays in the bare `metadata` attribute, as on main - `output_messages` returns a list again, and the arize v2 b3 key pin is back to main's set * fix(langfuse_otel): drop the caller identity first when a span hits its attribute limit v1 wrote the identity attributes after the OpenInference ones, so a span near the 128-attribute limit (about 33+ tools) evicted openinference.span.kind, metadata and llm.model_name instead. The identity now goes first on a fresh span, so the SDK evicts it before anything else, and a caller's active span that already carries attributes does not get it. v2 sheds the Langfuse identity keys before any indexed message attribute when a span is over budget, so a span keeps every attribute it carried without them. * fix(langfuse_otel): keep other spans and dropped counts as main exports them - v1 writes the caller identity only into the room the span's attribute limit leaves next to the error attributes a failed call adds, so the exported dropped-attribute count matches main - v2 strips the Langfuse request metadata only from LiteLLM's own spans; application, instrumentation and server spans keep every key they set - a stripped v2 span keeps the dropped attribute, event and link counts - a config that lists the Langfuse mapper without the langfuse_otel exporter also keeps the request metadata away from other exporters * fix(langfuse_otel): keep the caller identity off the caller's active span v1 writes LiteLLM's attributes onto the caller's active span when one is current instead of starting its own span. That span goes to the caller's own exporter, so the identity keys reached a non-Langfuse collector and could take attribute slots the caller fills later. The identity keys now go only on the span LiteLLM starts, whatever the caller's span carries. * fix(langfuse otel): keep the identity keys on the OTEL v2 Langfuse mapper only The v1 langfuse_otel callback goes back to main: it sends no langfuse.observation.metadata or langfuse.trace.metadata.user_api_key_* keys. The OTEL v2 Langfuse mapper keeps them. The integration cells now run every endpoint, client, redaction, failure and chaos case on v2. * test(langfuse otel): count only accepted batches in the sink outage cell The wire also records the batches the sink answered 503, so an exporter retry made a span look exported twice. The chaos cells now also check the spans that landed carry the caller identity. * test(langfuse otel): split stacked span comprehensions and drop constant comments * test(langfuse otel): split the metadata integration module into focused files * fix(langfuse otel): send the org id in trace metadata too The trace metadata keys now reuse IDENTITY_METADATA_FIELDS, so a key in an org gets langfuse.trace.metadata.user_api_key_org_id next to the generation's org id. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
21ab2cb87b
|
test: update stale integration and guardrail e2e tests to intended behavior (#48665)
* test(integration): update stale expectations to intended provider behavior Each expectation moved because a merged PR changed behavior on purpose. - chatgpt string input is wrapped as one user message (#41522) - Gemini 2.5 tool media sits beside function_response (#48428) - sampling knobs move to Gemini 3.5 Flash, which keeps them (#47645) - a body base_model is stripped without allow_client_pricing_override (#47639) - ai_decide answers without usage return usage null (#48261) - azure_ai forwards mapped client anthropic-beta values (#48033) - the OpenCode scripted upstream mints a fresh id per request * test(e2e-ui): type the guardrail name before deleting it GuardrailDeleteModal requires the exact name before Delete enables since #45893. All three guardrail delete flows now go through one helper. * test(observability): accept any sampled traceresponse flags and expect a 400 for a non-object body OpenTelemetry 1.45 (#48055) sets the W3C random trace id flag on new traces, so the gateway's traceresponse now ends in -03. The producer copies the server span's own trace flags, so the tests now accept any two-digit flags with the sampled bit set instead of pinning -01 #48438 made a non-object JSON body a 400 "Invalid JSON payload: expected an object" on purpose, so the [1] body case is renamed non_object_body and expects that 400 * test(observability): stop waiting on OpenTelemetry's export log line in the langtrace outage test The outage test waited for "Exception while exporting Span batch" in the proxy log, which is OpenTelemetry's wording and no longer appears with 1.45. The test keeps asserting what the gateway guarantees, callers unaffected during the outage, a live gateway, and every span after recovery delivered once with no outage span duplicated * test(guardrails): count the Responses disconnect spend rows #47611 added Since #47611 a streamed /v1/responses disconnect writes its own spend row, so the concurrent disconnect test saw 20 rows where it counted 10 by model group. It now reads each row's call type and expects ten acompletion rows with the outage verdict and ten aresponses rows with success. /v1/messages disconnects still write no row on main, so they are left out of the count rather than pinned as absent The responses-httpx case of the single disconnect test no longer needs its LIT-8603 skip and passes * test(observability): size K4's timeout from the harness boot budget K4 boots three two-worker proxies in a row. On CircleCI each boot takes 43 to 58 seconds, so a passing run took 179.97s against its fixed 180s cap and 6 of 30 runs hit pytest-timeout. The cap and the restart gate wait now come from readiness_seconds and graceful_stop_seconds, like other multi-boot tests |
||
|
Kerry Lu
|
59b8c73930
|
fix(proxy): abort the request with a 499 when the client disconnects during the body upload (#48640)
* fix(proxy): abort the request with a 499 when the client disconnects during the body upload Co-authored-by: Sinfony8838 <Sinfony8838@users.noreply.github.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): half-uploaded body on chat and vector store routes never reaches the provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): close the auth-time OTel server span before aborting a half-uploaded body --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Sinfony8838 <Sinfony8838@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0657ed46cf
|
fix(langfuse_otel)!: derive trace-level name, input, output and tags on traces litellm starts (#42705)
* fix(langfuse_otel): derive trace-level name, input, output and tags on the generation span Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep every reasoning summary in the derived observation output Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): leave joined traces untouched by derived trace fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): restore preset cache key fallback and type the derived-field helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep trace tags off joined traces entirely Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): tolerate out-of-band upstream requests and add sink outage chaos cell Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): build the langfuse rig config without mutating the loaded yaml Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(langfuse_otel): flatten output items with itertools.chain to satisfy LIT014 budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse_otel): type the _RecordingSpan and _emitted unit helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse_otel): keep joined-trace tags, mask trace io, leave v2 trace io to the root Joined traces (existing_trace_id) keep the caller tags they carried before, and only the derived trace fields are skipped for them. mask_input and mask_output now redact the derived trace input and output, matching the vanilla Langfuse callback. The v2 mapper derives the trace name only, because the root observation already carries the trace input and output there. Uses the public preset cache key helper, moves imports to module top, restores the original formatting of the unit test file, and adds integration cells for the v2 llm_only scope and SDK sync/async calls. * fix(langfuse_otel): split string caller tags the way Langfuse reads them A caller tags value given as a comma list or a JSON array string (body metadata.tags or the langfuse_tags header) used to reach Langfuse raw, where ingestion split it into separate tags. Merging it with the derived tags kept it as one tag, so parse it the same way before the merge * test(otel): expect langfuse.trace.name on the v2 langfuse_otel generation span * fix(langfuse_otel): read string mask flags the way the classic callback does The langfuse_mask_input / langfuse_mask_output headers arrive as strings, so "false" redacted the derived trace io. Parse them with str_to_bool and fall back to truthiness, as the classic langfuse callback does * fix(langfuse_otel): derive trace fields only on traces litellm starts A litellm span that joins a trace an app or caller started (inbound traceparent, the app's active span, a caller-provided parent span) no longer gets a derived trace name, input, output or tags, like a request joining an existing_trace_id. Langfuse applies the later-starting span's trace update, so the derived values overwrote the app's own trace name and io. OTel v2 derives the name on the request root only when it has no parent, and the generation span keeps main's caller-only trace name. * test(langfuse_otel): keep main's OTel v2 generation key set in the arize family cell The OTel v2 generation span no longer carries a derived langfuse.trace.name, so the pinned key set goes back to main's. * fix(langfuse_otel): name the OTel v2 request root once, from the request's own call The v2 Langfuse logger re-derived langfuse.trace.name on the request root for every LLM call in the request. A post_call llm_as_a_judge guardrail's judge call replaced the caller's trace_name on the root with litellm-acompletion, and with another v2 callback (otel, arize_phoenix) listed before langfuse_otel that vendor's root span gained langfuse.trace.name on every request. The derived name now fills only an unnamed root this logger's provider started, and never comes from a call litellm makes for its own features. A caller trace_name still names the root on every call, as on main. * fix(langfuse_otel): keep the derived trace fields from pushing other attributes off a full span The derived trace name, tags, input and output were written after the OpenInference attributes with no room check, so a span near the SDK's attribute limit (about 34 tools at the default 128) evicted keys main exported, such as openinference.span.kind, metadata and llm.model_name, and raised the dropped count. They are now written last and only into the room the limit leaves next to the error attributes a failed call adds, so a full span exports what it would without them. * test(langfuse_otel): give the full-span cells' derived request a caller trace id, as the joined one's traceparent does * chore(langfuse_otel): drop the docstrings and comments the house rule disallows and type the new tests' parameters * test(langfuse_otel): set the span attribute limit the full-span tests run at and keep Final locals out of the app-trace loop --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
715e9bb08d
|
fix(prometheus): count provider-side 401s in LLM failure and deployment metrics (#46668)
* fix(prometheus): count provider-side 401s in LLM failure and deployment metrics The invalid virtual key skip no longer applies to an error the LLM provider returned while the deployment was called with its own credential, so a 401 for a bad deployment credential increments litellm_llm_api_failed_requests_metric, litellm_deployment_failure_responses and litellm_deployment_total_requests and marks litellm_deployment_state as a partial outage again. Proxy-gate rejections and a failing credential that belongs to the caller (a rejected per-user credential, an api_key sent with the request, a per-user GitHub Copilot session) keep the skip * fix(prometheus): keep a caller's own Anthropic OAuth token failure out of the deployment metrics A provider 401 on an Anthropic OAuth token the caller forwarded through the proxy is the caller's credential failing, so it stays out of the deployment failure and outage series like a caller-supplied api_key. Adds integration cells for an Anthropic deployment's own key, the forwarded caller OAuth token and a direct SDK call that brings its own key * test(prometheus): pin how a caller's own key or Authorization header on a deployment is counted An SDK Router call that passes its own api_key is counted against the one-off deployment the router builds for that key, so the operator deployment keeps only its own failure. A provider 401 on an Authorization header the caller sends through the proxy counts against the deployment, as on v1.80.12 * fix(prometheus): keep caller-sent provider credentials and token-exchange 401s out of deployment metrics A caller can send its own provider credential in the request body, such as azure_ad_token, azure_scope, the Azure Entra app fields, watsonx token, gigachat_access_token, OCI keys, AWS keys or Vertex credentials, at the top level or inside extra_body. A provider 401 on that credential is the caller's failure, so it stays out of the shared deployment's failure, request, LLM-failed and state series, as on main. A token-exchange deployment trades the caller's identity token on every call, so its 401s stay out as well, matching the router cooldown. * test(prometheus): Final locals, precise helper types and one for per comprehension in the new test helpers * test(prometheus): pin that a provider 401 at a caller-sent api_base counts against its one-off deployment * fix(prometheus): keep a provider 401 on caller-sent headers out of deployment metrics A caller can override the deployment's credential with headers it sends in the request body: headers or provider_specific_header, or an Authorization, api-key, x-api-key, x-goog-api-key or Ocp-Apim-Subscription-Key entry in extra_headers. A provider 401 on such a request is the caller's failure, so it stays out of the shared deployment's failure, request, LLM-failed and state series, as on main. * fix(prometheus): skip a provider 401 only when the caller's headers carry a credential The proxy now records headers, extra_headers or provider_specific_header in credential_fields only when they name a provider credential header (Authorization, api-key, x-api-key, x-goog-api-key, Ocp-Apim-Subscription-Key), the same way api_key is recorded. A caller that sends a plain header such as a trace id still has a provider 401 on the deployment's own key counted against the deployment. * fix(prometheus): stamp credential_fields after the proxy forwards the caller's headers A provider credential header the proxy forwards from the caller's HTTP request (forward_llm_provider_auth_headers + forward_client_headers_to_llm_api) lands in data["headers"] after proxy_server_request is built, so the stamp missed it and a 401 on that caller credential was counted against the deployment. Compute the stamp after the header forwarding steps and pin both forwarded cases. * fix(prometheus): skip a provider 401 when the caller sends any header or an OpenAI organization The proxy now records headers, extra_headers or provider_specific_header in credential_fields whenever the caller sends or the proxy forwards any header there, and records organization when the caller sends one in the body or as a forwarded OpenAI-Organization header. A forwarded header can be a gateway key such as x-portkey-api-key or an OpenAI-Project id, and a wrong organization gets a 401 from OpenAI, so these 401s stay out of the deployment metrics as on main instead of marking a healthy deployment as a partial outage. * fix(prometheus): ignore the user information the proxy adds when stamping caller headers With add_user_information_to_llm_headers on and header forwarding enabled, the proxy adds x-litellm-user_api_key_* headers to the forwarded headers. Those carry the proxy's own key and user details, not the caller's, so they no longer make a request look caller-scoped, and a provider 401 on the deployment's own key on such a request is counted. * fix(prometheus): skip a 401 raised before the provider is called or at the caller's own endpoint A deployment with no key of its own fails with LiteLLM's local "Missing Anthropic API Key" 401 before any provider call, and a caller who sends aws_bedrock_runtime_endpoint (or another provider endpoint field) can get a 401 from their own gateway. Both were counted against the shared deployment and marked it as a partial outage, where main skipped every 401. Count a 401 only once the provider was called (api_call_start_time is set), and treat caller-sent provider endpoint fields like caller credential fields. * fix(prometheus): count the deployment's own 401 on requests that carry only anthropic-version or anthropic-beta The Anthropic SDK sends anthropic-version on every request, and the proxy copies anthropic-version and anthropic-beta into provider_specific_header for Anthropic, Bedrock and Vertex. Both were treated as caller headers, so a 401 on an Anthropic deployment's own key from any Anthropic SDK client stayed out of the deployment metrics. These two headers only pick the API version and beta features. Anthropic answers a bad anthropic-version or an unknown anthropic-beta with a 400, never a 401, so they cannot turn the deployment's own key into a 401 and no longer mark the request as carrying a caller header. * test(prometheus): read and write the owned proxy's own database in the expired-key and MCP cells Since #47213 an owned proxy boots on a private database, so the expired-key update and the MCP guardrail row read now go through the gateway's database_url instead of the worker's. * fix(prometheus): keep credential_fields as main stamps it and give Prometheus its own caller_credential_fields credential_fields tells the Rust OCR bridge which stripped body fields the caller sent. Repurposing it for the Prometheus caller-credential check changed its value (organization and extra_headers added, anthropic-version-only headers dropped). Restore main's stamp unchanged and stamp the Prometheus classification under caller_credential_fields, which only Prometheus reads. * fix(prometheus): skip a provider 401 on a stored credential the caller picks by litellm_credential_name A request body litellm_credential_name swaps the deployment's stored credential for one the caller chose, so a 401 on it is the caller's, not the deployment's. Treat it like the other caller credential fields so it no longer marks a healthy shared deployment as a partial outage. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
07a82a287b
|
fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text (#48276)
* fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text The pass-through guardrail translation json-dumps the request body as the scan text when no request_fields are configured. The unified guardrail hook injects litellm_metadata into that body first, so the dump carried the caller's key metadata. Exclude it alongside metadata and litellm_logging_obj; vendors still receive it through request_data. * test(guardrails): type the pass-through scan-text tests per AGENTS.md |
||
|
yucheng-berri
|
38f318e736
|
fix(otel): decide per-team Arize sampling from the trace id so every pod agrees (#47468)
* fix(otel): decide per-team Arize sampling from the trace id so every pod agrees * test(otel): cover per-team sampling across pods, workers, endpoints and restarts * fix(otel): remember only a failed tree's sampling verdict so a later failed call answers to the error rate A successful tree's verdict follows from the trace id alone, so remembering it only made a later failed call of the same trace follow the success rate on the pod that served both, while other pods judged it at the error rate * test(otel): flush every worker before the restart cell kills it and cover a failed call after a dropped one * fix(otel): decide team sampling per request, not per trace Calls sharing a caller's trace id are separate requests, keyed by their root span. Each is decided by the trace hash at the rate for its own outcome, and only its own late spans follow its verdict, so no pod applies an earlier call's verdict to a later call of the trace * fix(otel): forget a span's request root when it ends, not when newer spans start The root of each open span was kept in a first-in-first-out table of 16384 entries, so a span two levels below the root that stayed open while the worker started more spans than that lost its entry and ended under its parent's id. Its request then never closed and was held from every sampled destination until shutdown. Each open span now keeps its entry until it ends, and a span starting under a parent that has already ended reads the root from the parent span itself * fix(otel): decide a request as its spans end once it falls out of the open span table A request whose span was pushed out of the table of open spans, because the worker held more than 16384 open at once, ended that span under another key and was held from every sampled destination until shutdown. Pushing a span out now drops its request's open count too, so the request is decided as each of its spans ends, the way one pushed out of the open request table already is * test(otel): annotate the team sampling test locals as Final * test(otel): build the caller's remote parent in one helper instead of rebinding it per trace * fix(otel): keep each open span's request until it ends, bounded by the open requests The span table had its own bound of 16384, so a worker holding more spans open than that pushed out the oldest, and a failure in such a span no longer reached its request. Each request now holds the ids of its open spans, and a span's entry in the lookup goes when the span ends or when its request is pushed out of the open request table, the same bound main already applies. A failure anywhere in a tracked request reaches its verdict however many spans are open --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
alberto
|
f768fb32ef
|
fix(streaming)!: send the include_usage chunk with an empty choices array (#47109)
* fix: Send the include_usage chunk with an empty choices array Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Skip chunks with no choices when reading streams Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix: Strip usage from plain-iterator async stream chunks Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * test: Expect empty choices on the usage chunk in the gemma tracing test Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> * fix(streaming): send the empty-choices usage chunk only to callers that asked for include_usage The usage chunk now has choices=[] only when the caller set stream_options.include_usage itself. When the OpenAI handler, the proxy's always_include_stream_usage setting, or the harness added include_usage on the caller's behalf, the chunk keeps its legacy single empty-delta choice, so clients that read choices[0] on every chunk keep working. The proxy and harness mark their injection with the private _litellm_stream_usage_injected flag, and the proxy scrubs a client-sent copy. Plain-iterator streams keep usage on their chunk when usage was not requested, and the LLM Shield trailing flush ignores chunks with no choices. * fix(proxy): mark the default stream usage injection so hooks keep the legacy usage chunk shape The proxy's default strip-mode injection adds include_usage for clients that did not ask and drops the usage chunk before it reaches them, but streaming iterator hooks still saw it with choices=[]. Marking that injection with _litellm_stream_usage_injected keeps the old single empty choice for them. * fix(router): treat deployment and router-default stream_options as injected, and drop the client flag on the queue route * refactor(router): move the stream usage injection check into router_utils * fix(router): read stream_options from pydantic deployment params when marking injected usage Update the mantle, openinference, and text-completion integration tests to expect the include_usage chunk with empty choices. * fix(proxy): keep the stream usage injection marker out of stored request bodies * test(streaming): cover who asked for stream usage on chat, queue, deployment and spend log paths * test: type the queue flag test, pin the logging start time, and build the deployment usage config without mutation --------- Signed-off-by: Alberto Perdomo <aperdomo@redhat.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
dd511a2b22
|
fix(logging)!: fire one CustomLogger failure hook per async call outside the async flag list (#47124)
* fix(logging): fire one CustomLogger hook per async call outside the async flag list aspeech, atranscription, aimage_generation, aimage_edit, aocr and asearch are missing from is_sync_litellm_request's flag list, so the sync success and failure handlers called a CustomLogger's sync hook on top of the async hook the async handler had already called for the same request. The sync handlers now skip a CustomLogger's log_success_event and log_failure_event when that run is paired with an async handler run (for_async_request) and the async handler gives the logger its own async hook. The async client wrapper, dispatch_failure_handlers, the proxy-only failure dispatch and the native bridge's sync failure step for an async call mark their sync failure run as paired. Resolves LIT-9486 * fix(logging): keep the sync CustomLogger hook when no async success runs The async client wrapper skips async success logging for calls carrying a fallbacks kwarg, and the native bridge skips it for internal and fallbacks calls, yet the paired sync success run still dropped a both-hooks logger's sync hook, so that logger got no success event at all. The sync success run now suppresses the sync hook only when the async success handler was actually scheduled for the call: the client wrapper passes async_handler_runs=not is_completion_with_fallbacks, and the native host records per call whether it enqueued or deferred the async success. Tests that pinned the doubled failure hook on native aocr, messages and responses calls and on proxy-rejected requests now expect the async hook alone. The integration test adds /v1/messages and /v1/responses cells and runs the key-auth rejection on every route. * fix(logging): type dispatch_success_handlers kwargs The new async_handler_runs keyword on success_handler made the untyped **kwargs that dispatch_success_handlers forwards an unknown argument for it, which grew the reportUnknownArgumentType count past its merge-base ceiling. Typing the forwarded kwargs as Any, as _success_handler_body already does, removes those diagnostics without changing what is forwarded. * test(integration): cover CustomLogger hook counts across modes, chaos and router strategies Moves the hook count rig into tests/integration/_support/hook_counts.py and extends the cells it drives: - streamed chat, Messages and Responses requests, success and failure - loggers the async handler never reaches (sync-only overrides, sync event lists only) keep one sync hook per request on every surface outside the async flag list, key-auth rejections included - a concurrent burst while the logger's destination stalls and then fails, and a burst while a worker is killed, never give a request two hooks - least-busy keeps counting a request in flight while async speech requests finish around it, and usage-based routing v2 counts an async image request's tokens once * test(integration): Datadog and LangSmith deliver one record per async request Points the built-in Datadog, LangSmith and OpenTelemetry loggers at a local sink and checks every async surface outside the flag list: one Datadog log entry and one LangSmith run per request (two on the merge base), and never two OpenTelemetry spans, which its own emit-once guard already ensured. * fix(logging): give a both-hooks logger its sync hook when the async run never reaches it The sync hook was skipped whenever the logger sat in the async callback list, so a logger the async run never reached got no hook at all: a sibling stalled past the logging worker timeout, a batch poll that skips default logging, or an async failure run cut off. The sync run now holds the hook and the async run hands it back unless it reached the logger * fix(logging): run the held sync success hooks when the proxy drops a blocked response's success A post-call guardrail block drops the parked async success, which left the sync success hook held for it unrun. Base gave a both-hooks logger that hook on these call types, so the drop now ends the pairing and the held hook runs as before * fix(logging): run held sync hooks only after every sync run of the request has closed A held hook released by the async run could run on another thread while the sync run was still walking its other callbacks. The pairing now waits until the async run has ended and every sync run has closed, and the last sync run to close runs the held hooks itself * fix(logging): end the async run when its coroutine is dropped before it starts A logging queue that drops the async success coroutine, or a deferred success that fails to schedule, left the sync hooks held for it unrun. The enqueued async run now ends when its coroutine is discarded, and the proxy ends it when firing the deferred success raises * fix(logging): leave a still-queued async run to the exit flush The discard finalizer ran at interpreter exit before the logging worker's exit flush, so a queued request could get its held sync hook and then its async hook * test: keep the router and file-stream tests in step with main Main gives each owned proxy a private Redis by default, so the router cells share the worker's Redis to read its counters, and the file-stream stub takes the hook pairing the stream now passes * fix(logging): run held sync hooks when a during_call guardrail raises or the exit flush gives up A during_call guardrail that raised after the LLM call parked its deferred success skipped the proxy's flush, and async runs the exit flush never started were never ended, so a both-hooks logger lost its sync hook in both cases * fix(logging): skip the sync failure hook when the async failure run logs to the logger An async call outside is_sync_litellm_request's flag list ran both failure handlers, so a CustomLogger with both failure hooks got both. The sync run of an async request now skips log_failure_event for a logger that overrides async_log_failure_event and is registered for async failures. The four sync runs of async failures (dispatch_failure_handlers, the @client async wrapper, the proxy-only failure thread and the Rust host) pass for_async_request=True. Drops the HookPairing fallback. Successes are left to #47192, which already gives such a logger only async_log_success_event. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Emerson Gomes
|
74230b4d50
|
fix(logging)!: redact Responses instructions and callback output (#43315)
* fix(logging): redact Responses instructions and callback output * fix(logging): avoid copying Responses output twice * fix(logging): return redacted Responses output without mutation * fix(logging): type Responses redaction payloads * fix(logging): redact streamed Responses instructions * fix(logging): redact request-side Responses instructions in logged params and spend log bodies * fix(redaction): keep per-callback opt-out off shared instructions and skip redacted replays A callback that sets message_logging=False no longer strips instructions from sibling callbacks, the spend log, or the client's response. Dict logging surfaces keep the reasoning config, and previous_response_id no longer sends the redacted placeholder upstream as the system prompt. * fix(redaction): redact instructions in an opted-out callback's own copy A callback with message_logging=False now gets instructions redacted in its kwargs copy and response object, while sibling callbacks and the spend log keep them * fix(redaction): give a message_logging false callback its redacted copy on sync calls The sync log_success_event and log_stream_event branch now hands an opted-out callback the same copy as the async path. * fix(redaction): redact a message_logging false callback's sync copy and keep failure estimates whole Give an opted-out callback a copy with messages, input, prompt, the sync stream's complete_response and its response object redacted, and stop the proxy's failed request estimate from counting the redaction marker in place of real instructions * test(redaction): wait on callback events instead of sleep polls The new redaction tests now wait for each callback to log through an event it sets, so they no longer poll on a timer * test(redaction): expect redacted follow-ups to replay no instructions This PR stops bridged previous_response_id follow-ups from reusing instructions the spend log stored redacted, so the integration test from #45923 that asserted the replay now asserts that neither the original instructions nor the redaction sentinel reach the provider * test(proxy): type the redacted-instructions failure estimate test --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
81d8c9af38
|
fix(decisions): bill unpriced returned TypeSafe versions at the requested model and name a missing answers field (#47508)
* test: bill each completed request once and price unpriced returned decision models at the requested model Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(logging,decisions): bill each request once and price returned decision models only when they carry rates Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(decisions): leave the double-billing fix to #47510 and keep the decisions pricing and missing-answers fixes * fix(decisions): name malformed answer entries and pick the Arize cache-hit span by attribute * test(decisions): cover a returned-model row with metadata but no rates * fix(decisions): keep upstream reply text out of the parse error so it stays a 500 * fix(decisions): answer every malformed upstream reply with a 500, whatever text it carries A reply that fails the decisions response schema now raises DecisionsResponseValidationError, which the decisions call maps straight to InternalServerError instead of passing it through exception_type. Its text matching read an answer key like rate_limit, or a value like "Request timed out", as a 429 or a 408, so the router cooled down a deployment for a reply that fails the same way on every retry. The message also drops the empty location for a reply that is not an object, the Arize family test loses the cache_hit entries main already added, and the Bedrock Mantle count-tokens wire test calls _metadata_deployment, since main renamed the _deployment helper under it. --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
e78f7b4602
|
test(integration): repair seven stale integration tests on main (#47600)
The Gemini TTS billing and text completion guardrail tests read spend rows from the owned proxy's own database, which #47213 made private. The cursor wire test derives its effort levels from the gpt-5 cost map row that #43935 now enforces, and the Azure effort check accepts the unquoted level. The pin TTL test asserts the one hour pin #46411 writes for a tool-only mark, the video callback test decodes ids with the product decoder after #44614, and the two Gemini 3.8 flash stream cost cases bill audio at the text rate their rows imply |
||
|
Max Azatian
|
5145250d83
|
fix(langfuse_otel)!: send the completion start time (#45597)
* fix(langfuse_otel): send the completion start time langfuse_otel never set langfuse.observation.completion_start_time, so Langfuse showed no time to first token for any call, streamed or not. The legacy langfuse callback already sends kwargs["completion_start_time"]. Send the same value as an ISO timestamp with an explicit offset. * fix(langfuse_otel): type kwargs and test against a real span Type the kwargs of _set_langfuse_specific_attributes as dict[str, Any], the declared type of model_call_details, so the new completion start time read adds no unknown-type errors. The tests now check a real SDK span instead of patching safe_set_attribute, and the parameter is typed. * fix(otel): send the completion start time from the v2 Langfuse mapper With LITELLM_OTEL_V2 the langfuse_otel spans come from LangfuseMapper, which had no completion start time either. Carry the payload's completionStartTime on LLMCallSpanData for streamed calls, the same calls that get time_to_first_chunk_seconds, and map it to langfuse.observation.completion_start_time. * fix(langfuse_otel): take the logging kwargs as a read-only Mapping The new kwargs annotation tripped LIT001. Both helpers only read the logging kwargs, so type them Mapping[str, Any] instead of dict. * fix(langfuse_otel): time /v1/messages streams from the call start and send the V2 completion start time like the default emitter * fix(langfuse_otel): time Gemini, Interactions and A2A streams from their first chunk and send the start time as JSON The native Gemini, Interactions and A2A streams never recorded a first chunk, so with completion_start_time now sent both emitters showed TTFT equal to latency on them. They now record it, and Gemini and Interactions take the call's start time like /v1/messages. Both emitters send the completion start time as a JSON string, the form the Langfuse SDK and the langfuse callback send, through one serializer. * fix(langfuse_otel): time A2A streams from the call start and record a cached replay's first event A2A streams took their start time when the iterator was built, so their latency left out the work before it. They now use the call's start time like every other route. Cached /v1/messages stream replays recorded no first event, so their time to first token fell back to the end time. The replay now records it on the first event it yields. The tests use the real Interactions config and typed logger hooks. * fix(a2a): time streamed A2A calls from before the agent card fetch and make the cached replay test clock-free * test(a2a): type the card fetch test and stop rebinding the seed helper's recorder * test(otel): expect completion_start_time and cache_hit in the B3 span keys --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c4d24ec2f1
|
test(integration): remove BUG skips for 3 tests that pass on main (#47364)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
77bd7d6de8
|
test(integration): expect the requested model in streamed Responses events on the arize v2 rig (#47407)
#46662 made streamed /v1/responses lifecycle events report the requested model in the nested response.model, matching the non-streamed body. The arize v2 caller-stream helper still expected the upstream deployment name there, so f1 and a6 failed on every main run since
|
||
|
yucheng-berri
|
365d7342c0
|
fix(prometheus): label deployment failures with the deployment's provider instead of None (#47384)
* fix(prometheus): label deployment failures with the deployment's provider instead of None * test(prometheus): drive every deployment failure surface through the provider label Streaming chat, responses and messages, dropped and stalled upstreams, JSON-configured, custom-provider-map, prompt-management and unprefixed deployments, forged request metadata, an unauthenticated request, the OpenAI and Anthropic SDKs sync and async, and a concurrent burst counted exactly once per deployment --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
89a194bbd0
|
test(integration): hold the upstream until the disconnect is settled before reading its spend row (#47303)
test_client_disconnect_before_any_content_sends_no_response_scan released the gated upstream right after
the client closed, so the upstream's next chunk raced the disconnect into the proxy. When the chunk won,
the proxy had " tail" on record for the client and the cut-short scan recorded a post_call success on
the spend row, which the test asserts is absent (CircleCI integration-extensions job 2263560 on
|
||
|
yuneng-jiang
|
eb114e638d
|
test(integration): look up owned cache keys and clean up sweep models (#47255)
Three integration tests diffed the whole shared Redis keyspace (scan_iter or dbsize before and after), so a sibling test's cache write could satisfy or break them. Each now finds its own entry by a value only it wrote: the Gemini reasoning test by a per-run Gemini responseId, the max_messages test by a per-test answer text, and the S3 cache-hit test by its request marker The sweep auth tests created models through raw /model/new on the shared gateway and never deleted them. They now create them through Scenario.model, which deletes each model on exit, including when setup fails |
||
|
yuneng-jiang
|
44d99475a5
|
test(integration): stop GET /__observations from draining the upstream log (#47256)
* test(integration): stop GET /__observations from draining the upstream log GET /__observations used to empty the queue, so a second read in a test saw nothing and a "never reached upstream" check could pass whether or not the request was forwarded. GET now returns every request since the last reset and DELETE /__observations is the reset. An autouse fixture resets the worker upstream before each test, which also covers tests that read without the gateway fixture. Discarded reads used as resets become DELETEs, helpers that accumulated drained reads now read one snapshot, and tests with several phases reset between them. The three vacuous negative checks (rag allowlist, guardrail opt-out chaos, xAI batch output) now assert the absence on the same read that shows a request which did reach the upstream. * style(integration): space the observations unit test like its neighbours |
||
|
moyai-berriai[bot]
|
f370235bb4
|
fix(guardrails)!: preserve Bedrock Converse redactions and honor tool-only scans (#45874)
Some checks failed
unit / assert-shard-coverage (push) Waiting to run
unit / enterprise-managed-files (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / proxy-server (push) Blocked by required conditions
unit / lens-python-310 (push) Blocked by required conditions
unit / caching-local (push) Blocked by required conditions
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Modules / fmt, validate, test (gcp) (push) Has been cancelled
* fix(guardrails): preserve Bedrock Converse redactions and honor tool-only scans * Preserve main's per-block streaming fixes and regression coverage * fix(bedrock): apply Converse guardrail rewrites texts-first and keep main's write-backs Map tool results onto their text and json parts only, accept a structured row the flat texts already reproduce, let a flat-text mask win over a structured leaf, ignore texts that do not line up with the request when structured rows come back, keep tool call arguments out of the rows under skip_tool_message_in_guardrail and send no tool definitions to tool-only guardrails * fix(bedrock): accept blank content on tool-call-only Converse turns A vendor that returns "" or [] where an assistant row carried only tool calls no longer fails the request closed. Also type the leaf lookup without Any and restore the original layout of the existing unit tests. * fix(bedrock): scan additionalModelRequestFields in Converse guardrail rows and split the handler Guardrails that read structured_messages instead of texts (Akto, Prompt Security, CrowdStrike AIDR and others) stopped seeing strings in additionalModelRequestFields, which Bedrock can forward into the model prompt. Those strings now go to the guardrail as one user row placed before the conversation, and a row-only edit to them is written back. The Converse to OpenAI row conversion moves to litellm/llms/bedrock/chat/converse_to_openai.py and the returned-row diffing to structured_edits.py, so the handler keeps only the glue. Rows record the Converse path of every text part, scope flags travel as one value, and path lookups return a value instead of raising. Unit tests use a real CustomGuardrail subclass instead of mocks, and the integration vendor modes get descriptive names. * fix(bedrock): keep new Converse helpers inside the type discipline rules * test(bedrock): pin Converse row edits that change shape, role or placement * fix(bedrock): type the Converse body once so its edits keep their reference * fix(bedrock): list request-field texts where their Converse row sits * fix(bedrock): send a reshaped Converse tool result as one text block A vendor that compresses or summarises a tool result, such as Headroom collapsing a JSON list into a table string, returned a 500 because the row could not be split back into the original text and JSON parts. The tool result now carries the vendor's string as one text block where its parts were, keeping images and documents in place. It still fails closed when the flat texts masked that tool result or the row changed its tool call id. * fix(bedrock): apply guardrail tool edits on Converse and keep compression guardrails off it A guardrail that masks a Converse tool description or schema now has that mask written into toolConfig, a tool it removes is dropped, and a tool it adds, renames or reorders is refused before Bedrock. Headroom, Compresr and TypeSafe get only the flat texts on Converse again, as on main, because the pass-through route has no loop to answer their retrieve tool * test(bedrock): refuse a guardrail that changes a Converse tool's kind * fix(bedrock): keep partial-row and row-rebuilding guardrails on flat Converse texts OpenAI moderation, the LLM judge, CrowdStrike AIDR, Prompt Security, PANW Prisma AIRS and HiddenLayer scan only part of the rows or rebuild their returned texts from them, so on Converse they get the flat texts only, as on main. When a vendor returns changed rows, its texts are applied only if they keep the flat layout and change only rows the vendor also changed, so row-shaped texts of the same length no longer overwrite unrelated Converse fields. * test(bedrock): pin how flat texts sit next to rewritten Converse rows * fix(bedrock): keep the Bedrock guardrail on flat Converse texts With experimental_use_latest_role_message_only, the Bedrock guardrail places its mask by counting text parts per row, and Converse rows do not line up with the flat texts, so the mask could land in tool call arguments. Give it the flat texts only, as on main. * fix(bedrock): keep Akto on flat Converse texts Akto maps its masked payload back onto the flat texts by value, and a Converse row joins or serialises tool call arguments and JSON tool results, so a masked card there could not be placed and the request was blocked. Give Akto the flat texts only, as on main, where the mask is applied. * fix(bedrock): list request-field texts last again, as on main Flat texts listed additionalModelRequestFields strings right after the system prompt, so flat-text guardrails got a different order than on main. Akto's live verdict on a benign request depends on that order and started blocking it. Rows already keep their own order and texts that do not follow rows are dropped, so the flat texts go back to main's order. * fix(bedrock): list request-field texts next to their row for row readers Row readers that rebuild texts from rows could swap a masked request field with a masked user turn when the field strings came last. Row readers now get those strings right after the system prompt, where their row sits. Flat-text guardrails keep the end-of-list order. * fix(bedrock): list nested request-field texts in their row's order for row readers Row readers that rebuild texts from rows could write a masked request field onto a sibling key when the field held nested objects, because the texts walked the fields depth first and the row breadth first. Row readers now get those strings in the row's own order. * fix(bedrock): walk request fields breadth first instead of resolving each path Resolving every request-field string by its path revalidated the parent object per string, so a wide additionalModelRequestFields took seconds. The string walk now goes breadth first for row readers, the same order the row lists them, in one pass. A unit test pins that texts and rows map the same way when the vendor masks them differently. * fix(bedrock): fail closed when a vendor's rows drop a mask its texts carry When a vendor returns changed rows whose texts do not line up with them, the rows win. A text the vendor masked on a field its rows left untouched (a tool call argument, say) then reached Bedrock unmasked. Refuse that rewrite instead, unless the edited body holds the changed text elsewhere, as texts rebuilt from the rows do. * fix(bedrock): read same-length texts by position when rows drop a mask Matching a dropped text against any string of the edited body let a row mask on another field with the same token cancel it, so the raw tool call argument still went out. Texts of the length that was sent are now checked position by position, and only texts of another length, which cannot be placed, are matched by value. * test(bedrock): count each rebuilt text against the rows' edits * fix(bedrock): keep PromptGuard on flat texts and refuse texts that cannot be placed PromptGuard rebuilds its returned texts from the user rows, and a vendor that does so while masking only row content let a card in a tool call argument reach Bedrock, which main masked. PromptGuard now gets the flat texts as on main, and a reply whose rows changed next to texts of another length fails closed, since what those texts mask cannot be checked. * test(bedrock): let the structured checks decide the unsafe tool call and missing row cases The replies in these tests carried texts that no longer matched the sent texts, so the new refusal for texts that cannot be placed answered first and the tool call and row count checks went unexercised. * fix(bedrock): send request field strings to row readers after the conversation, where main puts them Row readers got the additionalModelRequestFields row before the conversation. Singulr judges only the latest user message, so an injection in a request field that main blocked reached Bedrock. The row and its texts now come last, as main sends those strings in texts * test(bedrock): pin nested request field order for flat readers and the texts view Both request field tests now use nested fields whose depth-first and breadth-first orders differ, so a flat reader getting the row order or a texts view written in main's order fails * refactor(bedrock): ask each guardrail whether Converse should hand it rows Converse picked the guardrails that get only the flat texts from a list of provider names. CustomGuardrail now has supports_structured_messages, True by default, and the ten classes on that list override it to return False, so the handler asks the guardrail instead of the list. Every guardrail class gets the same answer as before * fix(bedrock): keep Singulr on flat Converse texts as on main Singulr wraps each flat text as a user message and its backend judges only the latest user message, so on main a Converse tool result was the turn it judged. Handed rows, the tool result arrives with role tool and the latest user row is the original question, so an injection in a tool result reached Bedrock. Singulr now returns False from supports_structured_messages * fix(bedrock): keep Qualifire on flat Converse texts as on main Qualifire builds its evaluation input from the last user row only when that row's content is a string, so Converse rows (content-block lists) gave an evaluation_id policy an empty input. Opt it out of structured messages so it gets main's flat texts again. * fix(bedrock): keep Conduct on flat Converse texts as on main The conduct-litellm-guard plugin scans only user rows, so with Converse rows a system prompt, assistant text or tool result left its scan. Opt it out of structured messages so it gets main's flat texts again, all sent as user turns. * fix(bedrock): refuse tool removals that leave a toolConfig Bedrock rejects A guardrail that removed every tool, or the tool toolChoice names, produced an empty tools list or a dangling toolChoice that Bedrock answers with an opaque ValidationException. Refuse those rewrites before Bedrock, like other tool changes this route cannot carry. * fix(bedrock): list request-field strings in main's order for every guardrail The request-field row walked additionalModelRequestFields breadth first, and guardrails that get rows had their texts follow that order. With two or more strings the last text changed, so Ovalix, which blocks only when the last text is flagged, rewrote an injection main blocked. The row now walks the fields in the order main lists them in the texts, and the texts use main's order for every guardrail. * test(bedrock): pin a container's own request-field strings before its nested ones * test(bedrock): pin tool_policy and MCP end-user filtering on Converse tools * fix(bedrock): rebuild Converse tools the guardrail changed and drop blanked text like chat A guardrail that adds, renames, reorders or retypes a tool now gets its list sent to Bedrock, converted the way chat completions converts tools, with cache points kept behind their tool. When every tool is removed, toolConfig is dropped and earlier tool calls and results become text, as chat does. Text blocks a guardrail blanks are dropped, and a turn left empty fails closed. The body copies used to place edits are JSON copies, so deeply nested tool JSON no longer hits the recursion limit. * test(bedrock): keep the client tool schema on Converse masks and refuse deep JSON text drops cleanly * fix(guardrails): keep Airia on flat texts for Bedrock Converse so its redactions still apply * fix(guardrails): a pipeline step whose rewrite the endpoint cannot apply follows on_fail on_error: allow sent the original request on when the endpoint refused a guardrail rewrite it could not apply, so a Converse request the guardrail masked reached Bedrock raw. The refusal now counts as a fail on every endpoint. * fix(guardrails): a pipeline step blocks when its rewrite cannot be applied, whatever on_fail and on_error say A rewrite the endpoint cannot apply is neither a content verdict nor a technical error, so on_fail next or allow and on_error allow must not send the original request on * test(bedrock): trim the Converse guardrail tests Drops the chaos and per-vendor integration cells and the duplicate unit cases, and leaves the existing handler tests as they are on main * fix(bedrock): put Converse request-field strings in the latest user message for row readers A guardrail that judges only the latest user message, like Noma v2, read the additionalModelRequestFields row placed after the conversation, so a request with extended thinking or anthropic_beta sent the user's card to Bedrock and let an injection through. Those strings now follow the latest user message's text in its row, and get a user row of their own only when no user message has text. * fix(bedrock): create toolConfig when a guardrail adds the first Converse tool A guardrail that returned a tool for a Converse request with no toolConfig got a 500, since the tool write-back only rebuilt an existing list. It now creates toolConfig, as chat completions does. Guardrails that get only flat texts have their returned tools ignored, as on main. * test(guardrails): check Converse flat-text opt-outs without patching apply_guardrail Each built-in opt-out is checked through its own supports_structured_messages(), the handler's flat path through an injected scripted guardrail, and Headroom through the real class, which would call its unreachable service if it got rows * fix(guardrails): keep Converse tool matching and write-back linear in request width Matching each returned tool rescanned the whole tool list, and every write-back step revalidated the list or object it passed through, so a request with thousands of tools or request fields held the event loop for seconds. Returned tools are now looked up by name in one index, and path steps read the child directly. * fix(guardrails): let later Converse guardrails scan guardContent text When a guardrail removes every tool on a Converse request that carries guardrailConfig, tool results are rewritten as guardContent blocks. The flat text extractor read only top-level text, so a later masker in the same request never saw them and a card in a tool result reached Bedrock unmasked. Chat completions and /v1/messages keep tool results visible to later guardrails after the tools are removed. The extractor now reads guardContent text the same way the structured rows already do, so flat-text guardrails also scan guardContent text a caller sends. * fix(guardrails): keep caller guardContent out of Converse flat texts and send tool history as plain text Flat-text guardrails read only top-level text blocks again, as on main. A guardContent block after the caller's text no longer becomes the last text, so guardrails that judge the last text (HiddenLayer, Qualifire) check what the caller wrote When a guardrail removes every tool, earlier tool calls and results are now written as plain text blocks instead of guardContent. Later guardrails still scan and mask them, and Bedrock's own guardrail keeps checking every message, since a single guardContent block makes it skip all plain text * fix(guardrails): split a newline-joined Converse message back into its parts A vendor that returns a multi-part message as one newline-joined string now has its masks written back to each part, guardContent included, when no part sent already holds a newline. * fix(bedrock): keep Converse tool history as tool messages until send and match chat on emptied turns When a guardrail removes every tool, the tool calls and results stay tool messages for the guardrails after it, so a scan_only_tool_results guardrail still masks them. The toolConfig with no tool is dropped and the history turned into text only when the request is signed, as chat completions does at request build. A caller's toolConfig with no tool gets the same treatment, where Bedrock returned 400 Request-field strings join the latest user message that has text when a tool-result-only turn follows it, so a guardrail judging only that message reads the caller's text with them A turn the vendor empties entirely is sent as written, as on main and chat completions, instead of a 500 Messages go to the guardrail through a JSON round trip instead of a deep copy, and a reply that changes nothing returns early, so a long Converse conversation guards about as fast as on /v1/messages --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
fc27dfdc7d
|
test(integration): give each owned proxy a private database and Redis by default (#47213)
* test(integration): give each owned proxy a private database and Redis by default owned_proxy and owned_proxy_process now boot on a scratch database and a per-proxy redis-server unless the caller opts into the worker's state with state=OwnedState.SHARED_WITH_WORKER, passes its own DATABASE_URL, or strips DATABASE_URL. Under the replica profile the private database gets the writer and reader roles so the reader path stays covered. Gateway carries the database_url its proxy writes to, and the Scenario cleanup checks and delete_key_if_present read from it, so they keep checking something on a private database. The 189 test files with owned proxy sites that depend on the worker's database or Redis opt in with a shares_worker_state module marker, which a module-scoped conftest fixture turns into the shared state. The 7 helper modules with such sites opt in per call, since conftest fixtures and other test files reach them. * test(integration): reach session fixtures with the shared marker and keep the worker retry default The module-scoped fixture that applied shares_worker_state ran after session-scoped fixtures, so proxies those booted went private in marked files. The marker is now applied around each marked item's setup, call and teardown, which covers fixtures of every scope. Owned proxies on the worker database inherited the worker gateway's router_settings.num_retries of 0 from it. A private proxy now gets the same default in its config copy unless the config sets num_retries itself. Five providers files read the owned proxy's spend rows or response cache through the worker's database or Redis and opt into sharing. |
||
|
yucheng-berri
|
b3c0f3457d
|
fix: handle missing choice indices in streamed output recording (#46947)
* fix: handle missing choice indices in streamed output recording * test: verify recorded partial output on client disconnect * test: correlate streaming regressions with persisted request IDs * test: build progress-stream configs without mutation --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
f6ae25939e
|
feat(guardrails): add num_retries to generic_guardrail_api for checks lost after connect (#46965)
* feat(guardrails): retry generic_guardrail_api checks lost after connect with num_retries * fix(guardrails): close each guardrail retry connection before the next attempt and cap num_retries at 5 A failed retry's fresh client stayed open until every later retry finished, so a check that kept timing out held one idle client per retry. Each retry now closes its own client as soon as it ends, and num_retries is capped at 5 so the retry chain stays shallow * refactor(guardrails): type the generic_guardrail_api inbound header sanitizer as a mapping Keeps the type check gate at its merge-base count after the num_retries constructor params added unknown kwargs errors in the Qohash subclass * chore(guardrails): regenerate OpenAPI snapshot and UI types for num_retries, allow the retry client The guardrail retry opens its own client on purpose to get a new connection and closes it after the attempt, so the per-request async client check allowlists generic_guardrail_api.py |
||
|
devin-ai-integration[bot]
|
8226b2be9a
|
fix(otel): report cache hits at zero cost with a cache-hit flag in V1 and V2 spans (#43604)
* fix(litellm): report zero cost and cache-hit attrs on cache-served spans
* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects
gauntlet-mini on
|
||
|
yucheng-berri
|
223800f916
|
fix(presidio): stop overlapping entity spans from corrupting numbered PII tokens (#47024)
* fix(presidio): resolve overlapping entity spans in numbered tokens anonymization (#42130) Mirror of BerriAI/litellm#42133 by @PINYOPATTANAWASANPORN, squashed onto main with the unit test moved to tests/unit after the test tree relocation * fix(presidio): label an overlapping run by its outermost detection, add integration coverage A run of overlapping detections now takes its entity type from the highest-scoring detection that no other detection in the run contains, matching Presidio's own anonymizer, so a phone number with a date-shaped fragment inside stays PHONE_NUMBER. The resolver returns typed detections instead of Any. * fix(presidio): drop the redundant comment in the numbered token loop * fix(presidio): bump a numbered token another part of the request already holds Each message and content part numbers its tokens from 1 into one request-wide pii_tokens map. Collapsing overlapping detections made an overlapping value's token well formed, so it collided with the same type in another part and both restored as the later value. A token that maps to a different value now moves to the next free number, and a token for the same value is reused * fix(presidio): give a clashing token a spare number instead of scanning for one Scanning upward from the clashing number probed every token an earlier part had claimed, so two parts with thousands of values of one type took seconds of synchronous work. No token in the map is numbered above the map's size, so numbers above both that size and the part's run count are always free * test(presidio): make the anonymizer double reject detections the analyzer never returned Pick the token in one conditional expression instead of reassigning it, and have the scripted anonymizer answer 500 when the proxy forwards spans that differ from the analyzer's, so the default-scope cells check what was sent --------- Co-authored-by: PINYO PATTANAWASANPORN <pattanawasanporn@gmail.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
46ae68021e
|
fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events (#46096)
* fix(anthropic): carry safeguard_results and stop_details in synthesized message_delta events When the proxy renders a complete Anthropic Messages response as an SSE stream for a client that asked to stream, the message_delta event carried only stop_reason and stop_sequence. A real Anthropic stream puts safeguard_results and stop_details inside message_delta.delta, so a client such as Claude Code auto mode lost the server-side classifier verdict and a refusal lost its stop_details. Both stream builders now copy those keys from the source response through one shared helper when they are present. * fix(guardrails): keep safeguard_results and stop_details when re-emitting a rewritten Anthropic stream A post-call guardrail that rewrites an Anthropic stream (presidio, bedrock, tool permission, model armor) now carries the upstream message_delta safeguard_results and stop_details into the re-emitted stream. The router fallback message_start no longer leaks those final fields early * fix(guardrails): keep stop_details and safeguard_results in rewritten Anthropic streams without explanations A guardrail that rewrites a /v1/messages stream now carries stop_details (type and category) and the dangerous_tool_use verdicts for the tool calls it still delivers, with every explanation dropped so a masked stream never echoes the unmasked text through those fields. * fix(guardrails): copy only upstream stop_details and forward-compatible safeguard_results into rewritten Anthropic streams A rewritten stream now takes stop_details and safeguard_results from the upstream message_delta alone, so a refusal the upstream sent without stop_details is never given one by the OpenAI round trip. Unknown safeguard types, statuses, reasons and extra fields pass through, a verdict follows a tool call only when its input is unchanged, and tool_permission keeps the explanations since it rewrites no text. * fix(guardrails): apply the safeguard_results and stop_details rules to policy pipeline stream rewrites * fix(router): deliver a refused stream built from a finished agentic turn instead of falling back Websearch, MCP and advisor streams replay a complete reply whose tools already ran, so a refusal fallback would run every tool again. Also give the recovered response's message_start a null stop_details as the API sends it, drop safeguard_results a masking rewrite cannot read, and add the audit cells for synthesized streams. * test(router): annotate the finished-turn refusal stream test locals with Final --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
c143954493
|
fix(streaming): report the deployment's api_base when a stream fails mid-way (#47045)
* fix(streaming): report the deployment's api_base when a stream fails mid-way * fix(responses): report the deployment's api_base when a native responses stream fails mid-way * fix(streaming): name the deployment's configured api_base in mid-stream errors and keep prompt text out Mid-stream errors on chat and native /v1/responses streams now prefer the router's configured api_base over the logged request URL, websocket turns keep the connection's api_base, and the mid-stream mapping no longer adds the prompt to the debug info. * fix(streaming): keep api_base credentials out of error debug info and leave websocket turn logging unchanged * fix(streaming): name the websocket connection's api_base on turn failures and strip scheme-less userinfo * fix(streaming): strip api_base userinfo before cutting the query so a raw ? or # in a password stays out Adds the audit's integration tests for the mid-stream alert API Base line and the SDK debug info. * fix(streaming): leave the API Base line out when an @ follows the host A raw / ? or # in a password and an @ in a query or path read the same in a malformed api_base, so either guess could show a secret or a wrong host. * fix(streaming): keep the API Base line when an @ sits in a provider path Vertex partner model ids like claude-opus-5-5@default put an @ in the URL path, so the fail-closed rule left out the line main printed. The line is now left out only when the text before the first / ? or # is not a host, and a :// counts as the scheme only after a valid scheme name. * test(streaming): cover a numeric password with a slash in the API Base line The audit found that a password like 1234/function-secret, which looks like a port, had no case. The line is left out on this branch and printed the raw secret on main |
||
|
yucheng-berri
|
9f115be88d
|
fix(bedrock)!: scan Claude invoke-with-response-stream passthrough output with post_call guardrails (#46524)
* fix(bedrock): scan Claude invoke-with-response-stream passthrough frames with post_call guardrails (LIT-9618) Invoke streams of Claude models now go through the same frame-by-frame relay as converse-stream. Each chunk frame's Anthropic event is scanned by the /v1/messages stream translation, a block ends the stream with an AWS exception frame, and a held stream gets its masks written back into re-encoded chunk frames. Claude is told by its model id or, behind an application inference profile, by the anthropic_version its request body carries. Streams of other model families stay as they were * test(bedrock): cover invoke stream vendor outage, upstream exception and release on scan (LIT-9618) * fix(bedrock): hold Claude invoke streams for guardrails the relay cannot run and mark each scan as streaming (LIT-9618) A guardrail released on scan or run in parallel left a Claude invoke-with-response-stream unscanned, while converse-stream holds the whole stream and scans it once. Claude invoke streams now take the same buffered path, with the rewrites written back into the chunk frames that carried each text. Each invoke scan also sends stream true and the events relayed so far as responses, as /v1/messages does, so a vendor can tie the scans of one streamed reply together. * refactor(bedrock): mark the request dicts the buffered stream paths hand to post_call hooks (LIT-9618) * fix(bedrock): scan a Claude invoke stream the upstream breaks off as a finished stream (LIT-9618) An invoke stream that ended on an upstream exception, without a message_delta stop_reason, was scanned as still in flight, so its tool input never reached the guardrail and a held stream released it unscanned. The scan and its scan key now close such a stream the way converse-stream treats the end of its upstream. * test(bedrock): pass the converse-stream endpoint where the tool call without input test frames its stream (LIT-9618) * test(bedrock): expect the ValidationException main now returns when a guardrail blocks a Claude invoke stream before its first frame (LIT-9618) * test(bedrock): type the Claude invoke event helpers in the passthrough guardrail tests * test(bedrock): type the remaining parameters of the Claude invoke stream tests * refactor(bedrock): bind the buffered invoke rewrite in its condition --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
ef1cb19351
|
fix(bedrock)!: stream Converse passthrough frames through post_call guardrails instead of buffering (#46307)
* test(integration): Converse stream post_call guardrail relays frames as they arrive (LIT-9515) * fix(bedrock): stream Converse passthrough frames through post_call guardrails instead of buffering (LIT-9515) A post_call guardrail that scans through apply_guardrail now runs on /bedrock/.../converse-stream the way it runs on /v1/messages: frames reach the caller as the upstream sends them, every fifth frame and the end of the stream are scanned, and a block after the first frame ends the stream with an AWS exception frame. streaming_buffer_until_moderated still holds the stream and applies masks per content block. Guardrails that need the whole response (own async_post_call_success_hook, mask_response_content, native hooks, policy pipelines) keep the buffered path. * fix(bedrock): apply held Converse stream masks when the upstream ends with an exception frame (LIT-9515) * fix(proxy): log a relayed passthrough stream's spend before its mid-stream failure hook (LIT-9515) * fix(bedrock): close and report every way a relayed Converse stream ends (LIT-9515) - The framer ends the stream with an empty item once the upstream closes, so held masks are applied whether the upstream ends with messageStop, an exception or error frame, or a dropped connection - A guardrail HTTPException after the first frame reaches the relay, which logs the stream's spend, runs the failure hook and ends the stream with the exception frame, instead of ending inside the guardrail with no failure row - A failure before the first frame waits for the stream's spend to be logged, as a mid-stream failure does - Closing the response before its body starts closes the guardrail layers and the upstream - Each stacked guardrail layer gets its own copy of every frame, so an earlier layer's end-of-stream rewrite never reaches frames a later layer still holds - Deltas whose text ends in half a surrogate pair are decoded with the standard JSON parser * fix(bedrock): relay a Converse frame whose payload is not UTF-8 instead of ending the stream (LIT-9515) * test(integration): cover Converse stream relay failure, end-of-stream-only, stacked, disconnect and burst cells (LIT-9515) * fix(bedrock): estimate prompt tokens for a passthrough stream cut before Bedrock reports usage (LIT-9515) Bedrock sends usage only in the last frame of a stream, so a stream ended by a guardrail block or a client disconnect logged 0 prompt tokens while AWS still charges for the prompt. The stream collector now estimates them from the request the way chat completions streams do * fix(bedrock): read tool result blocks without recursion in the passthrough prompt estimate (LIT-9515) * fix(passthrough): keep the spend flush running when the wait for it is cancelled (LIT-9515) * fix(bedrock): keep a converse-stream buffered when its guardrail releases a held stream on scan (LIT-9515) A held post_call guardrail with streaming_buffer_release_on_scan releases each window as received once its scan passes, so the relay delivered the original text instead of the guardrail's masks. Such a stream keeps the buffered path, which applies masks per content block as on main. * fix(bedrock): keep a converse-stream buffered when a post_call guardrail runs in parallel (LIT-9515) The relay ran every guardrail in config order, so a run_in_parallel guardrail listed before a masking guardrail scanned the unmasked text. Main's buffered path runs parallel guardrails after the sequential ones and discards their rewrites, which the relay cannot reproduce, so these streams stay buffered * fix(bedrock): mark Noma v2 relay scans like chat and stop reading Bedrock when a held stream's client leaves - Noma v2 scans on the converse-stream relay carry stream, and the last one carries the finished Converse response with its stopReason, as chat sends them, so Noma keeps one record per reply. Other guardrails get the request unchanged. When the last sampled scan already covered the whole answer, the end-of-stream scan still runs for Noma so it gets the finished answer - A held relay stream whose client leaves before the first frame stops reading Bedrock and returns 499, like chat. When Bedrock sent nothing yet, the budget reservation and the parallel-request slot are released - A cancelled wait for the first streamed chunk now stops the upstream read instead of leaving it running - A Noma v2 scan cancelled before Noma answers records not_run instead of success with an empty response * fix(bedrock): send Noma the finished Converse reply's tool calls and reasoning as Converse blocks * fix(bedrock): give Noma the ended scan of a Converse reply that only calls a tool without input * test(bedrock): expect the ValidationException main now returns for a held block before the first frame Main's AWS-shaped passthrough errors (#46522) answer a guardrail block before the first frame with x-amzn-errortype ValidationException, so boto3 reads the code and the guardrail message instead of a bare 400. * fix(bedrock): keep held masks when an earlier sampled guardrail ends the Converse stream with its message A sampled guardrail that blocks with disable_exception_on_block ends the stream with its message as raw event-stream bytes. A held guardrail after it on the same key only read frame objects, so it never saw the messageStop, skipped its write-back and released the held text without its masks. The Converse handler now reads those bytes back into frames, so the held guardrail sees the end of the message and masks the text * fix(proxy): free the parallel slot when a caller leaves a Converse relay holding part of the first frame The relay skipped releasing the slot and budget reservation whenever a spend flush was scheduled, but a flush of a partial first frame builds no response and logs nothing, so the slot stayed held until it expired. The flush now says whether it logged the stream. --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
jesus-berri
|
7f603da9f1
|
fix(openai_like)!: surface provider response headers on non-streaming chat completions (#46928)
* fix(openai_like): surface provider response headers on non-streaming chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(openai_like): mark the header test locals Final * fix(proxy): leave out response header values HTTP cannot carry Non-streaming OpenAI-like chat now forwards every provider header, so on the httpx transport a provider header value outside latin-1 made the proxy answer 500 after the provider had served the call. get_custom_headers now leaves such a value out and keeps the other headers, which also ends the same 500 on streaming Adds integration cells for the provider request id on non-streaming chat and the Responses bridge, LiteLLM headers an upstream cannot overwrite, spend parity under an upstream cost header, the latin-1 case, and the logging callback payload * fix(cost): ignore an upstream response cost header that is negative or not a number * test(integration): cover invalid upstream cost headers and non-latin-1 headers on every endpoint * fix(prometheus): keep deployment metrics when a provider rate-limit header is not a number * test(prometheus): type the rate-limit header test and drop its clock reads * test(integration): cover deployment metrics under an unparseable provider rate-limit header --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
910c30381b
|
test(integration): fix seven CircleCI integration tests that race or assume a worker spread (#47001)
* test(integration): expect the tool search call replayed beside the function call for Anthropic targets * test(integration): wait for the team spend flush before killing a worker * test(integration): poll for the S3 attempt the batch sinks write concurrently * test(integration): queue the terminal object and its sibling into the same s3_v2 flush * test(integration): time the long Bedrock version against an equally long unversioned name * test(integration): pin each half of a burst to one worker instead of hoping the kernel spreads it * test(integration): assert parsed SSE events for live pings instead of transport chunk boundaries |
||
|
ryan-crabbe-berri
|
0dac61a555
|
fix(guardrails): accept Gemini tools in generic_guardrail_api, add opt-in forward_messages_as_sent, name the error type on fail-open (#46338)
* fix(guardrails): accept Gemini tools and Anthropic content blocks in generic_guardrail_api, name the error type on fail-open GuardrailToolParam required a type key, so Gemini native tools (function_declarations, googleSearch) failed validation inside the hook and either returned a 500 or skipped the scan when fail_on_error was false. The type key is now optional and left out of the forwarded tool when absent, so an echoing guardrail server cannot write type: null into the provider request. structured_messages was re-validated against the chat completion message types, which silently emptied tool_result, tool_use, image and document content blocks to content: [] and dropped undeclared keys before the guardrail saw them. The request model now forwards rows as opaque mappings, which also retires the write-back shim that restored the original rows for echoed-back entries. The fail-open critical log line now carries error_type so a timeout, a refused connection and an upstream HTTP status can be told apart without the traceback. * fix(httpx): retry aiohttp server disconnects the way httpx does aiohttp raises ServerDisconnectedError when a kept-alive connection is closed by the peer before a response arrives, which is what a load balancer idle timeout shorter than AIOHTTP_KEEPALIVE_TIMEOUT produces. The transport mapped it to httpx.ReadError, which the handler's connection-error retry does not cover, so the request failed instead of being resent on a fresh connection. It now maps to httpx.RemoteProtocolError, the error httpx itself raises for the same condition, so the existing retry fires. * test(httpx): build the stub server reply in one bytes literal * fix(httpx): resend a dropped connection's request as sent and keep a streamed resend readable The connection-error retry closed its fresh client before a streamed body was read, so an aiohttp disconnect now billed a second upstream request and still failed. It also skipped masking on a status error from the resend, resent PATCH and DELETE as POST, dropped multipart files and ignored the handler's ssl_verify. The retry now resends the request the first attempt built, on a client with the handler's TLS settings, closes that client when a streamed body is closed, and masks a status error the same way the first attempt does. * fix(guardrails): log HTTPStatusError, not the masking wrapper, as the fail-open error_type * fix(guardrails): resend a generic guardrail check cut before any response A load balancer that closes an idle keepalive connection made the guardrail POST fail with "Server disconnected", because #46336 only resends POST on ConnectError. AsyncHTTPHandler.post now takes a keyword-only idempotent flag, defaulting to False, that gives a RemoteProtocolError the same single fresh-client resend PUT and DELETE get. Only GenericGuardrailAPI.apply_guardrail opts in, since a guardrail check is a pure verdict over the request. Every other POST keeps the #46336 rule The fail-open log line now puts error_type at the end so the pre-PR text stays contiguous for existing alert matchers test_rows_reach_the_guardrail_verbatim now reads the body off a real local guardrail server instead of mock call args * fix(http): keep a mid-body server disconnect a ReadError and move socket tests to integration A ServerDisconnectedError raised while reading a response body now stays httpx.ReadError, as it was before this PR. Only a disconnect before any response maps to RemoteProtocolError, which is the case the resend rule is about. Stream readers that end on ReadError, like the SAP stream iterator, keep their old behavior The loopback-server disconnect, TLS and guardrail resend tests move to tests/integration/sdk/test_disconnect_resend_wire.py, since tests/unit is in-process only. Unit keeps in-process MockTransport checks for the idempotent POST resend, masking on the resend, streamed and buffered client closing, the guardrail resend, and the guardrail receiving rows verbatim * fix(guardrails): resend a reset guardrail check and keep typed echoes on texts An idempotent POST now resends once on any network error after connect (ReadError, WriteError, RemoteProtocolError), so a guardrail behind a load balancer that answers a stale keepalive with a TCP reset rides it out too. That makes the aiohttp ServerDisconnectedError remap unnecessary, so it goes back to ReadError. This stops PUT and DELETE from stacking litellm's resend on top of aiohttp's own idempotent retry (4 sends instead of 2) and stops Vigil Guard from replaying its analyze POST on a disconnect Every resend now runs inside the method's own error mapping, so a timeout on the resend is a litellm.Timeout like the first attempt and a status error is masked once by the caller A guardrail server that parses rows with its own message types and echoes them back without the keys those types lack (annotations, tool_calls[].index) is read as an echo again, so its texts redaction reaches the LLM. A row only counts as an echo when no value changed and every key the chat completion types declare is kept The wire tests for the PUT/DELETE resend, its masking and its TLS settings now run on the httpx transport, where litellm's resend is the only retry, and the POST and guardrail tests cover both transports and both cut styles * test(guardrails): black box wire tests for generic guardrail shapes, fail-open error type and idle cut resend Integration tests against an owned proxy and scripted peers. Gemini native tools reach the guardrail and Gemini as sent, the guardrail sees chat rows and an Anthropic image block as sent, an echoing guardrail leaves a /v1/messages request unchanged, the fail-open log line names ConnectError, Timeout or HTTPStatusError, and a guardrail check cut on an idle pooled connection is resent once on each worker. Each fails at the merge base and passes at the head * fix(guardrails): check echoed rows without recursion and keep pytest.raises to one statement * refactor(guardrails): move the dropped-connection guardrail resend out of this PR The resend on a dropped or hung generic guardrail call, and the shared AsyncHTTPHandler retry rework behind it, move to a follow-up PR that adds a per-guardrail num_retries setting. This PR keeps the request and response shape fixes and the error_type on the fail-open log line * test(guardrails): pick the closed guardrail port from the integration port allocator Reading a port back from a port-0 bind trips TQ012, so the fail-open connect error case now takes its unused port from allocate_port * feat(guardrails): forward structured_messages as sent only behind forward_messages_as_sent generic_guardrail_api keeps main's OpenAI-format rows and echo comparison by default. Setting forward_messages_as_sent: true on the guardrail sends rows as the client sent them and treats an echo that only drops undeclared keys as unchanged --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c1a8575e58
|
test(integration): remove BUG skip for 1 test that passes on main (#46937)
test_two_identical_cache_hit_rejects_write_matching_spend_rows passed every run with the skip removed, so the dropped spend row it was guarding against no longer reproduces Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
f281f77bc0
|
fix(guardrails): record every content filter match in logging_only and offer the mode in the UI (#46673)
* test(guardrails): pin content filter logging_only matches and UI modes in an integration test * fix(guardrails): record every content filter match in logging_only and offer the mode in the UI * style(tests): format the content filter logging_only integration test * refactor(guardrails): keep the logging_only predicate and scan override within the type gates * fix(guardrails): flag tool arguments a content filter would reject in logging_only and keep scanning * test(integration): expect guardrail_flagged for an observe-only content filter block in the scope config test * test(integration): pin logging_only flags for tool arguments an enforcing content filter rejects * refactor(guardrails): drop the content filter flagged branch no logging_only scan can reach * test(integration): pin OTel spans, Guardrails Monitor and a provider outage burst for a logging_only content filter * test(integration): pin MCP tool call verdicts and fail the Monitor cell when usage never rolls up * test(integration): pin category and competitor intent verdicts and the provider failure row in logging_only --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
a3251a3bfd
|
fix(responses): preserve provider request IDs in streaming failure logs (#45453)
* fix(responses): preserve provider request IDs in streaming failure logs * fix(bedrock): log the provider request id for in-stream errors on bridged Bedrock streams Bedrock models that run through the chat-completions bridge (Claude, converse, invoke) raise in-stream failures from the event-stream decoder as a BedrockError without headers, so their failure logs kept error_provider_request_id null. The decoder now passes the stream's request-id headers to those errors. Both the native Responses iterator and the decoder pass only the PROVIDER_REQUEST_ID_HEADERS subset, so error responses and failure logs pick up the request id and none of the 200 response's other headers. MidStreamFallbackError gets its response at construction instead of mutating it afterwards. --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
cda19d6a29
|
test(integration): wait for the gen-AI span before reading the excluded-services operator trace (#46846)
The operator trace wait stopped at the root and redis spans, but the gen-AI span is closed later by the async success callback. When it had not landed yet, the tenant mirror was built without it and the gen-AI assertion failed. Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
b602209ca6
|
test(integration): hand every harness server a port from one per-worker block (#46761)
* test(integration): hand every harness server a port from one per-worker block
About 45 places in the integration harness picked a port by binding port 0,
reading the number back, closing the socket and starting a server on that
number later. Another process could take the port in between, which surfaced
as "Address already in use" flakes. Earlier fixes retried around the race in
single call sites without removing it.
tests/integration/_support/ports.py now owns every port the harness picks
ahead of a server. Each pytest process claims a 200-port block between 10000
and 32767 by taking an exclusive flock on a per-block lock file in the system
temp directory, so parallel runs on one machine never share a block and the
lock goes away with the process. The range is above the fixed service ports
and below the Linux and macOS ephemeral ranges. Ports go out in order, are
never reused in a run, skip anything an outside process already holds, and
a process that exhausts its block claims the next free one.
The three _free_port copies, worker_endpoints' port-0 binds,
free_loopback_port, the owned proxy's lost-port-race retry and the owned
Redis retry loop are gone. Tests that restart a server on the same port now
allocate that port up front instead of reading it back from the first
server. TQ012 in scripts/check_test_quality.py rejects a port-0 bind read
back with getsockname anywhere under tests/integration.
* ci(integration): record listening sockets and the ephemeral port range
* Revert "ci(integration): record listening sockets and the ephemeral port range"
This reverts commit
|
||
|
yuneng-jiang
|
46913be92b
|
test(integration): wait on the right signal in five flaky integration contracts (#46737)
* test(integration): wait on the right signal in five flaky integration contracts The Prometheus shared-label tests waited for the latency metric and then read the request counter, which the logger writes after an await. They now wait for the counter The auto-router session poll returned on the first 200 while the caller expected two turns, so it now waits for the expected turn count The S3 queue cap test let its failing sink recover on a 15s clock that could expire mid-flush. The sink now stays down until a flush fails after every event is queued K3 normally takes 168-175s against a 180s limit, so the limit is now 300s The MCP env var browser spec reused an idle keep-alive socket that the proxy closes after 5s, so its API calls now send Connection: close * test(integration): add until_asserted so a wait polls the exact asserts it guards eventually() takes a predicate separate from the asserts that follow it, which lets a test wait on one value and then assert on another written later. until_asserted reruns a block of asserts until they all pass and re-raises the last failure on timeout The Prometheus shared-label tests and the auto-router session check now use it |
||
|
devin-ai-integration[bot]
|
3d526bfe3f
|
test(integration): remove BUG skips for 3 tests that pass on main (#46725)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
30ada9d26a
|
feat(batches): store each batch line's request and response in S3 and GCS (#41691)
* feat(batches): emit per-line JSONL batch records to callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): only emit line items for final batches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(deps): bump soupsieve to 2.9.2 for GHSA-gjv8-xp57-g29c and GHSA-j934-xhv5-fg8f
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(batches): satisfy ruff format for line item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for store_batch_line_items_in_callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): cover line-item edge shapes and general settings propagation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep deployment credentials out of callback-visible params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(batches): declare credential transport attribute on Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): integration contract for per-line batch callback events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): resolve store_batch_line_items_in_callbacks through the settings store
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert per-line events beside the aggregate and create events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drop resurrected use_legacy_interactions_schema shim from merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): assert line item events carry no credentials
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reset batch line-item flag when the setting is absent
Removing store_batch_line_items_in_callbacks from the DB general settings left the module flag at its previous value because the effect early-returned when the key was neither in the DB nor owned by YAML. The proxy owns the global, so an absent resolved value now writes False.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): map anthropic batch lines through the anthropic message transformer
Anthropic result lines carry a message-shaped body, so ModelResponse(**body) produced empty choices and a usage key that blocked the batch-rate usage attach. Extract the existing AnthropicResponse -> ModelResponse conversion used for /v1/messages logging into anthropic_message_to_model_response and reuse it for anthropic line items.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: tag the anthropic result cast with a cast-ok reason
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: shorten cast-ok reason so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip line-item callbacks for providers without a line reconstruction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reconstruct bedrock and mistral batch lines through their provider transformations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap widened provider literals so ruff format passes
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): pair bedrock lines by recordId and type them from the reconstructed response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep line item events out of proxy limiters and strip credentials from child params
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep cast-ok tags on the flagged cast lines for the discipline gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip failed line item events in parallel request limiter failure hooks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): tolerate lifecycle spend rows when asserting no line item rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): place mutable-ok tags on the lines the discipline gate flags
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip router budget accounting for batch line item events
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep anthropic line errors and skip native vertex batch rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): restore test-quality tags on line item tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: move batch line-item unit test to tests/unit after merging main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): emit batch line-item callbacks once per batch
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when fan-out emitted nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): make the line-item claim release ownership-safe
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): keep bodyless failed batch lines reaching callbacks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(logging): seed batch line hidden params in get_hidden_params defaults
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* perf(batches): reuse fetched batch result files for line-item logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): release line-item claim when nothing was emitted
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): type batch line-item callback sites to satisfy basedpyright budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: wrap batch hidden-params cast for ruff format
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): reason suppressions for batch line-item lint deltas
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: drop stale mutable-ok suppressions after LIT002 removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters (#44486)
* fix(batches): skip batch line-item events in ClickHouse spend sink and router quota counters
Batch line-item callbacks (call_type=acompletion + litellm_params.batch_parent_id)
bypassed the call-type-only batch guards:
- ClickHouseSpendLogger wrote one spend row per line on top of the aggregate
aretrieve_batch row, billing every batch twice with the flag on.
- Router deployment callbacks and the routing-strategy usage handlers charged the
historical batch's tokens/rpm to the live TPM/RPM counters, so after
Router.aretrieve_batch the next chat raised RouterRateLimitError.
Skip these events wherever the aggregate batch retrieve is already skipped.
* fix(logging): keep internal litellm_params authoritative in model_call_details
A request body key named litellm_params lands in optional_params, and
update_environment_variables merged **optional_params after the internal
entry, so a caller could overwrite the logging kwargs' litellm_params and
forge markers like batch_parent_id. Live-reproduced: a Gemini chat whose
body carried {"litellm_params": {"batch_parent_id": ...}} returned 200
with no ClickHouse spend row. Re-assert the internal litellm_params after
the merge so every sink (ClickHouse, Postgres, router quota counters)
reads the same trusted value, and assert the aggregate aretrieve_batch
row still writes in the ClickHouse line-item test (review finding).
* fix(clickhouse): skip failed batch line items in the spend sink too
A failed JSONL line is part of the aggregate aretrieve_batch row's
request counts; writing it its own spend_logs row inflated request
counts with the line-item flag on. Extend the batch line-item guard to
the failure handler (review finding) and pin both directions: a failed
line item writes no row, a genuine live failure still does.
* fix(batches): declare batch_line_items service family on claim calls
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(integrations): keep batch line items out of the built-in metering sinks
With store_batch_line_items_in_callbacks on, per-line batch events carry a
real response_cost and reached six metering sinks that lacked the
is_batch_line_item_event guard, so a completed batch metered aggregate +
per-line (roughly 2x true spend): prometheus (litellm_spend_metric and
request counters), openmeter, lago, datadog cost management (FOCUS
BilledCost), newrelic metrics, and the OTel v1/v2 gen_ai.usage.cost
metrics. Skip line-item events in each sink's success/failure metering
entry points, matching the guards already on the Postgres and ClickHouse
spend sinks. OTel spans for line items are still emitted; only their
cost/token metrics are skipped. Tests drive real sink instances on the
real dispatch lists through a completed 2-line batch and assert each
sink meters exactly the aggregate.
* style: format metering-sinks test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(otel): skip synthetic line-item durations in v2 metrics; isolate prometheus test
Follow-ups from review on the sink guards: record_failure() now skips
batch line items too (a failed line's interval measures retrieval and
emission, not its model call), and the prometheus guard test snapshots
and restores the process-wide registry collectors instead of leaving
them unregistered. The OTel v2 test is now a flag on/off A/B over the
whole metric snapshot: cost and usage values must match exactly and no
metric may gain datapoints with the flag on.
* chore: finish main merge conflict resolution for batch line items
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: sort imports in litellm_logging after main merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(batches): use the public hook class names main now exports
* fix(batches): only the batch writer can mark a callback event as a line item
amoderation, transcription and speech build litellm_params from every request kwarg,
so a body field named batch_parent_id reached Logging.litellm_params and every
spend, budget and rate-limit hook skipped the request. update_environment_variables
now drops an incoming batch_parent_id; the line-item writer sets it through
Logging.mark_batch_line_item. Batch line hidden params are emitted only when set.
* chore(batches): keep the lint gates within main's ceilings after the merge
Typed the helpers main now types (Mapping litellm_params, Any Span), replaced the
banned cast with a literal match, dropped isinstance checks main made redundant,
and moved test-quality reasons onto the lines the gate reads.
* fix(anthropic): read the message id off the validated message
The anthropic_messages logging path can hand over an already parsed message object,
which has no .get; main read its id through getattr, so take it from the validated
AnthropicResponse instead.
* fix(batches): an unreadable error file no longer drops the output line events
Costing counts an error file it could not fetch as no failed lines; line logging
refetched it and the raise aborted the whole fan-out, so the successful lines were
never emitted. The error file now reads as empty with a warning.
* test(batches): pin billing for batch_parent_id on every endpoint and value shape
Chat completions, embeddings, responses and messages join moderations, speech and
transcriptions, and speech is driven with an empty, int, list, object and 5 KB value.
* test(batches): bill every request of a concurrent burst mixing forged and plain bodies
* revert: keep the line-item fan-out all-or-nothing on an unreadable error file
|
||
|
yuneng-jiang
|
0ececab7a1
|
test(integration): update stale CircleCI integration expectations (#46570)
Brings integration tests in line with intentional behavior changes from #46331, #46291, #46341, #46355, #46368 and #46493, moves the Databricks non-Claude cases off llama-3 names so they no longer download a Hugging Face tokenizer, drops the unrealistic count_tokens tools=5 case and skips the Mantle region-prefixed v1 base case as unrealistic input |
||
|
adityajoy-1902
|
20da5b949c
|
feat(proxy): return the W3C traceresponse header on traced responses (#42079)
* feat(proxy): expose trace ID response header * style(otel): format trace correlation middleware * test(otel): cover websocket trace middleware * fix(otel): disable correlation on setup failure * refactor(otel): inject instrumentation setup * fix(proxy): drop a relayed upstream traceresponse when the gateway sends its own * fix(proxy): preserve upstream traceresponse without gateway injection * fix(proxy): replace relayed streaming traceresponse * test(otel): cover trace response fallback paths * test(proxy): cover the streaming pass-through relay when the gateway propagator is off * fix(proxy): keep a relayed upstream traceresponse and stamp the call id on the server span The gateway's response propagator now leaves a response that already carries an upstream traceresponse untouched, so pass-through, file content and plugin proxy routes return one header and untraced pass-through routes keep the upstream's. The per-helper strips are gone. The propagator also sets litellm.call_id on the server span from the response's x-litellm-call-id. * fix(proxy): add traceresponse only to sampled HTTP responses, tuple headers and unhandled 500s included * fix(proxy): write only traceresponse and link a relayed header from a restarted upstream trace * fix(proxy): send traceresponse only from the proxy app and expose it through CORS only with OTel V2 * test(proxy): check the CORS expose list contains the UI headers instead of pinning its identity * test(integration): check traceresponse names the gateway server span across routes, errors, pass-through and chaos * test(integration): read the cache-hit spend row by response id and assert only the traceresponse expose entry --------- Co-authored-by: adityajoy-1902 <bittu19jain@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
6e0f0f7adb
|
fix(s3_v2): bound each upload attempt, add jitter, retry 429 and transport errors in the call (#46340)
* fix(s3_v2): bound each upload attempt, add jitter, retry on 429 and 503 Each S3 PUT now carries a per-attempt httpx timeout read from S3_UPLOAD_ATTEMPT_TIMEOUT_SECONDS (default 30s, clamped to 1..3600) instead of inheriting the handler's unbounded default. Retries back off with jitter instead of a fixed 1s then 2s, 429 joins 403, 500 and 503 in the in-call retry set, and transport errors or litellm.Timeout on an attempt are retried in the same call instead of ending it. Each flush logs one warning per batch of dropped uploads that names how many were terminal and how many aged out. * fix(s3_v2): retry uploads in a bounded loop and align tests with the new retry contract The async and sync retry helpers called themselves, which the recursive-function check rejects. They are now a bounded loop over the attempt count with the same outcome. The 429 integration test now expects in-call jittered backoff, and the aged-out test matches the new drop warning text. The sync transport test drives the real HTTPHandler so a dropped per-attempt timeout or a missed litellm.Timeout fails it, and the drop-reason test uses unequal counts so swapped labels fail. --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
Mateo Wang
|
d81a5719ad
|
fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359)
* test(integration): follow count_tokens error surfacing and decisions pricing in CircleCI suites Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): restore Responses input_file PDF pricing coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep integration AGENTS guide unchanged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cache): preserve provider field presence on cache hits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect credential display name in response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): align OpenInference refusal field expectations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect Akto request ingestion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): expect missing tool error wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): expect no refusal field in cursor chat wire Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
cfa98e2f40
|
test(integration): cover pass_through endpoint contracts (#44690)
* test(integration): cover pass_through endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): run configured endpoint contracts on one proxy worker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): prove pass-through streams, SDK spellings, endpoint renames and bedrock stream bodies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover pass-through wire behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover Vertex and Gemini passthrough authentication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Prove pass-through runtime method, credential-less Anthropic, stream barriers and Vertex strip layers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(pass_through): gate unquoted Bedrock stream targets and assert configured endpoint part types and methods after rename Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): assert pass-through query pairs so duplicate params fail Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
02f56c6319
|
test(integration): cover guardrails_policies endpoint contracts (#44722)
Some checks are pending
unit / core-utils (push) Blocked by required conditions
unit / endpoints (push) Blocked by required conditions
unit / enterprise-package (push) Blocked by required conditions
unit / enterprise-repositories-secrets (push) Blocked by required conditions
unit / integrations (push) Blocked by required conditions
unit / llms-anthropic (push) Blocked by required conditions
unit / llms-bedrock (push) Blocked by required conditions
unit / llms-openai-meta (push) Blocked by required conditions
unit / llms-providers (push) Blocked by required conditions
unit / llms-vertex-ai (push) Blocked by required conditions
unit / mcp-elicitation (push) Blocked by required conditions
unit / mcp-server (push) Blocked by required conditions
unit / proxy-auth (push) Blocked by required conditions
unit / proxy-endpoints (push) Blocked by required conditions
unit / proxy-extras (push) Blocked by required conditions
unit / proxy-feature-endpoints (push) Blocked by required conditions
unit / proxy-hooks-client (push) Blocked by required conditions
unit / ui-unit (push) Waiting to run
unit / docs (push) Waiting to run
unit / helm (push) Waiting to run
unit / OpenAI 2.20.0 compatibility (push) Waiting to run
unit / OpenAI 3.0.0 compatibility (push) Waiting to run
unit / OpenAI 3.25.0 compatibility (push) Waiting to run
unit / redis-py 5.3.1 compatibility (push) Blocked by required conditions
unit / redis-py 6.4.0 compatibility (push) Blocked by required conditions
unit / redis-py 7.4.1 compatibility (push) Blocked by required conditions
unit / redis-py 8.0.1 compatibility (push) Blocked by required conditions
unit / coverage (push) Blocked by required conditions
unit / unit passed (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* test(integration): cover guardrails_policies endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): let the admin submissions list carry other rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): close guardrails policies review gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover guardrail auth and policy attachments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tidy guardrail policy test helpers * test(integration): keep plain non-admin submission refusal live * test: expand policy attachment integration coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): accept the intended 403 for plain submission keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin multi-tag header and spend row tags for policy tag scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the duplicate pydantic import left by the main merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Devin <noreply@devin.ai> |
||
|
devin-ai-integration[bot]
|
d3cc77def4
|
test(integration): cover chat_completions endpoint contracts (#44688)
* test(integration): cover chat_completions endpoint contracts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin per-call prompts and both n=2 chat choices Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover mixed path-model auth, string guardrail rejection and typed text-completion forms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): drop the prompt-spellings cache workaround and cover Azure SDK text completions on the deployments alias --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
b581ebc405
|
fix(prometheus): skip custom metadata labels a metric already carries (#46267)
* fix(prometheus): skip custom metadata labels a metric already carries A custom metadata label whose sanitized name matches one of a metric's built-in labels was appended a second time, so every .labels() call on that metric raised ValueError. With custom_prometheus_metadata_labels: [project_alias], the per-project per-model rate limit gauges raised on every request from a project key with per-model limits, and the rest of the success event (latency, deployment success, total requests) was skipped. * test(prometheus): cover shared custom labels on every endpoint, failure, metadata and outage burst * test(prometheus): drain the shared upstream at the start of each integration test * test(prometheus): assert the forwarded route, model and stream per endpoint cell --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
0b3fcc197d
|
fix(logging): redact nested callback input and provider response fields (#45923)
* fix(logging): redact nested callback input and provider response fields * fix(logging): keep shared request body for per-callback redaction and drop hidden provider fields The deprecated per-callback message_logging=False path ran perform_redaction on the shared model_call_details, which now also redacted the nested proxy_server_request body. The spend log Request column and previous_response_id replay read that body, so one opted-out callback blanked them for everyone. That path now keeps the shared body and hidden params, and redact_streaming_responses_for_custom_logger hands the opted-out callback its own redacted litellm_params. Redacted litellm_params also clear provider_specific_fields copied into metadata.hidden_params (Anthropic thinking text), in litellm_params and in the nested request body, and the nested body redacts system and instructions. * fix(logging): keep replayed instructions and router usage marker under redaction The shared request body keeps system and instructions again, so spend logs store them and previous_response_id follow-ups replay the real instructions. Per-callback copies still redact them. Provider-specific fields in metadata.hidden_params are cleared in place, so the metadata dict keeps its identity and the router's counted-usage marker stays visible to deployment_callback_on_success (no double tpm/rpm count). * fix(logging): never copy shared metadata dicts during redaction The shared redaction pass now leaves litellm_params metadata dicts as they are, so a hidden_params write from the other success handler between the in-place clear and the copy can no longer move the router usage marker to a stale dict. The key-presence predicate change is reverted as unneeded. * refactor(logging): redact nested request body and provider fields with the existing message rules Run perform_redaction on a copy of proxy_server_request.body, the same call the spend log already makes, and redact reasoning inside provider_specific_fields the way the message's own reasoning_content and thinking_blocks are redacted, instead of clearing every provider field. Opaque fields such as Gemini thought_signatures stay, and the metadata dicts and streaming objects are handled as on main. --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |