mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-15 04:19:08 +00:00
91 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
adityajoy-1902
|
20da5b949c
|
feat(proxy): return the W3C traceresponse header on traced responses (#42079)
* feat(proxy): expose trace ID response header * style(otel): format trace correlation middleware * test(otel): cover websocket trace middleware * fix(otel): disable correlation on setup failure * refactor(otel): inject instrumentation setup * fix(proxy): drop a relayed upstream traceresponse when the gateway sends its own * fix(proxy): preserve upstream traceresponse without gateway injection * fix(proxy): replace relayed streaming traceresponse * test(otel): cover trace response fallback paths * test(proxy): cover the streaming pass-through relay when the gateway propagator is off * fix(proxy): keep a relayed upstream traceresponse and stamp the call id on the server span The gateway's response propagator now leaves a response that already carries an upstream traceresponse untouched, so pass-through, file content and plugin proxy routes return one header and untraced pass-through routes keep the upstream's. The per-helper strips are gone. The propagator also sets litellm.call_id on the server span from the response's x-litellm-call-id. * fix(proxy): add traceresponse only to sampled HTTP responses, tuple headers and unhandled 500s included * fix(proxy): write only traceresponse and link a relayed header from a restarted upstream trace * fix(proxy): send traceresponse only from the proxy app and expose it through CORS only with OTel V2 * test(proxy): check the CORS expose list contains the UI headers instead of pinning its identity * test(integration): check traceresponse names the gateway server span across routes, errors, pass-through and chaos * test(integration): read the cache-hit spend row by response id and assert only the traceresponse expose entry --------- Co-authored-by: adityajoy-1902 <bittu19jain@gmail.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Tobias Mayr
|
d03923e6a9
|
fix(responses): propagate cache_control through responses->chat completion bridge (#32511)
* fix(responses): propagate cache_control through responses->chat completion bridge User/assistant messages, tool call outputs, and function calls built from Responses API input items were dropping cache_control, breaking prompt caching for Bedrock/Anthropic when routed through the litellm_completion transform. Also propagate cache_control onto the last content element of user/assistant messages, since Anthropic/Bedrock read it from the content block rather than (or in addition to) the message level. Adds the missing cache_control field to ChatCompletionToolMessage and ChatCompletionResponseMessage TypedDicts to keep the assignments type-safe. * chore(ui): regenerate schema.d.ts for ChatCompletionToolMessage.cache_control Adding cache_control to the ChatCompletionToolMessage TypedDict changed the proxy's OpenAPI spec; regenerated via `npm run gen:api`. * fix(responses): don't overwrite existing content-block cache_control; add list-content test Per Greptile review on the PR: - The message-level cache_control was unconditionally overwriting the last content block's cache_control when that block already carried its own (preserved by _transform_responses_api_content_to_chat_completion_content). Now only propagates onto the last block when it doesn't already have one. - Added the missing test coverage for the content-block propagation path with list content, which no existing test exercised (all used plain string content, so the isinstance(content, list) branch was never hit). * style(responses): extract cache_control into a named variable, trim comment Per style review: the message-building branch read cache_control 3 different ways in 5 lines (.get() twice, then [] indexing once), unlike the sibling tool-call-output/function-call functions this PR touches, which each extract it into a named variable up front (output_cache_control, function_call_cache_control). Matched that convention here too. Also trimmed the 3-line inline comment to one line, matching the file's usual comment length for similarly subtle behavior. * chore: refresh lazy OpenAPI snapshot * fix: type cache control propagation * fix: place cache control type suppressions * style: format cache control type conversion * refactor(responses): build bridged cache_control messages in one shot * fix(responses): keep cache_control marks on bridged tool calls and within the provider cap A function_call item's mark now rides on the tool call the bridge builds, an assistant text item merged behind a function_call keeps its own mark, a developer item keeps its mark when it is mapped to a system message, and a previous_response_id replay yields its stale marks to the newest ones so the provider cap holds. The Responses layer counts the marks on function_call and function_call_output items before it places an ordinal injection point, and the chat pass replaces that count with its own instead of adding to it. * fix(responses): count each cache_control mark once and keep the bridge within the provider's limits List content carries an item-level cache_control on its last block only, so the provider sees one cache point and the breakpoint census counts one. Marked function tools and SDK-typed input items are handled by the Responses-layer reserve, assistant text merged into a function call turn keeps its mark on its own block, and the previous_response_id replay budget drops marks per turn, newest kept, instead of stripping every older turn once one overshoots. The request's own marks now fall under the same budget, newest kept, so five marked items no longer answer the provider's 400. A cache_control value that is not an ephemeral mark with an optional ttl is left off the bridged message instead of reaching the provider as sent, and a developer message maps to the system role with its content, name, and cache_control only, so unknown keys never reach a strict provider. * fix(responses): send item cache_control only to cache-point targets and follow drop_params Item marks reach the provider only when the target turns them into cache points, and every other target keeps main's behavior of stripping them. Without drop_params a mark the provider would refuse reaches it as sent, so the client gets the same 400 as on /v1/chat/completions. With drop_params a malformed mark is dropped and only the newest 4 are kept. The shared input-to-messages helper and the developer mapping in base_utils go back to main, and a marked developer item reaches a cache-point target as a system message in the bridge. * fix(responses): budget top-level marks, follow drop_params on Nova tool marks, and match chat-keeping providers * fix(responses): validate replayed marks and count input tokens without item marks * fix(responses): leave unknown Bedrock ARNs unmarked under drop_params, match OpenRouter's list, and reserve only forwarded marks * test(responses): cover item cache_control parity per chat route, drop_params trimming, and replay budgets * fix(responses): keep item marks off inspection paths and match each chat route's keep rule Only the request the bridge sends gets item marks now. Guardrails, token counting, rate limiting, and input_tokens read the input with no item marks, as on main, so an off-spec mark cannot fail a guardrail call. Tool message and tool call marks are typed inside the bridge, leaving the shared chat types and guardrail schemas as on main. Databricks keeps marks only for Claude, Snowflake uses its chat route's Claude check, and every Anthropic deployment keeps them whatever its model name. The injection census skips SDK-typed tools, and the owned-proxy test expects the marked messages on /v1/responses. * test(responses): cover guardrail reads without item marks, Snowflake name parity, and SDK-typed tools in the census * fix(responses): leave a non-list input untouched when stripping item marks * test(responses): cover string, list, object and tuple input with and without item marks kept Each input shape now answers the same whether the unmapped Bedrock ARN keeps item marks or strips them under drop_params, through the proxy and the SDK. Object input also runs through the OpenAI-compatible chat bridge, /v1/responses/input_tokens and the project input-token limiter * test(cache): derive owned-proxy cell timeouts from the harness boot budget The capped-burst cells and the uncapped response-cache cells boot their own two-worker proxy, and a fixed 240 or 360 second timeout cut them during boot on a slow runner before the harness readiness budget could decide. They now carry the same derived budget as the sibling chaos and guardrail files, with four boot and stop budgets for the restart cell that boots twice --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Kerry Lu
|
f10134728d
|
fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger (#46484)
* fix(caching): collect the Redis circuit-breaker state gauge as one series per state across worker processes The gauge declares multiprocess_mode=livesum, so PROMETHEUS_MULTIPROC_DIR apps get one series per state summed over the live workers instead of one series per worker pid Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): register Redis circuit-breaker metrics only when LiteLLM prometheus logging is on The breaker collectors join the default registry once the prometheus or prometheus_system callback is configured or a Prometheus logger starts, with earlier breaker events tallied and seeded in, and litellm.disable_redis_circuit_breaker_metrics turns them off for apps that export their own metrics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): drop the disable_redis_circuit_breaker_metrics opt-out Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): keep the Redis breaker metrics gate to a per-state count and a publish hook Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): pin the Redis breaker metrics gate against an injected registry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): publish the breaker collectors before reading them in the existing metrics test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): report Redis circuit-breaker metrics through the Prometheus services logger Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(integrations): build the Redis breaker collectors with typed prometheus_client constructors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
6e0f0f7adb
|
fix(s3_v2): bound each upload attempt, add jitter, retry 429 and transport errors in the call (#46340)
* fix(s3_v2): bound each upload attempt, add jitter, retry on 429 and 503 Each S3 PUT now carries a per-attempt httpx timeout read from S3_UPLOAD_ATTEMPT_TIMEOUT_SECONDS (default 30s, clamped to 1..3600) instead of inheriting the handler's unbounded default. Retries back off with jitter instead of a fixed 1s then 2s, 429 joins 403, 500 and 503 in the in-call retry set, and transport errors or litellm.Timeout on an attempt are retried in the same call instead of ending it. Each flush logs one warning per batch of dropped uploads that names how many were terminal and how many aged out. * fix(s3_v2): retry uploads in a bounded loop and align tests with the new retry contract The async and sync retry helpers called themselves, which the recursive-function check rejects. They are now a bounded loop over the attempt count with the same outcome. The 429 integration test now expects in-call jittered backoff, and the aged-out test matches the new drop warning text. The sync transport test drives the real HTTPHandler so a dropped per-attempt timeout or a missed litellm.Timeout fails it, and the drop-reason test uses unequal counts so swapped labels fail. --------- Co-authored-by: yuneng-jiang <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
aa0279e0a2
|
feat(anthropic): generate auto mode safeguard_results with a classifier model on the /v1/messages chat bridge (#45600)
* feat(anthropic): generate auto mode safeguard_results with a classifier model on the /v1/messages chat bridge * fix(anthropic): check caller limits for the safeguards classifier, mark truncated tool calls, and keep pre-compaction context * fix(anthropic): log classifier spend under the caller's key hash and read list-shaped classifier replies * feat(anthropic): generate safeguard_results on the Responses bridge and native non-Claude /v1/messages providers * fix(messages): keep safeguards requests on Python under the Rust opt-in * fix(anthropic): keep non-ASCII text unescaped in the safeguards classifier prompt * test(anthropic): build the text-parts classifier reply without openai's construct path Message.model_construct goes through openai's construct_type, which caches discriminated unions in a weak-key map. On Python 3.12 a PEP 604 union cannot be weak-referenced, so the fixture raised inside the fake classifier and the test read an unavailable verdict. model_copy sets the list content without that path. * fix(anthropic): scan only new bytes for SSE frame ends on safeguarded native streams * fix(anthropic): answer safeguards on web search short circuits and widen the classifier reply cap * fix(anthropic): answer unsupported when the proxy serves no deployment for the safeguards classifier * fix(anthropic): keep a backend's stream verdicts and join split SSE data lines before stamping * fix(anthropic): map only the router's no-healthy-deployments refusal to unsupported * fix(anthropic): answer unavailable for the turn on a transient caller denial and let the classifier deployment's timeout apply * fix(anthropic): keep the user's whole opening message in the classifier transcript past its budget * fix(anthropic): run the proxy's own access checks for the safeguards classifier call The classifier and compaction sub-calls now pass through the same model access checks as the parent request (key, team, team member, user, project, end user, managed agent grants), with budgets left to the per-model budget check so nothing is reserved. They also carry the parent's logging params and headers, so a redaction header or per-request callback params apply to their own rows. Only per-model RPM limits count against a sub-call now, since a key-wide or team-wide RPM window is already spent by the parent request. The MCP auto-execute loop classifies only its final response, a chat-bridge stream without safeguards yields frames directly, and a classifier provider failure logs one warning line instead of a traceback * fix(anthropic): keep safeguards sub-calls off the parent's session routing and carry a body trace id The classifier and compaction summary calls no longer get the parent's proxy_server_request, so a Claude Code session bound to a router no longer pulls them off their configured model. Redaction still reads the headers carried in metadata. Their access checks now use the parent's resolved litellm_trace_id when the parent metadata has no trace_id, so a team that requires a trace id accepts a sub-call whose parent sent it in the body. * fix(anthropic): keep the disable-callbacks header on safeguards sub-calls * fix(proxy): charge per-model budgets from litellm_metadata when metadata holds only routing keys * test(integration): add safeguard_results cells for the bridges, native routes, caller checks, and classifier failures * fix(proxy): check the caller's budgets on the classifier call and keep failed access reads retryable --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
b581ebc405
|
fix(prometheus): skip custom metadata labels a metric already carries (#46267)
* fix(prometheus): skip custom metadata labels a metric already carries A custom metadata label whose sanitized name matches one of a metric's built-in labels was appended a second time, so every .labels() call on that metric raised ValueError. With custom_prometheus_metadata_labels: [project_alias], the per-project per-model rate limit gauges raised on every request from a project key with per-model limits, and the rest of the success event (latency, deployment success, total requests) was skipped. * test(prometheus): cover shared custom labels on every endpoint, failure, metadata and outage burst * test(prometheus): drain the shared upstream at the start of each integration test * test(prometheus): assert the forwarded route, model and stream per endpoint cell --------- Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
0ebec123fd
|
chore(typing): clear 670 basedpyright Any errors by validating provider payloads at the boundary (#45388)
* chore(typing): validate provider payloads at the boundary to clear 464 basedpyright Any errors Replaces untyped reads of decoded JSON in 48 provider, guardrail and proxy modules with one pydantic validation at the point the payload enters the module, so every value read afterwards carries a real type. reportAny 6111 -> 5663 and reportExplicitAny 1416 -> 1400. No basedpyright, ruff-strict or type-discipline rule is higher repo-wide or in any single file, and no cast, ignore or suppression marker was added. * chore(typing): replace 47 Any annotations that every caller already satisfies with object Each annotation here is a parameter or return type whose value is only stored, compared, logged or forwarded, so object describes it exactly and no function body changes. A site was kept only if no basedpyright, ruff-strict or type-discipline rule rose in any file that references it. reportExplicitAny 1400 -> 1353 and reportAny 5663 -> 5655. * chore(typing): type upstream responses and loose annotations across providers, proxy and integrations Second pass of the Any reduction. Decoded upstream JSON is validated once at the provider boundary and read through typed views afterwards, annotations that every caller already satisfies move from Any to a concrete type, and the nine search transformations validate each result item on its own so error text no longer carries the item index. No cast, no type: ignore, no suppression marker and no budget file edit * ci(unit): run the brave, google_pse, nlp_cloud and opensandbox tests in the providers shard |
||
|
yuneng-jiang
|
1587faeeb9
|
test(unit): wait on the work itself instead of fixed sleeps (#46261)
* test(unit): wait on the work itself instead of fixed sleeps in 21 flaky files Replace sleeps that raced background logging, cache writes, refreshes and thread work with either awaiting that work directly (logging worker flush, the batch logger's flush_queue, the tasks the call spawned, thread joins) or an Event the test's own fake sets when it runs. Each wait keeps a 5s hang guard that never decides pass or fail. Tests that need a fake clock, real sockets, or measure latency are left unchanged, and no production code moves * test(unit): await the tasks each completion spawns instead of yielding once The logging tests yielded once with asyncio.sleep(0) so the task that enqueues the success log could run before the worker flush. That relied on the enqueue happening before the task's first await. Snapshot the running tasks around each acompletion call and await the ones it spawned, so the guarantee holds however many steps sit before the enqueue |
||
|
yucheng-berri
|
a5d05355b6
|
feat(focus): add ternary destination for FOCUS export (#46106)
* feat(focus): add ternary destination for FOCUS export Add a `ternary` FOCUS export destination (plus a `ternary` callback) so a LiteLLM proxy can push FOCUS-formatted spend to Ternary with `callbacks: ["ternary"]` -- no separate export job - FocusTernaryDestination: a pure sink modeled on the sibling vantage/mavvrik/gcs destinations; POSTs the FOCUS CSV to the Ternary cost-ingestion endpoint, chunking an oversized backfill (csv-module repartition, quoted-newline safe; oversized single row raises rather than silently dropping) with a stable X-Ternary-Upload-Id + 0-based index/total so the receiver stages-then-swaps. connection_id is validated + URL-encoded; base_url must be HTTPS (loopback exempt for local); the upload has a bounded timeout; it aborts on the first failed chunk so nothing partial lands - TernaryLogger: presets the FOCUS pipeline (CSV) and enriches the FOCUS Tags JSON with the per-request token counts the shared transformer drops (Ternary weights allocation by tokens); the enrichment is confined to this path and only adds keys. Cadence is daily (default) or interval (validation loops, an explicit interval-seconds required); the export window is day-aligned and drops days before the window start so a whole-day-replace receiver never truncates - Registration mirrors vantage/mavvrik: ternary callback literal, logging dispatch, scheduler job, and CustomLoggerRegistry entry -- no shared-code changes - Tests: mocked unit tests only (no live calls), 100% line coverage of both new modules * fix(focus): warn instead of silently dropping token tags on mismatch _merge_token_tags returned the frame unchanged when the transformed data had no `Tags` column or its row count disagreed with the source, so every token count was dropped with no signal -- the receiver then lands NULLs that read as "this gateway reports no tokens". Emit a warning on each degenerate path (mirroring _drop_days_before), and cover both with caplog tests asserting the warning and the unchanged return. * feat(focus): upload one UTC day per Ternary signed-URL PUT Replace the multipart POST with chunking by a per-day signed-URL upload, matching the Mavvrik destination's pattern. The logger partitions the raw usage rows on `date` (dropping unparseable dates with a warning), then transforms, enriches and delivers each day separately; one day's failure is logged and the remaining days still upload, with a single error raised at the end. The destination requests a signed URL for the day, validates it is an HTTPS GCS URL, and PUTs the CSV with exactly the signed headers. Removes the 30 MiB / 100k-row chunking, the X-Ternary-* upload headers and the .partN filenames; a re-sent day overwrites its object, so retries stay idempotent without them. * fix(focus): re-send the last two whole days so late-flushed spend lands LiteLLM writes DailyUserSpend in batches, so spend dated yesterday can be flushed after the 00:05 UTC daily export. That export only covered the previous UTC midnight onward, and the next day's export moved past it, so the tail of each day was never sent. Start the window at UTC midnight two days back: the following run re-sends yesterday whole, overwriting its per-day object with the complete day. * fix(focus): lazy-load the Ternary engine, fetch whole days, keep ternary out of the generic factory Every proxy imported polars at boot through the logger registry, so a proxy without polars failed to start even with no FOCUS callback. The polars-backed engine now lives in its own module and loads on the first export A non-UTC Postgres session timezone shifted the naive updated_at filter, so the oldest day in each run lost rows and Ternary's whole-day replace made the loss permanent. The Ternary read now starts one day early with no upper bound and keeps rows by their own UTC date callbacks: ["focus"] with FOCUS_PROVIDER=ternary uploaded a parquet blob of one interval over a whole-day CSV file. The generic factory no longer builds a Ternary destination and the Ternary engine injects its own * fix(focus): keep only the days a Ternary export window covers Dropping the updated_at upper bound let a bounded export_usage_data window upload every day through today. Rows are now kept by their UTC date between the window's first day and the day before its end instant * refactor(focus): leave the shared FOCUS export engine as on main The Ternary engine sets the base engine's fields itself and passes its own destination, so focus/export_engine.py and its test match main again. --------- Co-authored-by: Aaron Sproul <aaron@ternary.app> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
mrinal-berri
|
231721f20a
|
feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination (#45880)
* feat(otel): let a team or key excluded_services override the global setting for its OTel v2 destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: ruff format otel excluded_services changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: use raw regex for excluded_services match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): reject non-string destination callback vars Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): harden destination callback var validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): compare parsed values in callback var conflict check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
c2d0726a08
|
fix(anthropic): carry cache_control to prompt_cache_breakpoint on the /v1/messages to Responses bridge (#46082)
A caller hitting /v1/messages with a cache_control mark and an OpenAI gpt-5.6 or newer deployment behind it lost the mark when the request was rebuilt as a Responses request, so OpenAI never cached at the boundary the caller chose. The bridge now rewrites each marked system, user and tool_result block to prompt_cache_breakpoint on the same block when the deployment reads breakpoints, keeps the system prompt as a developer message so its mark survives, and still strips the mark for deployments that do not read it The hook's gate becomes public so the bridge and the hook share one answer for which deployments read breakpoints, and the one-block swap lives next to the existing breakpoint helper in prompt_templates |
||
|
devin-ai-integration[bot]
|
a7a3a6b375
|
test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e (#45940)
* test: migrate legacy unit dirs and unrun folders into tests/unit, tests/integration and tests/e2e * test: fix disk cache collection, ungate saturation check, drop RSS memory tests * test: assert one delivered log per completion in Datadog and LangSmith checks * test: drop SDK-direct e2e copies and align cost-map path filter cases --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
1dbde0845e
|
test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758)
* test: move guardrails, image, audio, batches, search legacy tests to unit * test: restore the custom guardrail and remaining chunk 1 unit rows * test: move the ocr matrix to e2e * test: drop comments from moved tests and fix vertex ocr credentials * test: move litellm utils and unified google legacy tests to their tiers * test: move logging callback and pass-through unit legacy tests to their tiers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: move store_model_in_db legacy tests and start the admin ui migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: finish the proxy job rows and drop their CircleCI jobs Moves the remaining multi-instance, openai endpoint, otel, pass-through and top-level proxy tests into tests/integration with scripted upstreams, rewrites the weak replacements to assert the outbound wire and exact responses, and removes the build_and_test job group this PR owns from CircleCI * test: tighten the key generation and error log assertions, rename migrated unit files to their mapped names * test: shard the pagerduty unit tests and record the rust harness file on the coverage allowlist * test: drive the SSO sign-in tests through a scripted IdP and isolate the cost center stand-in * test: type the e2e beta header, prompt cache and secret manager values * test: assert the missing model error message on the OpenAI routes * test: drop the deleted otel base class import and restore mcp redaction and exact langfuse usage checks * test: make the langfuse fixture directory a package for the layout check * test: restore collected no-match cases, real langfuse and gcs payload checks, and gate the aws secret lane * test: keep the ocr e2e pdf fixture next to the test * test: put meta below covers on the streaming tool search e2e * test: run the OCR matrix against a scripted upstream and fix two runner-only e2e failures * test: count only OCR calls in the OCR matrix recorder * test: pin OCR parse bodies and billed cost, group new e2e cases into classes * ci: restore the python image anchor dropped in the main sync --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c5ac1845b0
|
fix(langfuse): log Responses API token usage instead of zeros (#42212)
* fix(langfuse): log token usage for /v1/responses calls ResponseAPIUsage exposes input_tokens/output_tokens, so the Langfuse generation logger read prompt_tokens/completion_tokens as 0 while cost was still correct. Normalize the usage with ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage before extracting usage_details. Resolves LIT-8238 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse): adapt responses-api usage regression test to langfuse v4 spans The v4 migration exports usage_details as an OTEL span attribute via an in-memory exporter instead of a mocked generation() call. Assert on the exported span's langfuse.observation.usage_details with the same token counts. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(langfuse): log token usage for streamed /v1/responses calls An assembled /v1/responses stream reaches the logger with its usage stored as a plain dict of chat fields, which the attribute reads saw as zero. Route both ResponseAPIUsage and that dict through one normalizer. * fix(langfuse): keep generations whose usage dict fails validation; cover streaming in the e2e A usage dict whose token counts are not integers falls back to the raw object, so the generation is still logged with zero usage instead of being dropped. The live e2e now runs streamed and non-streamed /v1/responses. * ci: re-run the e2e gate * test(langfuse): run the real prompt helper and label the responses usage e2e cases Drop the _add_prompt_to_generation_params pass-through patch from the Responses usage unit tests so the real helper runs with the empty metadata they already pass. Declare a Subject on both e2e parametrizations (observability, /v1/responses, openai, stream and nonstream). --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
60f35e35ad
|
feat(azure_storage): upload audit logs to Azure Storage via audit_log_callbacks (#43838)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dc95de5209
|
feat(otel)!: let a team capture_message_content override the global setting for its OTel v2 destination (#44264)
* feat(otel): let a team strip captured message content from its own OTel v2 destination Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): split the shared callback option conflict check and sync dashboard API types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): address review feedback on team capture_message_content Classify capture_message_content as NotSecret in the credential slot registry, inject test exporters through register_exporter_factory instead of patching _exporter_from_spec, type the test helpers, and stop reassigning processor in build_tracer_provider Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): add integration cells for team capture_message_content Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): cover weave_otel and newrelic team destinations in capture_message_content cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): stop rejecting client-sent capture_message_content at the request body boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(otel): let a team capture_message_content override the global setting in both directions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep a team's capture_message_content on its own OTel v2 destination The setting no longer rides the flattened team callback vars, so a sibling destination and a routed provider follow the global mode. Request bodies get the base 400 back, a failure-only entry rejects the setting, and entries for one backend must agree Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): type the capture_message_content callback validation tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): type the remaining capture_message_content test signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): mark the new capture_message_content test locals Final Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): default tenant capture to callback_settings.otel and keep CaptureMessageContent formatting bare Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): keep CaptureMessageContent a plain str like main The str-to-Enum switch changed the public type: the config field's type, repr and model_dump, a constructor that now raised on unknown values, and unknown env values coerced to no_content. CaptureMessageContent is the plain str subclass main ships again, the field is a str, and a captures_span_content helper replaces the enum property. A test pins that a published preset's capture mode stays the default for an unowned destination, which is what main forwards. * test(otel): drop the module-level CaptureMessageContent import the rewritten test no longer uses * fix(lint): keep the merged branch within main's type budgets Freeze the capture-mode settings model (LIT015) and read stored logging entries through one typed sequence helper, so the conflicting-capture checks add no unknown types. * fix(otel): reject a destination capture_message_content while OTel v2 is off The legacy OTel callbacks never read the per-destination setting, so a team could register no_content and still export prompts and replies under the global capture mode. Registration now refuses the var unless the proxy runs with LITELLM_OTEL_V2=true, the same rule the Arize transport and New Relic routing vars follow. * test(otel): type the OTel v2 fixtures and the v2-off regression test * fix(otel): accept only no_content and span_only on a team or key destination OTel v2 writes captured content only to span attributes, so event_only behaved like no_content and span_and_event like span_only on a destination. The proxy-wide setting keeps all four values * docs(otel): say an omitted capture setting follows the proxy's own callback for that backend * test(otel): describe the general capture fallback in the unowned destination test --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <yucheng@berri.ai> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
9b53ee878f
|
perf(ci): isolate and shard integration suites (#44146)
* perf(ci): distribute unit cases and reuse security sweep connections * revert(ci): reject first speed trial after new failures * perf(ci): avoid repeated test discovery and parsing * perf(ci): isolate parallel security workers * revert(ci): restore measured security execution isolation * perf(tests): reuse source scans and deduplicate cache cleanup * perf(ci): restore parallel security with safe log batching * test(ci): isolate fixture state and schema comparisons * test(ci): make schema and scheduler fixtures deterministic * test(ci): drain queued logging before fixture teardown * test(ci): preserve logging delivery without idle worker restarts * test(ci): keep guardrail sinks available during parallel bursts * test(ci): isolate full proxy lifecycle from gateway imports * test(ci): preserve first failure responses and sweep evidence * test(ci): restore documentation URL settings between cases * test(ci): isolate open MCP tools in private gateways * test(ci): group private MCP cases and handle exited workers * test(ci): retain proxy exits and MCP failure details * test(ci): repair merge interactions with main Key the schema shard's full clone on artifact-name, since main removed the unit-flag input. Give each owned Redis its own log file again. Hand the realtime transcription tests main added a logging object whose dispatch handlers are awaitable, so the per-test logging drain no longer waits on a MagicMock task forever * perf(ci): shard owned integration cases and close lifecycle leaks * fix(ci): qualify shard execution and harden interrupted cleanup * fix(ci): bound worker supervision and own fault traffic * perf(ci): reuse unparametrized fixtures across bounded case groups * fix(ci): preserve boot fault allowances and complete exit records * test(ci): tolerate startup retries and retain readiness details * perf(ci): balance integration shards by recorded duration * fix(ci): load migration database helpers only for workers |
||
|
devin-ai-integration[bot]
|
8a103e26ab
|
test: move routing, caching and callback tests out of local_testing (#45761)
* test: move routing, caching and callback tests out of local_testing Moves the 37 routing, caching and callback files in tests/local_testing into mirrored tests/unit paths and tests/integration sdk and security groups per the migration manifest, and deletes the 8 rows it marks for deletion * test: cover least-busy routing and tighten migrated router, cache and pass-through tests Add router-level least-busy tests for the three legacy rows that had no destination, move completion fallback and model alias tests to their mirror files, move router batch tests next to the router, and strengthen tests whose mutants survived (specific deployment, batch pair grouping, custom cache key, router cache ttl). Add a unit test for configured pass-through Authorization forwarding. * test: tighten migrated callback, proxy config and cache ttl tests Assert the full StandardLoggingPayload key set and the wire user on callback tests, add an async embedding failure callback test, drive the prometheus redis failure through a real RedisCache, use the real salt key in proxy config tests, pin the redis ttl value and rename fallback test helpers * test: fix semantic cache assertions and order-dependent router and callback tests Count semantic cache embeddings on the route, drain async cache writes before the second lookup, echo the requested model from the callback provider mock so response_cost does not depend on model_cost entries other tests register, and move the two-key authorization assertion to the auth failure test it belongs to * test: cover model_list fan-out and the usage-based routing tpm limit lost in the move * test: wait for the streamed cache entry in redis and drop deleted files from the hosted mock guard * test: restore sync callback, audio redaction and langfuse export coverage, keep redis-compat filter on the moved file * ci: run the remaining router-keyword legacy tests on one node * test: refuse the redis write in-process instead of resolving an invalid host --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
yucheng-berri
|
7ff5010674
|
feat(guardrails): split logging_only_scope direction from input-failure policy (#45709)
* feat(guardrails): split logging_only_scope direction from input-failure policy logging_only_scope is now direction-only (input, output, unset for both directions). The new logging_only_continue_on_input_failure flag controls whether a flagged or raising logging_only input scan still lets the response be scanned. "both" stays accepted everywhere as a deprecated alias for unset scope plus the flag set to true, normalized at callback init while stored rows keep the written value. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dashboard): narrow continue-on-input-failure update type and stay under guardrail_info max-lines Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dashboard): keep custom-code guardrail controls readable with logging_only Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(dashboard): scope continue-on-input-failure to the default direction and restore verdict assertion Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(guardrails): remove the both value from logging_only_scope logging_only_scope only shipped its both value in dev prereleases, so it is dropped instead of kept as a deprecated alias. Leaving the scope unset and setting logging_only_continue_on_input_failure gives the same behavior. A stored both value is now treated like any other invalid scope: it is logged and ignored on load, and rejected with 422 on create or update. * fix(guardrails): tolerate an invalid logging_only_scope in the v1 guardrail list A config.yaml guardrail whose logging_only_scope is no longer valid, such as the removed both value, made GET /guardrails/list return 500 for every guardrail. The v1 list now parses params with the same tolerant parser as the v2 list, info and PATCH paths, so the bad value is logged and dropped. * test(guardrails): drive continue-flag rejection tests through the real guardrail handler --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
13928bdabe
|
fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream (#45433)
* fix(websearch_interception): drop stream_options when forcing stream=False and replay it on the final stream * test(websearch_interception): type the conversion helpers and drive the replay test through mock_response * test(websearch_interception): pin the replay test's values as Final and record follow-ups immutably * fix(proxy): strip every agentic-loop control key from client bodies, the web search keys included * fix(interception): move stream_options off forced non-stream calls for code interpreter and headroom, and keep adapters from replaying nested calls * test: annotate new interception test locals with Final * fix(responses): drop params a deployment hook removed on sync Responses calls * test: drive the sync Responses stream_options regression through a mock transport and real Azure config * test: type the agentic loop recorder and capture the sent request on an object * test(integration): cover interception stream_options on chat, responses, messages and the SDK * test(integration): assert the follow-up error text and release a held burst on failure * test(integration): pin the follow-up failure contract and wait for the replacement worker before teardown --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
63d823d28a
|
fix: support OpenAI SDK 3 while retaining SDK 2 compatibility (#44927)
* fix: support OpenAI SDK 3.x while retaining 2.x compatibility * fix: preserve HTTPX compatibility across OpenAI SDK versions * refactor: trim OpenAI SDK compatibility patch * refactor: simplify SDK client defaults * test: cover SDK compatibility across provider routes * fix: use shared names for SDK transport helpers * fix: address SDK compatibility review feedback * test(openai): send requests through the SDK API factory clients * fix(exceptions): keep the provider response on PaymentRequiredError under httpx 2 * ci(base-sdk): count httpx2 as base-only when a base dependency brings it in OpenAI SDK 3 depends on httpx2, so the litellm-core install at highest resolution pulls it in through openai. The base-only check now resolves the declared base dependency closure of litellm and litellm-core and accepts extras-only modules owned by a distribution inside that closure * test: cover OpenAI SDK 3 client paths end to end and read SDK HTTP headers case-insensitively * test: read C02 spend rows once, wait out the worker healthcheck on respawn, and resolve module owners on Python 3.10 --------- Co-authored-by: Marcus Wood <marcuswood@openai.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Yassin Kortam
|
d1945f69da
|
chore(github_copilot): sync model catalog with Copilot's served models (#45386)
* chore(github_copilot): sync model catalog with Copilot's served models * test(github_copilot): assert catalog invariants instead of pinning vendor facts * fix(github_copilot): keep thinking capabilities on served Claude catalog rows * fix(github_copilot): keep supports_reasoning on catalog rows that shadow the family fallback * fix(model-catalog): preserve merged Copilot Haiku metadata --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yuneng-jiang
|
5e1c5c0bd1
|
test: move offline proxy auth, hook, spend and pass-through tests to tests/unit (#45568)
* test: move offline proxy auth, guardrail hook, spend, budget reset, GCS payload and pass-through tests into tests/unit Relocate 53 legacy tests that pass with no network or keys into the tests/unit files that mirror the code they exercise, drop 2 llm_guard tests already covered by test_llm_guard_call_type_aliases, and delete the emptied legacy files. * test: freeze the reset clock and run the real reset methods in the moved budget reset tests Pin datetime in reset_budget_job and timezone_utils to a fixed instant and await the service-hook tasks the job spawns instead of sleeping. Drive the per-user failure with a malformed budget_duration row from the prisma mock so the real _reset_budget_for_key/user/team run instead of patched replacements. * test: use fixed timestamps in the moved pass-through and spend log tests * test: mock the moderation HTTP call and type the moved moderation and spend log tests Serve the OpenAI moderation response through an httpx MockTransport so the real async_make_request runs, annotate the moved tests' locals with Final, and pass get_logging_payload typed kwargs instead of an untyped input_args dict. * test: drive the moved proxy tests through boundaries and annotate their locals with Final Serve GCS downloads and the Router moderation call through httpx mocks with fake Google credentials, let the real pass-through success handler fail on the mocked upstream response, and fold the Vertex live route endpoint check into the existing route test. Annotate moved locals with Final, build the budget reset rows without rebinding, and type the remaining fakes without **kwargs |
||
|
yuneng-jiang
|
3250ccc802
|
test: move offline logging, secret manager and provider tests from legacy dirs into tests/unit (#45554)
Moves 58 legacy nodes (cyberark, logging callback manager, langfuse handler and helpers, SQS, app crypto, bedrock nova and invoke, cloudflare, litellm_proxy, nvidia nim, replicate, triton, convert_dict_to_response, get_model_info, vcr live-call probe) into the tests/unit files that mirror the modules they exercise. Calls that reached huggingface.co, langfuse or 0.0.0.0 are refused at the HTTP boundary with respx. Tests that need credentials, a license, a spawned server, sleeps, a subprocess, or assert nothing stay in their legacy files |
||
|
devin-ai-integration[bot]
|
bbe5595a99
|
fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge (#45500)
* fix(responses): keep a flagged hosted deployment's prompt cache breakpoint on the chat bridge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): read the serving provider's row for prompt cache breakpoints on the chat bridge and base_model * fix(anthropic_cache_control_hook): rename the module-level provider resolver so the recursion check passes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): follow the credential dialog's new name field and auth method id in the federation e2e spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
488594e03f
|
refactor(proxy): inject one UTC clock read per operation into gateway tracking, PTU rollup and Mavvrik export (#45520)
* fix(proxy): read the UTC clock once per operation in gateway tracking, PTU rollup and Mavvrik export * refactor(proxy): default the injected clocks to get_utc_datetime instead of three private copies --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
joshua-berri
|
911aff2752
|
refactor(sdk): separate core AWS and tokenizer dependencies (#44447)
* refactor(sdk): separate core AWS and tokenizer dependencies * fix(sdk): preserve runtime tokenizer alias compatibility * test(sdk): compare tokenizer installs through existing entry point * fix(sdk): preserve bearer headers and optional dependency interfaces * fix(sdk): retain safe tokenizer fallback diagnostics * test(sdk): inject tokenizer dependency for fallback diagnostics * fix(core): preserve AWS dependency errors with retries * fix(core): centralize optional AWS dependency handling * refactor(core): reuse optional import helper for Invoke streams * test(core): compare complete shared dependency requirements * fix(core): keep tokenizer logging import compatible with main * fix(core): preserve native token decoding and Responses dependency errors * test(core): isolate optional dependency import failures * fix(sdk): retain typed exception message access * test(sdk): scope HTTP verification environment changes * fix(sdk): preserve bearer request typing across Bedrock callers --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f09408dd37
|
test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362)
* test: replace 61 live logging and otel tests with offline unit and integration coverage * test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts. * test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request * test: isolate the prometheus latency test on its own deployment so counts do not depend on order * test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only * test: drain the redis test's logging and scope otel span checks to the test's own trace * test: drive the periodic slack flush without a wall-clock interval * test: scope router hook events to the test, script the db clock, split a nested comprehension --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
308c42a4a2
|
fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698)
* fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): shed OpenInference output tool calls individually under the span attribute budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): avoid mutation in Arize OTel v2 integration helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): audit Arize OTel v2 OpenInference spans across endpoints, modes and chaos Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): wait for each exported span before the next request in Arize OTel v2 audit tests * test(otel): make Arize OTel v2 audit absence and outage checks deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): collect Arize OTel v2 outage spans through an in-order sentinel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): add skipped BUG cells for pre-existing Arize OTel v2 gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): repair Arize OTel v2 regressions from #43698 (linear fit, metadata slot, repr tool args) (#44488) * fix(otel): keep Arize OTel v2 regressions in check — O(n) fit, metadata slot, repr tool args Three regressions from #43698's OpenInference tool-call/metadata emission: 1. Metadata evicted indexed message attributes: the new `metadata` key competed for the 128-attribute span budget, and the fit sheds whole message groups BEFORE `span.set_attribute`, so the SDK's dropped counter stayed 0 — an invisible eviction (live A/B: input-message attributes 86 -> 84). Two-part fix: the fit pins `metadata` behind every message group (it sheds only once all indexed messages are gone), and the span budget no longer charges pre-set attributes the mappers overwrite in place — a boundary-opened LLM span already carries keys like `gen_ai.request.model`, so the old accounting reserved slots the fit could never spend. Live: 86 input-message attributes with the metadata attribute riding alongside. 2. Quadratic shed on long prompts: `_message_shed_groups` rescanned the full group map once per message (measured on a real acompletion: 0.032/0.128/0.478/1.910s at 1000/2000/4000/8000 messages vs 0.007/0.010/0.022/0.029s at base). Index the tool-call groups once by (family, message index): the fit is linear again (0.008/0.007/0.014/ 0.031s, same rig). 3. Malformed Python tool arguments lost the whole span: provider adapters and `model_construct` responses hand over raw objects, and `json.dumps` raises on tuple-keyed dicts (TypeError) and cycles (ValueError) before the span is exported. Serialize with a repr fallback; both cases now export with a readable arguments attribute. The attribute budget change affects every boundary-opened LLM-call span (strictly more attributes retained, never fewer); the mapper changes only touch the OpenInference vocabulary. * refactor(otel): build the tool-call group index in one shot Review follow-up: the dict.setdefault/append seeding in _tool_call_groups_by_message violated the no-mutation coding convention (AGENTS.md: build values in one shot with comprehensions or generators wrapped in tuple()/MappingProxyType()). Rebuild it as a sorted groupby comprehension; randomized parity harness confirms the shed order is byte-identical to the seeded version (400 trials). Also pin the overflow corner Greptile asked about: a pre-set indexed-message key the fit sheds keeps its earlier value in place, so the span total can never exceed the SDK limit (new emitter test). * fix(otel): key the groupby with an explicit tuple to keep basedpyright at budget The slice-keyed groupby (group[:2]) widened the key to tuple[str | int], adding one reportGeneralTypeIssues over the codebase ceiling. Key by the explicit (family, message index) pair instead; shed order unchanged (300-trial randomized parity harness). * fix(otel): read pre-set span keys through a helper typed for both runtime shapes The SDK annotates ReadableSpan.attributes as a Mapping, but an ended span hands back a tuple of pairs, so the inline isinstance branch narrowed to Never and pushed reportGeneralTypeIssues one over the codebase ceiling. Extract _carried_keys with the runtime union declared on the parameter; behavior unchanged. * test(otel): drop the wall-clock bound from the long-prompt attribute fit test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): annotate to_openai_dict as Mapping after the rebase onto main Main's type-discipline budget tightened since the branch point; the plain dict return annotation is the one violation the rebased branch adds. Callers only serialize the result, so the read-only view is accurate. * chore(otel): drop the restating docstring from to_openai_dict --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
85a3869dfd
|
feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes (#43801)
* feat(guardrails): run each mode only on streaming, non-streaming, or both
Add stream_scope so a rail can target streaming inference, non-streaming inference, or both per pre, during, and post mode
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor stream_scope in pipelines and dashboard types
Pipeline steps skipped the stream_scope filter, direct construction ignored mixed-case maps, and schema.d.ts was stale.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): skip unmatched stream_scope steps instead of allowing
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): keep stored stream_scope keys for modes not on screen
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): format stream_scope helpers and update fork MCP unit tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): keep hang cancellation tests from timing out during setup
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor path-defined streaming for stream_scope
Passthrough routes like Gemini streamGenerateContent decide streaming from the URL, so stamp that onto hook data before guardrails run.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): copy AnthropicModelCapabilities instead of cloning
Clippy treats clone-on-Copy as an error, which failed rust-lint on the messages request tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): trust only server stream classification
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): keep streaming marker through deepcopy
scan_raw_request snapshots copy each field, so a plain object() marker would lose identity and skip streaming-only rails.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cost-map): drop duplicate perceptron-mk1.5 row
Two main cost-map PRs both added the OpenRouter model, so the merge left a second key that CI rejects.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): expect native transcription 429 as RustUpstreamError
HTTP status errors from native routes map through route_error_to_pyerr, so the wheel SIGINT child was dying on an outdated RuntimeError check and never reached the hang probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): follow Google Interactions OpenAPI without hardcoded names
The live spec dropped CreateModelInteractionParams and renamed the item path to {interactionsId}. Misc CI failed because the compliance tests still looked those names up as literals.
* test(guardrails): reproduce stream_scope bedrock and passthrough field gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover invalid stored stream_scope reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify bedrock stream actions and keep caller is_streaming_request
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): make stream_scope_allows public and drop mutable builds
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify pass-through and Bedrock stream scopes
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden stream classification and validation
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add stream scope integration audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid mutating passthrough custom body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): run stream scope audit without enterprise license
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): keep stored scope restart cell on one worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): scope stream scope audit sink assertions to the rail under test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert logging_only scope absence behind an ordered barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert one logging_only scan per phase after the barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): set request scope on pass-through stream fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid YAML stream_scope in v1 guardrails list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: merge main into litellm_guardrail_stream_scope_fixes
Update pass-through pre-call test callbacks for main's endpoint_type argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add request paths to pass-through fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover websocket pass-through stream scope and LIT-9050 outage spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): remove unused type discipline suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): script the vertex live upstream in the websocket stream scope test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep stream marker json-serializable and restore pass-through helper names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): allow required Bedrock action re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): strip the stream marker from pass-through payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): drop stream marker by value in scans and snapshots, plain-tuple stream scope state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): render tag-scoped guardrail modes read-only in the custom code editor
A guardrail whose litellm_params.mode is the tag-scoped dict {tags, default}
crashed the Custom Code editor on open: normalizeMode wrapped the dict into
the mode array and StreamScopeFields rendered it as a React child (error #31,
whole dashboard unmounted). Treat a non-string non-array mode as no editable
modes, show formatGuardrailMode(mode) in a disabled input (read-only, matching
the guardrail info view), and keep mode/stream_scope out of the update payload
for such guardrails.
* fix(ui): resolve merge fallout in guardrails components
Deduplicate toModeArray import after the merge, and move the read-only
guardrail details block back into GuardrailReadOnlyDetails (now rendering
the shared mode/logging-only rows plus the stream-scope detail) so
guardrail_info.tsx stays under the 800-line lint budget. Guardrails UI
suite: 298 passed.
* fix(ui): drop duplicate toModeArray import reintroduced by merge
* chore(pass-through): document the deliberate in-place marker strip as mutable-ok
The clear/update on _parsed_body is load-bearing: rebinding to a fresh
mapping instead breaks 75 pass-through tests because the marker-free body
must propagate through the caller's request dict so downstream guardrail
scans and snapshots never observe the server streaming marker.
* fix(guardrails): move stream_scope after timeout in CustomGuardrail init
Inserting stream_scope before the existing timeout parameter shifted the
positional slot of timeout, so positional callers constructed with their
timeout bound to stream_scope (ValueError) and timeout silently None.
Restores the base parameter order; keyword callers are unaffected.
* fix(guardrails): typing pass for the lint gates
stream_scope leaves the declared constructor parameters (restoring the
base positional surface; keyword construction unchanged), unknown config
values crossing the new stream-scope code paths get typed locals or
cast-ok boundaries, and the passthrough payload literals are annotated.
All three lint gates pass against current main; the guardrail suites are
unchanged (236+5019 passing; one known anyio-driver failure pre-existing
on base).
* style: sort cast imports for the ruff gate
* chore(pass-through): drop unused BEDROCK_STREAMING_ACTIONS re-export
The streaming check now uses is_bedrock_streaming_endpoint; nothing in
the repo imports the name from this module.
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: gabriele <gabriele@berri.ai>
|
||
|
devin-ai-integration[bot]
|
d5173e3d70
|
test: move 76 live top-level tests to offline unit and integration coverage (#45352)
* test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
9f150f74ed
|
feat(prometheus): expose per-project per-model rate limit allowed and used gauges (#43561)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
df13d59306
|
fix(logging): skip sync success callbacks for internal sub-calls, deflake RAG and Langfuse tests (#45345)
* fix(logging): skip sync success callbacks for internal sub-calls Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(langfuse): assert no retry sleep instead of a wall-clock budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: add types to deflake regression coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(tests): avoid unrelated formatting changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
04d97abffb
|
test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs (#45298)
* test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs * test: restore lost coverage in litellm_utils offline replacements Bound live health-check tasks to the concurrency limit so eager task creation behind a semaphore fails, route the Langfuse trace-id check through completion -> Logging.get_trace_id with all four metadata combinations, run the callback dedup checks through acompletion, pass the dynamic key via litellm_params, cover the env-inferred default model list, and add the per-provider audio transcription config lookup * test: restore router coverage lost in the offline move Add the missing non-stream LIT-3058 header/count-once test, pin the UTC minute on every usage-counter test, assert router-level client reuse for transcription, and replace the private selector-attribute checks with routing outcomes per strategy. Assert outbound bodies for speech, rerank, image, assistants and moderation, add timeouts to event waits, and drop the router_unit_tests husk files that no longer hold tests * test: restore sync stream, router sync stream, error event and field-type coverage for migrated responses tests Add the sync streaming logging and sync Router.responses streaming cases the offline replacements dropped, port the legacy per-event and response field-type validation, assert raw headers, the float created_at conversion, MCP tool headers, the search_context_size input, and add an offline replacement for the in-stream context-window error event. Remove helpers left dead in the legacy file. * test: run numpydoc-backed unit tests in GHA and drop a test that pinned a crash * test: restore sequence number, item id and content part checks in the responses stream validator * test: match router logging events to the test's own deployment so late events from other tests are ignored * test: restore the s3 cold storage history test to its original assertions --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Priyansh Nandwana
|
7c7b0ea85b
|
fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729)
* fix(cache_control): let a hosted deployment opt into the OpenAI cache dialect _targets_openai_prompt_cache_breakpoint gated on custom_llm_provider == "openai" unconditionally, so an OpenAI-shaped model served by another provider could never qualify, even with supports_prompt_cache_breakpoint set explicitly on its own cost-map entry. bedrock_mantle honours prompt_cache_breakpoint end to end, and had no way to say so. model_cost is keyed per exact deployment string, so a flag on the deployment's own entry states the dialect more precisely than a provider name can. Consult it before the provider check, and require the entry's litellm_provider to match the serving provider so an openai entry cannot license another provider. Entries for the openai provider keep their api_base check, so an OpenAI-compatible third-party host is still not assumed to speak the dialect. Fixes #38666 * test: drop a laziness assertion the existing suite already makes test_provider_lookup_skipped_for_models_below_gpt_5_6 already pins that _resolve_provider is not called for an unflagged model, and it covers the new flag lookup unchanged. The duplicate patched a litellm internal for no added coverage, which the test-quality gate counts against TQ008. * fix(bedrock_mantle): flag GPT-5.6 and newer rows for OpenAI explicit prompt cache breakpoints * fix(responses): predict the chat-completions bridge with the model name the dispatch resolves * fix(cache_control): read a region-prefixed Mantle GPT deployment through its region-free price-map row * fix(cache_control): let a bare hosted deployment name read its own provider's price-map row * fix(responses): hand the hook the provider the router resolved for a Foundry GPT deployment * fix(cache_control): read the deployment breakpoint flag strictly and default a null prompt_cache_options A cost-map or deployment `supports_prompt_cache_breakpoint` that is not the boolean `true` (the string "true", the integer 1, "false", 0, a long string) no longer opts a deployment into the OpenAI prompt cache dialect; only `true` does, the same reading Bedrock Converse applies to its own flag. A client that sends `"prompt_cache_options": null` on `/v1/chat/completions` or `/v1/messages` now gets the implicit default the hook already applied on `/v1/responses` when it placed a breakpoint; before, the null suppressed the default and the request left with a breakpoint and no options. * test(integration): audit cells for the Mantle GPT prompt cache breakpoint dialect Deterministic cells for the configured-breakpoint path on Bedrock Mantle GPT and Azure AI Foundry GPT-6 deployments: every endpoint, streaming and not, sync and async SDKs and raw httpx, the hostile option shapes, flag precedence, the response cache twin, a two-worker burst, a worker kill and a graceful restart. * fix(cache_control): ignore a malformed cache_control_injection_points value instead of failing the request A deployment or client `cache_control_injection_points` that is not a list of points (a string, an integer, a bare point dict, a list of strings) raised inside the prompt hook (`'str' object has no attribute 'get'`, `'int' object is not iterable`) and turned every request to that deployment into a 500 on `/v1/chat/completions`, `/v1/messages` and `/v1/responses`. Every entry point now reads such a value as no configured points, the way a `null` already read, and the request leaves without a breakpoint; entries of a list that are not points are dropped and the point entries kept. * test(integration): cover int, dict and string-list injection point shapes in the Mantle audit cells The D9 cell now also sends an integer, a bare point dict and a list of strings as the deployment's `cache_control_injection_points`, which the merge base answered with a 500 on every request. * fix(cache_control): stamp the OpenAI dialect for a bare deployment name served by a flagged provider A deployment written as `model: openai.gpt-5.6-sol` with `custom_llm_provider: bedrock_mantle` has no cost-map row of its own and no openai row of the same name, so the dialect stamp's cheap gate (`supports_openai_prompt_cache_breakpoint`) returned early and the configured point was never stamped. On `/v1/responses` and on the chat seeding path the hook sees no provider, so it fell back to the Anthropic `cache_control` marker, which Bedrock Mantle strips. The gate now also passes a model whose serving provider is already known and whose provider-keyed row carries the flag, which is the same row the dialect resolution reads and costs no provider lookup. A bare name without a provider is still left alone, so models below GPT-5.6 keep their points untouched and resolve nothing. * test(integration): cover a bare Mantle deployment name with its provider in the audit cells A deployment configured as `model: openai.gpt-5.6-sol` plus `custom_llm_provider: bedrock_mantle` sends the breakpoint and the implicit options on all three endpoints; the merge base leaves it on the Anthropic dialect. * fix(cache_control): keep a configured point that sits beside junk entries on every chat seed path `configured_injection_points` kept the point dicts of a mixed `cache_control_injection_points` list as a tuple, but only read a `list` back. The chat seed and the `/v1/responses` dialect stamp write the normalized value back onto the request, and on a deployment the OpenAI dialect does not stamp (an Anthropic model, Claude on Bedrock Mantle) that value is the tuple itself, so the hook then read it as no configured points and the valid point was silently dropped. The stamped OpenAI dialect and the `/v1/messages` path kept it only because they build a new list. The normalizer now reads back the tuple it wrote, so the point reaches the wire on every path; an all-dict list still passes through as the same object. * test(integration): cover a configured point beside junk entries on a Claude Mantle deployment A deployment configured with `cache_control_injection_points: ["system", {system point}, 3]` marks the system block on the Anthropic dialect on all three endpoints; the merge base answers 500 on the junk entry. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
2803a16b36
|
feat(ui): add auto-router usage and savings table (#45152) | ||
|
devin-ai-integration[bot]
|
fa2c8984ba
|
test: move the unit half of 126 mixed legacy files into tests/unit (#45090)
* test: move the unit half of 126 mixed legacy files into tests/unit * test: restore litellm globals that moved tests set * test: finalize migration test cleanup * test: restore original bodies of moved legacy tests The move into tests/unit had rewritten 612 test bodies, and some of the rewrites dropped assertions. Each moved test now carries its original body from the legacy file, with only the imports, helpers, fake provider credentials and monkeypatched env it needs to run under tests/unit test_timeout_streaming goes back to tests/local_testing because it needs the fake OpenAI endpoint server. The image payload fixture moves with its only user, and two tests that leaked global state (a registered model cost entry and queued logging tasks) are now isolated * test: drop module imports shadowed by restored local imports * test: assert on LiteLLM output in no-assertion moved tests and isolate leaks Twenty no-assertion candidates get one assertion on the value LiteLLM returns, with the original lines unchanged. Four tests go back to their legacy files because they only check types or imports, write into the working directory, or cannot assert without a body change Two moved tests leaked globals into later tests in the same worker, so monkeypatch fixtures now restore the retry-after header parser and the end user cost tracking flags * test: drain queued logging tasks before the Phoenix span test The moved Phoenix test counted spans from logging tasks that earlier tests had queued, so the drain fixture moves to tests/unit/conftest.py and both it and the Datadog batch test use it. test_factory_function goes back to its legacy file because its returned wrapper calls the real Assistants API and cannot be asserted on without a body change --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
127278f951
|
feat(guardrails): add logging_only_scope to observe input, output, or both (#43695)
* feat(guardrails): add logging_only_scope to observe one direction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(guardrails): remove unrelated test churn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): allow native lifecycle logging-only scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate api types for logging_only_scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): remove callbacks when scope validation fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep output-only scans when request copy fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): configure logging_only_scope on guardrails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep guardrails enforcing when logging_only_scope is invalid at load Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): avoid inline guardrail test fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): support directional scope in PATCH and provider UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): format guardrail files for frontend lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): reset unsupported directional scope selections Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): normalize logging-only scope and sanitize warning logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): normalize scope in shared guardrail field Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(api): sync guardrail schema artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): tolerate invalid stored logging-only scopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): add logging_only_scope integration audit cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): fix scope test lint issues Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): extend K4 chaos test timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): restore stored guardrail row verbatim on rejected patch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): satisfy collection lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): validate masked params through TypeAdapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): tolerate invalid stored params on reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep constructor coercion in tolerant params parser Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): edit logging_only_scope in the custom code guardrail modal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): extract custom code logging-only scope control Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep no-op PUT and logging_only scan-failure semantics stable Three regressions from the logging_only_scope feature, fixed while keeping the input/output/both selection working: 1. PUT with byte-identical litellm_params no longer forces a teardown + re-init of the live callback. reject_invalid_logging_only_scope now only validates; re-init still happens exactly when params/name change. Previously a description-only PUT re-appended the callback at the END of litellm.callbacks, reordering guardrails: with a BLOCK guardrail created before a MASK one, the mask started winning and blocked requests started succeeding (400 -> PUT 200 -> 200). Invalid unchanged scopes are still rejected with 422 without touching the live instance. 2. CustomGuardrail._scan_logged_call no longer swallows input-scan exceptions per branch: a raising or BLOCKED input scan aborts the logging_only hook again (one policy call, one verdict) for guardrails that never selected a logging_only_scope. Explicit input/output/both selection keeps selecting which scans run. 3. A failed PATCH now rolls the DB row back to exactly the stored raw litellm_params (a legacy 4-key row stays 4 keys) instead of expanding it to a full LitellmParams dump; pinned with a test. This matches the PUT rollback shape and is the faithful rollback. * refactor(guardrails): type directional-scope provider list as a tuple * fix(lint): stay within the basedpyright budget - drop a Final annotation assigned inside the validation loop (reportGeneralTypeIssues over ceiling by one) - rename the PR-introduced _configured_event_hooks to public configured_event_hooks; its cross-module import added the two reportPrivateUsage errors that pushed the rule over its ceiling * fix(guardrails): keep the response verdict for explicit logging_only_scope=both on input-scan failure Greptile P1: an explicitly configured both-direction observer asked for a verdict on each direction, so a failed request scan must not silently drop the response verdict. The implicit default (logging_only_scope None) keeps the abort semantics of a logging_only hook whose scan raised, which is the base behavior the earlier fix restored. Also drops two test comments that restated their assertions (Greptile P2). * test(guardrails): align logging_only_scope integration rows with abort-on-input-failure and encrypted params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep the response scan when the request copy fails for explicit both scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
44bc1656e7
|
fix(prompt-caching): default injected /v1/messages breakpoints to implicit lookup (#45121) | ||
|
ahamedshaik16
|
2c1847f8a2
|
fix(prometheus): add model_group label to end-to-end latency metrics (#44860)
litellm_llm_api_latency_metric, litellm_llm_api_time_to_first_token_metric, litellm_request_total_latency_metric, and litellm_deployment_latency_per_output_token previously carried requested_model/litellm_model_name/model_id but not model_group, so pooled-deployment latency couldn't be grouped by model pool on dashboards -- only the proxy-overhead-only metrics (litellm_overhead_latency_metric and friends) had model_group. All four metrics read enum_values.model_group through the existing prometheus_label_factory plumbing, so no new parameter threading was needed, just the label-list addition. Co-authored-by: ahamedshaik16 <24526479+ahamedshaik16@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e2971e0af4
|
refactor(llms): expose public names for private provider helpers (#45037)
* refactor(litellm): migrate private usage in llms Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve Bedrock batch signature marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): migrate llms private usage symbols Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): preserve UUID Watsonx project IDs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): retarget llms mocks to public names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(litellm): limit llms changes to renames and forwarders Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(litellm): keep GCS mock client patching private Vertex auth methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(litellm): assert forwarder arguments and type forwarder signatures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Simon
|
48c88bae04
|
fix(responses): preserve prompt cache reuse in chat bridge (#42281)
* fix(responses): preserve prompt cache breakpoints in chat bridge * fix(responses): preserve multimodal cache breakpoints * fix(cache): retain implicit lookup with injected breakpoints * test(cache): expect implicit responses lookup * test(cache): expect implicit chat lookup |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
tin-berri
|
a193c67347
|
feat(otel): trace auto-router configuration and classifier failures (#44926) | ||
|
devin-ai-integration[bot]
|
282d733fb6
|
feat(auth): deny search tools by default when search_tool_deny_by_default is set (#44490)
Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
26a9b02f7b
|
feat(otel): let team and key Arize callbacks choose the OTLP transport (#44492)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
837c6a7481
|
fix(security): remove the publicly known master key from the repo (#44718)
* fix(security): hash the publicly known master key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: replace weak master key examples and regenerate artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: replace weak key fixtures with generated test keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: generate master keys for proxy startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve lens dev key entropy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore proxy key compatibility in scrub examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: scrub merged SSO fixture and refresh dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): stabilize test keys and metadata collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d4a791d650
|
refactor(types): replace Any with proven types in 157 files (#44798)
* refactor(types): replace Any with proven types in 299 files Clears 871 basedpyright Any errors (reportAny 6,703 to 6,240, reportExplicitAny 1,651 to 1,243) without adding a cast, an ignore or a suppression, and without touching any budget file Most edits are annotation-only: a parameter, return or local goes from Any to object, Mapping[str, object] or the concrete type the value always held. Fourteen files validate untyped JSON once where it enters, through a module-level pydantic TypeAdapter or model_validate, and then use real types No HTTP status, error type or response shape changes. Mistral speech and fal.ai Bria image generation now report a pydantic ValidationError instead of an AttributeError when the provider answers 2xx with a body that is not a JSON object * refactor(types): make the config locals fix effective and trim no-op edits The Mapping[str, object] annotation on `locals().copy()` removed no error, because the checker narrows the variable back to the dict[str, Any] the call returns. 21 provider config constructors now build the same copy with dict(locals()), which the checker infers as object values under that annotation, so each file loses one reportAny. The same annotation is reverted in 31 other config files where it stayed a no-op, together with the tests that were added only to cover those lines, and the one MCP server manager line that no CI coverage shard executes is reverted too. The pull request drops from 345 to 297 changed files. * refactor(types): accept only int in the proxy state setter get_proxy_state_variable is annotated to return int, but set_proxy_state_variable still took Any, so the checker could not hold callers to the type the getter promises. The setter now takes int, which is what its only caller already passes. * refactor(types): index the proxy state key so the getter returns int * refactor(types): keep public annotations and provider error text unchanged Restore every public return, public method parameter, public attribute and exported alias to its annotation on main so code that type-checks against the package keeps type-checking, and take the Mistral speech and fal.ai Bria changes back out so no provider error message differs from main * refactor(types): leave the Vertex RAG chunking read as it is on main Take the chunking format validation back out of the Vertex RAG ingestion path. It needs the vertexai SDK, a storage bucket and a RAG corpus to execute, so nothing here could run it end to end, and it cleared only two errors * test(integration): pin the validated provider boundaries on a live proxy * test(integration): give the held burst a client that outlasts the gate The fault cell holds a burst at the upstream for up to 60 seconds while it kills a worker, but sent the burst through the shared 15 second client, so a slow box could time the survivors out before the gate opened. The burst now goes through its own client whose timeout is twice the gate, and the gate length is one named constant. * refactor(types): keep the license reply handling and experimental MCP signatures as they were The license check validated the whole reply as a mapping, which changed the error text logged for a reply that is not an object. It now validates only the verify value, so every reply is handled and logged exactly as before while the value is still typed. Three files under the experimental MCP server changed annotations on public functions and methods (three returns and three parameters). They go back to their previous content so no public signature in the diff is narrowed. * test(integration): answer the proxy's model-list call in the OpenAI stand-ins Every 300 seconds each proxy worker asks an OpenAI deployment for GET /v1/models. Four new cells own an OpenAI stand-in that accepted only the call under test, so a refresh landing inside a cell failed it. The stand-ins now answer that call through the suite's own helper and the cells count only the provider calls they drive. |