mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
442 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
joshua-berri
|
a4fd58501a
|
fix(mcp): never forward the caller's LiteLLM key to MCP servers (#45401)
* fix(mcp): never forward the caller's LiteLLM key to MCP servers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): tidy caller-key scrub Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): avoid master-key scanner false positive Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): match admission when scrubbing the caller key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): drop header copy churn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): normalize bearer variants in caller-key match Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): inject admission header name and cover provider keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): exercise real REST header extraction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): cover caller-key scrub on the real REST path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): exclude gateway admission header from upstream forwarding * fix(mcp): skip empty admission headers when scrubbing keys * fix(mcp): use scrubbed server credentials for auth probes * fix(mcp): match probe credential precedence to upstream calls * test(mcp): simplify probe header assertion --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0d17f954c0
|
feat(credentials): add display_name and make credential_name immutable on PATCH (#43148)
* feat(credentials): add credential_alias and make credential_name immutable on PATCH Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): query credential rows by role to stay under the no-node-access budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): expect credential_alias in load_credential_list dump Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): drive the credential SearchSelect by placeholder and option roles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): pass the decrypted CredentialItem to update_db_credential during master key rotation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): trim credential_alias in CredentialModal so whitespace-only input clears the alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(credentials): replace credential_alias with display_name and reject edits to config credentials credential_name stays the immutable reference key. display_name is a nullable, trimmed, max 255 character label set on POST or PATCH (omit keeps it, null clears it, blank is a 400). Reads return display_name plus an in-memory source tag (db or config), and PATCH or DELETE on a config-defined credential answers 400 instead of a misleading 404 * feat(ui): show credential display names everywhere and lock config credentials The credentials table shows the display name with the credential name beneath it, badges config credentials and disables their edit and delete actions. The edit modal adds an editable display name next to the read-only credential name and only sends it when it changed. Every credential picker and reference (add model, model info, models table, vector store form and info) shows the label while still submitting credential_name * feat(cli): set display names on credentials and show them in the list * refactor(credentials): keep the new credential types inside the lint gate budgets * refactor(cli): print credential command JSON through one helper * fix(credentials): treat an empty credential_name on PATCH as omitted and pin the 404 for vanished rows A blank credential_name never renamed anything, so PATCH accepts it again instead of answering 400. New tests pin a trimmed display_name on PATCH, the repository carrying display_name, and a 404 (not the config-owned 400) for a DB credential another worker already deleted. The hydration helper no longer copies display_name, since none of its callers read it, and the CLI update passes display_name straight through because the exactly-one check already makes it None when clearing. * fix(ui): keep a model's credential when its picker text is emptied and skip the admin-only list for other roles Clearing the search text in the model edit form's credential picker used to submit null, silently detaching the model's credential on save. None is now the only way to clear it. The models table fetches /credentials only for proxy admins, since everyone else got a 403 on each page load and falls back to the raw name anyway. Also fixes a type error in the re-use credential dialog and adds tests for the display name surviving a provider switch, the 255 character limit, the request payload, and the vector store None choice. * fix(client): percent-encode the credential name when updating its display name A name with ? or # was cut short in the URL, so the PATCH landed on a different credential or 404'd. * fix(credentials): answer 405 with Allow: GET for PATCH and DELETE on config credentials A config-defined credential exists (GET returns it) but is read-only through the API, which is what 405 Method Not Allowed means. 400 described a malformed request, and 404 would claim the credential does not exist. * fix(proxy): ignore a display_name set on config credential_list entries so non-string values cannot fail boot --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5c6ea040b0
|
test: update stale MCP call_tool and usage card assertions, add timeout headroom to request-log index boot test (#45523)
Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
d34ad35281
|
fix(anthropic): count a leading system run through count_tokens' system parameter (#45463)
* fix(anthropic): count a leading system run through count_tokens' system parameter Anthropic's count_tokens rejects role "system" at the head of messages, so /v1/responses/input_tokens with instructions, and /v1/messages/count_tokens with a system-role message, fell back to the local tokenizer. The shared Anthropic count_tokens transformation now lifts the leading run of system messages into the top-level system parameter, the way the chat path sends it, after any system the caller set. Anthropic direct, Azure AI Anthropic, and Bedrock Mantle share that transformation. * test(integration): cover the count_tokens leading-system lift across Anthropic, Azure AI and Bedrock Mantle --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yuneng-jiang
|
72ab736863
|
test(routing): wait on deployment registration instead of boot model-info traffic (#45267) | ||
|
devin-ai-integration[bot]
|
50ab9a3211
|
fix(token_counter): price a base64 PDF document per page instead of as one image (#45301)
* fix(token_counter): price a base64 PDF document per page instead of as one image A `document` or `file` block carrying inline PDF bytes was priced like a single image (85 tokens), so when a provider's count-tokens endpoint rejected the model (Bedrock Opus) the local fallback answered 116 for a 12-page PDF the provider then billed at 35941 input tokens. The counter now reads the PDF with pypdf and prices each page as its extracted text plus the image Anthropic renders it to (1568 px long edge, 1.15 MP, 750 pixels per token), falling back to the old image pricing when pypdf is missing or the bytes are not a readable PDF. * refactor(token_counter): count PDF pages as they are read Sum each page's text and rendered-image tokens straight from the pypdf reader instead of materializing a page list first, keep the fallback to image pricing atomic when a page cannot be read, and annotate the new tests' locals as Final * test(integration): audit cells for page-priced PDF documents in count_tokens, pre-call checks and spend * test(integration): release the held peer when the concurrent budget wait times out --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
04f0ea8c33
|
feat(mcp): translate input requests and bind continuations (#45464)
Some checks failed
Unit Tests / misc (push) Blocked by required conditions
Unit Tests / misc-dirs (push) Blocked by required conditions
Unit Tests / proxy-auth (push) Blocked by required conditions
Unit Tests / proxy-hooks-client (push) Blocked by required conditions
Unit Tests / proxy-infra (push) Blocked by required conditions
Unit Tests / proxy-infra-root (push) Blocked by required conditions
Unit Tests / proxy-server (push) Blocked by required conditions
Unit Tests / unit (push) Blocked by required conditions
Unit Tests / responses-caching-types (push) Blocked by required conditions
Unit Tests / Lens Python 3.10 (push) Waiting to run
Unit Tests / assert-shard-coverage (push) Waiting to run
Unit Tests / auth-checks (push) Blocked by required conditions
Unit Tests / budgets (push) Blocked by required conditions
Unit Tests / custom-logging (push) Blocked by required conditions
Unit Tests / db-and-spend (push) Blocked by required conditions
Unit Tests / endpoints-and-responses (push) Blocked by required conditions
Unit Tests / guardrails-hooks (push) Blocked by required conditions
Unit Tests / jwt-and-keys (push) Blocked by required conditions
Unit Tests / key-generation (push) Blocked by required conditions
Unit Tests / logging-misc (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
Code Quality Checks / code-quality (push) Has been cancelled
Unit Tests: Documentation Validation / documentation (push) Has been cancelled
Code Quality Checks / python-310-import-smoke (push) Has been cancelled
CI Coverage / assert-ci-coverage (push) Has been cancelled
Helm unit test / unit-test (push) Has been cancelled
Lens Worker Image / lens-worker-image (amd64, ubuntu-latest) (push) Has been cancelled
Lens Worker Image / lens-worker-image (arm64, ubuntu-24.04-arm) (push) Has been cancelled
UI Unit Tests / ui-unit-tests (push) Has been cancelled
Lens Worker Image / Publish Lens development index (push) Has been cancelled
* feat(mcp): translate input requests and bind resumable continuations * fix(mcp): keep continuations bound to their original upstream * fix(mcp): sync advertised protocol API schema * test(mcp): run interaction regressions in GitHub CI * test(mcp): set source paths for interaction proxy processes * test(mcp): verify salt guidance across interaction carriers * fix(mcp): preserve interaction authentication and elicitation policy * fix(mcp): bound preflight bodies and preserve challenge routes * test(mcp): type interaction regression fixtures --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0ae62c5d02
|
feat(ui): add the evaluation mode to the Add Model form (#45481)
Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
390bea6a53
|
fix(proxy): save file details for every batch output file so they list and retrieve (#41761)
* fix(proxy): register bedrock batch output files with a file object so they list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): guard output file size when content is missing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover managed batch output file listings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): persist metadata for batch output files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover batch output file listing and retrieval end to end Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve provider metadata for managed batch files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update managed batch output registration mocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): inject fake prisma client in managed batch output file tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): retry provider file details and refresh fallback batch output entries Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover fallback batch output file details refresh after provider recovers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover model-name managed batch file retrieval Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(managed_files): bound the batch output file lookup so a slow provider cannot stall GET /v1/batches * fix(managed_files): skip the provider lookup for a fallback entry written moments ago * fix(proxy): safely refresh managed batch file details Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep batch listing provider-free Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): return metadata for empty S3 objects Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): strengthen batch fallback regression assertions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): describe provider reads and DB writes for batch output files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): sanitize caller-derived ids in managed file logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): simplify batch and file endpoint docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate API types from proxy OpenAPI spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(test): resolve managed files lint violations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): save refreshed managed file details through ManagedFileRepository Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): classify retrieved file purpose by its own bucket prefix Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): move batch and file endpoint behavior notes to litellm-docs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): resolve strict lint violations in file retrieval Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): mark retrieval request metadata handoffs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve managed batch type-check diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): avoid duplicate final response bindings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Yucheng He <yucheng@berri.ai> |
||
|
devin-ai-integration[bot]
|
8a176c0f0a
|
fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt (#45393)
* fix(router): drop the encrypted reasoning a fallback hop's target cannot decrypt An order-based or configured fallback hop replayed the failed provider's encrypted reasoning items to the next deployment, which answered 400 (Bedrock Mantle: invalid encrypted reasoning; OpenAI: invalid_encrypted_content), so every multi-turn Responses fallback for Codex-style clients failed. The hop now drops the encrypted reasoning its target cannot decrypt and keeps each item's readable summary. With encrypted_content_affinity on, the pin narrows to the hop's target order instead of emptying it, so the hop reaches the next order instead of failing with no deployments available. * fix(router): keep encrypted reasoning a same-boundary fallback hop can decrypt Unmarked encrypted reasoning on a hop is attributed to the deployment that just failed, read from the retry breadcrumb, so a hop to a deployment on the same api_base and api_key keeps it and a cross-provider hop still drops it. The hop tests script the upstream at the httpx boundary instead of doubling the handler, and the router coverage script lists the two hop helpers with their tests * fix(router): read the hop's failed deployment from its own metadata bucket and carry it into the Responses mid-stream snapshot * test(router): use a real Router without the origin deployment in the hop strip test * test(router): cover the hop strip when no failed deployment is known * fix(proxy): drop the router's fallback hop state keys from the client body * fix(proxy): keep a request's max_fallbacks cap, drop only the hop state keys * test(integration): audit cells for the fallback hop encrypted reasoning strip Forty-five checked-in cells under tests/integration/routing prove the hop strips the previous deployment's encrypted reasoning on /v1/responses, /v1/chat/completions and /v1/messages (httpx, OpenAI and Anthropic SDKs, sync and async, streaming and not), that the affinity pin yields to the hop, that a client-sent fallback_depth, _target_order and attempted_targets never move or strip a request, and that a concurrent burst, an order-1 outage and a killed worker keep every request stripped and logged once. Every call goes through a lane pinned to one worker that already lists the deployments it needs, because the peer worker learns a /model/new row through the config-sync resync up to sixteen seconds later --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
51fea825cc
|
feat(guardrails): support logging_only mode for the Akto guardrail (#45461)
* feat(guardrails): support logging_only mode for the Akto guardrail * test(guardrails): wait for the spend row before reading Akto calls and cover mixed modes * test(guardrails): cover Akto logging_only on MCP tool calls * test(guardrails): type the Akto logging_only unit tests and inject the HTTP handler * test(guardrails): cover failing Akto replies, provider failures and a mixed outage burst under logging_only * test(guardrails): cover an unreachable Akto with fail_open under logging_only |
||
|
devin-ai-integration[bot]
|
f45407b4c3
|
test(streaming): expect the unwrapped provider error in bridged /v1/messages error frames (#45465)
#44800 re-raises the provider's own error from the chat adapter, so a bridged /v1/messages stream's error frame now reads litellm.RateLimitError without the litellm.MidStreamFallbackError: prefix that #44989's tests pinned shortly before #44800 merged. The three tests now expect the provider error text once and no sentinel, matching the chat_limited case and every other assertion in the two files Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f09408dd37
|
test: replace 61 live logging and otel tests with offline unit and integration coverage (#45362)
* test: replace 61 live logging and otel tests with offline unit and integration coverage * test: restore datadog formatting, deliver datadog logs and redis failures over the wire, assert full router hook payloads, tighten stream usage and otel checks Restores the pre-existing test_datadog.py lines the branch had reflowed. Datadog success, failure and redis-failure replacements now assert the gzip body posted to the intake, with redis failing through a real cache on a closed local port. Router hook sequence recorder checks the legacy field types and asserts concrete payloads, exact streaming and fallback sequences. Stream usage asserts the default include_usage request body and the redacted messages value. Otel asserts response id and token counts. * test: assert budget envelope figures, guardrail inspection and exact prometheus samples per request * test: isolate the prometheus latency test on its own deployment so counts do not depend on order * test: cover timed slack delivery, keep the redis failure test off the network, and make new payload types read-only * test: drain the redis test's logging and scope otel span checks to the test's own trace * test: drive the periodic slack flush without a wall-clock interval * test: scope router hook events to the test, script the db clock, split a nested comprehension --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
308c42a4a2
|
fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans (#43698)
* fix(otel): emit OpenInference tool calls and metadata on Arize OTel v2 spans Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): shed OpenInference output tool calls individually under the span attribute budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): avoid mutation in Arize OTel v2 integration helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): audit Arize OTel v2 OpenInference spans across endpoints, modes and chaos Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): wait for each exported span before the next request in Arize OTel v2 audit tests * test(otel): make Arize OTel v2 audit absence and outage checks deterministic Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): collect Arize OTel v2 outage spans through an in-order sentinel Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(otel): add skipped BUG cells for pre-existing Arize OTel v2 gaps Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): repair Arize OTel v2 regressions from #43698 (linear fit, metadata slot, repr tool args) (#44488) * fix(otel): keep Arize OTel v2 regressions in check — O(n) fit, metadata slot, repr tool args Three regressions from #43698's OpenInference tool-call/metadata emission: 1. Metadata evicted indexed message attributes: the new `metadata` key competed for the 128-attribute span budget, and the fit sheds whole message groups BEFORE `span.set_attribute`, so the SDK's dropped counter stayed 0 — an invisible eviction (live A/B: input-message attributes 86 -> 84). Two-part fix: the fit pins `metadata` behind every message group (it sheds only once all indexed messages are gone), and the span budget no longer charges pre-set attributes the mappers overwrite in place — a boundary-opened LLM span already carries keys like `gen_ai.request.model`, so the old accounting reserved slots the fit could never spend. Live: 86 input-message attributes with the metadata attribute riding alongside. 2. Quadratic shed on long prompts: `_message_shed_groups` rescanned the full group map once per message (measured on a real acompletion: 0.032/0.128/0.478/1.910s at 1000/2000/4000/8000 messages vs 0.007/0.010/0.022/0.029s at base). Index the tool-call groups once by (family, message index): the fit is linear again (0.008/0.007/0.014/ 0.031s, same rig). 3. Malformed Python tool arguments lost the whole span: provider adapters and `model_construct` responses hand over raw objects, and `json.dumps` raises on tuple-keyed dicts (TypeError) and cycles (ValueError) before the span is exported. Serialize with a repr fallback; both cases now export with a readable arguments attribute. The attribute budget change affects every boundary-opened LLM-call span (strictly more attributes retained, never fewer); the mapper changes only touch the OpenInference vocabulary. * refactor(otel): build the tool-call group index in one shot Review follow-up: the dict.setdefault/append seeding in _tool_call_groups_by_message violated the no-mutation coding convention (AGENTS.md: build values in one shot with comprehensions or generators wrapped in tuple()/MappingProxyType()). Rebuild it as a sorted groupby comprehension; randomized parity harness confirms the shed order is byte-identical to the seeded version (400 trials). Also pin the overflow corner Greptile asked about: a pre-set indexed-message key the fit sheds keeps its earlier value in place, so the span total can never exceed the SDK limit (new emitter test). * fix(otel): key the groupby with an explicit tuple to keep basedpyright at budget The slice-keyed groupby (group[:2]) widened the key to tuple[str | int], adding one reportGeneralTypeIssues over the codebase ceiling. Key by the explicit (family, message index) pair instead; shed order unchanged (300-trial randomized parity harness). * fix(otel): read pre-set span keys through a helper typed for both runtime shapes The SDK annotates ReadableSpan.attributes as a Mapping, but an ended span hands back a tuple of pairs, so the inline isinstance branch narrowed to Never and pushed reportGeneralTypeIssues one over the codebase ceiling. Extract _carried_keys with the runtime union declared on the parameter; behavior unchanged. * test(otel): drop the wall-clock bound from the long-prompt attribute fit test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(otel): annotate to_openai_dict as Mapping after the rebase onto main Main's type-discipline budget tightened since the branch point; the plain dict return annotation is the one violation the rebased branch adds. Callers only serialize the result, so the read-only view is accurate. * chore(otel): drop the restating docstring from to_openai_dict --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
5d207d85ee
|
fix(proxy): accept team-scoped models by their public name on POST /fallback (#45455)
* fix(proxy): accept team-scoped models by their public name on POST /fallback create_fallback validated the primary and fallback models against the router's stored model names only, so a model created through POST /model/new with model_info.team_id was rejected with a 404 unless the caller used the generated model_name_<team_id>_<uuid> name. The endpoint now also accepts the team public model names, which request time fallback matching already keys on, and lists both kinds of names in the 404's available_models. * fix(proxy): read fallback rules fresh and clear the config cache after a fallback write A second POST or DELETE /fallback within the 60 s config cache TTL started from a cached copy of router_settings and dropped every rule stored since that copy was taken, by any instance. Both endpoints now evict the cached row before the read and invalidate it after the upsert * test(proxy): type the fallback endpoint tests and prove a team request fails over by public name * fix(proxy): keep fallback writes working when the Redis config cache is down and type the stored settings read * fix(proxy): keep the non-standard fallback shapes the router accepts on writes and resolve the config cache at call time * test(proxy): mark the config cache outage test's result as Final * fix(proxy): replace a same-key fallback rule in place and read a null rule list as empty * test(proxy): let the stored router settings fixture carry a null rule list * test(proxy): audit cells for fallback rules by team public name * test(proxy): delete the fallback rules the audit cells save --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
577d74c1ce
|
fix(bedrock): count tokens on bedrock-mantle when bedrock-runtime cannot count a Claude model (#45317) | ||
|
devin-ai-integration[bot]
|
406514fcaf
|
feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs (#44659)
* feat(spend_logs): configure which metadata fields are stored in LiteLLM_SpendLogs Adds general_settings.spend_logs_metadata_fields with mutually exclusive include and exclude lists. The filter runs on a copy of the row right before it is queued for Postgres, so daily spend rollups, budgets and callbacks still see every key. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): keep excluded auto-router savings keys out of published spend log metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): cover metadata retention across endpoints, failures, cache hits, batches, runtime updates and auto-router publication Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for spend_logs_metadata_fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): filter metadata at DB write so guardrail usage sees full rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): poll guardrail daily metrics instead of reading once Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend_logs): drop timeout comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(spend_logs): read spend_logs_metadata_fields through typed general settings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
4ed0267b5a
|
fix(lens): prevent progress updates from starving analysis budget reservations (#45432)
* fix(lens): serialize analysis budget reservations with progress updates * test(lens): verify budget reservations block competing progress writes |
||
|
devin-ai-integration[bot]
|
85a3869dfd
|
feat(guardrails): per-mode stream_scope with bedrock stream and pass-through fixes (#43801)
* feat(guardrails): run each mode only on streaming, non-streaming, or both
Add stream_scope so a rail can target streaming inference, non-streaming inference, or both per pre, during, and post mode
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor stream_scope in pipelines and dashboard types
Pipeline steps skipped the stream_scope filter, direct construction ignored mixed-case maps, and schema.d.ts was stale.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): skip unmatched stream_scope steps instead of allowing
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ui): keep stored stream_scope keys for modes not on screen
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(ci): format stream_scope helpers and update fork MCP unit tests
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(mcp): keep hang cancellation tests from timing out during setup
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): honor path-defined streaming for stream_scope
Passthrough routes like Gemini streamGenerateContent decide streaming from the URL, so stamp that onto hook data before guardrails run.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): copy AnthropicModelCapabilities instead of cloning
Clippy treats clone-on-Copy as an error, which failed rust-lint on the messages request tests.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): trust only server stream classification
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(guardrails): keep streaming marker through deepcopy
scan_raw_request snapshots copy each field, so a plain object() marker would lose identity and skip streaming-only rails.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(cost-map): drop duplicate perceptron-mk1.5 row
Two main cost-map PRs both added the OpenRouter model, so the merge left a second key that CI rejects.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(rust): expect native transcription 429 as RustUpstreamError
HTTP status errors from native routes map through route_error_to_pyerr, so the wheel SIGINT child was dying on an outdated RuntimeError check and never reached the hang probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(tests): follow Google Interactions OpenAPI without hardcoded names
The live spec dropped CreateModelInteractionParams and renamed the item path to {interactionsId}. Misc CI failed because the compliance tests still looked those names up as literals.
* test(guardrails): reproduce stream_scope bedrock and passthrough field gaps
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover invalid stored stream_scope reads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify bedrock stream actions and keep caller is_streaming_request
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(guardrails): make stream_scope_allows public and drop mutable builds
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): classify pass-through and Bedrock stream scopes
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): harden stream classification and validation
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add stream scope integration audit
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): avoid mutating passthrough custom body
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): run stream scope audit without enterprise license
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): keep stored scope restart cell on one worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): scope stream scope audit sink assertions to the rail under test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert logging_only scope absence behind an ordered barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): assert one logging_only scan per phase after the barrier rail
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): set request scope on pass-through stream fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): tolerate invalid YAML stream_scope in v1 guardrails list
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: merge main into litellm_guardrail_stream_scope_fixes
Update pass-through pre-call test callbacks for main's endpoint_type argument
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): add request paths to pass-through fixtures
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): cover websocket pass-through stream scope and LIT-9050 outage spend rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(lint): remove unused type discipline suppressions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(guardrails): script the vertex live upstream in the websocket stream scope test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep stream marker json-serializable and restore pass-through helper names
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(lint): allow required Bedrock action re-export
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): strip the stream marker from pass-through payloads
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): drop stream marker by value in scans and snapshots, plain-tuple stream scope state
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): render tag-scoped guardrail modes read-only in the custom code editor
A guardrail whose litellm_params.mode is the tag-scoped dict {tags, default}
crashed the Custom Code editor on open: normalizeMode wrapped the dict into
the mode array and StreamScopeFields rendered it as a React child (error #31,
whole dashboard unmounted). Treat a non-string non-array mode as no editable
modes, show formatGuardrailMode(mode) in a disabled input (read-only, matching
the guardrail info view), and keep mode/stream_scope out of the update payload
for such guardrails.
* fix(ui): resolve merge fallout in guardrails components
Deduplicate toModeArray import after the merge, and move the read-only
guardrail details block back into GuardrailReadOnlyDetails (now rendering
the shared mode/logging-only rows plus the stream-scope detail) so
guardrail_info.tsx stays under the 800-line lint budget. Guardrails UI
suite: 298 passed.
* fix(ui): drop duplicate toModeArray import reintroduced by merge
* chore(pass-through): document the deliberate in-place marker strip as mutable-ok
The clear/update on _parsed_body is load-bearing: rebinding to a fresh
mapping instead breaks 75 pass-through tests because the marker-free body
must propagate through the caller's request dict so downstream guardrail
scans and snapshots never observe the server streaming marker.
* fix(guardrails): move stream_scope after timeout in CustomGuardrail init
Inserting stream_scope before the existing timeout parameter shifted the
positional slot of timeout, so positional callers constructed with their
timeout bound to stream_scope (ValueError) and timeout silently None.
Restores the base parameter order; keyword callers are unaffected.
* fix(guardrails): typing pass for the lint gates
stream_scope leaves the declared constructor parameters (restoring the
base positional surface; keyword construction unchanged), unknown config
values crossing the new stream-scope code paths get typed locals or
cast-ok boundaries, and the passthrough payload literals are annotated.
All three lint gates pass against current main; the guardrail suites are
unchanged (236+5019 passing; one known anyio-driver failure pre-existing
on base).
* style: sort cast imports for the ruff gate
* chore(pass-through): drop unused BEDROCK_STREAMING_ACTIONS re-export
The streaming check now uses is_bedrock_streaming_endpoint; nothing in
the repo imports the name from this module.
---------
Co-authored-by: Shivi Jain <mobile.350017@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: gabriele <gabriele@berri.ai>
|
||
|
yuneng-jiang
|
a7b04a8028
|
test(decisions): post System One bodies to /v1/systemone in the translation bases (#45327) | ||
|
devin-ai-integration[bot]
|
56bafc29cf
|
feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305)
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings * test(gemini): annotate the embedding mock helper and keep the type comment short * fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly * test(caching): type the partial-hit cache test and pin its clock * fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id * fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs A bare object `input` answered 500 from both the caching handler and the transformation; both now answer 400. An unresolved `files/` reference on vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/ provider. An empty `format` passed through to the provider; it now answers 400 naming file.format. Unknown block keys (`detail`, an unknown video_metadata key, a top-level block key) are dropped under drop_params, global or per-request, and still answer 400 without it. The embedding cache counted file blocks against `max_messages`, so a request with 5 or more blocks was never cached; file blocks no longer count. * fix(caching): answer 400 for an object embedding input on the cache lookup * test(gemini-embeddings): annotate the new embedding tests with return types and Final * test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final * test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends * test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
fzowl
|
f663794fcd
|
feat(voyage): rebrand to VoyageAI by MongoDB and route MongoDB keys to ai.mongodb.com (#41812)
* feat(voyage): rebrand to VoyageAI by MongoDB, refresh model list, support both contextual input shapes * fix(voyage): send auto-chunking params for flat contextual inputs A flat list[str] or bare str for the contextualized embeddings endpoint is only valid as documents with enable_auto_chunking=True and input_type=document, or as queries with input_type=query. Default those params for non-query flat inputs so the request matches the live API contract, while letting caller-set values win. Nested list[list[str]] still passes through unchanged. Tests now assert the auto-chunking params instead of only echoing inputs. * feat(voyage): route MongoDB-issued keys to ai.mongodb.com Mirror voyageai.util.get_default_base_url from the official SDK: a key with the `al-` prefix is issued by MongoDB and is only valid on ai.mongodb.com, every other key on api.voyageai.com. The choice now lives in one shared helper that the embedding, contextual, multimodal, and rerank configs all use for both the default base URL and the Authorization header, so the host and the key always agree. Also drops the voyage-4-nano and voyage-multilingual-2 model map entries: voyage-4-nano is not served on the Voyage API and voyage-multilingual-2 is an older model, so neither belongs in this change. * fix(voyage): restore MongoDB routing on contextual endpoint and re-add voyage-4-nano after upstream merge The upstream merge replaced the contextual config's get_complete_url with a Voyage-only host and dropped voyage-4-nano from the model map. Reapply the al- key routing via get_default_base_url and add voyage-4-nano (open-weight, per docs.voyageai.com) so the branch keeps its task changes on top of upstream. * fix(voyage): drop voyage-4-nano and the contextual tests upstream already covers voyage-4-nano is not served on the Voyage API, so it does not belong in the model map. The contextual input tests duplicate tests/test_litellm/llms/voyage/test_voyage_contextual_embedding.py, which landed upstream with the auto-chunking fix this branch was carrying. * test(voyage): drop sys.path.insert from the voyage common utils test * fix(voyage): route rerank on the key it authenticates with get_complete_url picked the rerank host from the environment while the auth header carried the request key, so an explicit MongoDB-issued key was posted to api.voyageai.com. validate_environment now hands the resolved key to the config instance and get_complete_url reads it back, so the host and the credential always come from one key. The config is built per request, so nothing carries over between them. * test: allow ultrafast pricing keys in model prices schema * test: drop ultrafast schema keys now added upstream * test(integration): prove voyage key-based host routing on the wire --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3571b5dc72
|
fix(vertex_ai): forward the inline-tools-2026-09-15 beta to Vertex and Anthropic (#45297)
* fix(vertex_ai): forward the inline-tools-2026-09-15 beta to Vertex and Anthropic * test(vertex_ai): trim the inline-tools beta test docstrings and type their locals * test(vertex_ai): add audit cells for inline-tools beta forwarding on Vertex Anthropic * test(vertex_ai): send legal whitespace in the hostile beta header audit cell * test(vertex_ai): close the SDK clients in the inline-tools audit cells --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
487164f762
|
test(proxy): audit the websocket rejection log on every live route (#45293)
* test(proxy): audit the websocket rejection log on every live route * test(proxy): scan the stopped proxy log for the websocket crash and drop the uvicorn wording pin |
||
|
devin-ai-integration[bot]
|
b7ecb05340
|
fix(proxy): mark background responses stale_expired when provider returns 404 (#41724)
* fix(proxy): mark background responses stale_expired when provider returns 404 Responses deleted upstream (store=false / ZDR rows dropped after provider retention) now move to stale_expired on the poll instead of being retried every cycle until the 7 day stale cleanup. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): drop inline comment in check_responses_cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive _get_response seam in 404 stale_expired tests Assigning AsyncMock on the instance keeps the new tests inside the TQ002/TQ008 test-quality ceilings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only expire 404 responses polled through a resolved deployment A NotFoundError from the bare SDK fallback can mean a missing or misconfigured deployment, which a config fix inside the staleness window can still recover. Rows whose poll went through their router deployment are the ones the provider actually dropped, so only those move to stale_expired. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): resolve the deployment before fetching so via_router binds once Drops the reinitialised loop flag flagged by review and modernizes the moved annotation. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drive the real response fetch in the poll 404 regression tests * fix(enterprise): expire background response rows on any provider 404 status The GET path maps the provider's 404 body to litellm.BadRequestError that still carries status_code 404, so an except on NotFoundError never fired and the poll job kept retrying the row every cycle. Key the terminal decision on the status code instead of the class * fix(enterprise): expire a background response only on a 404 naming it and skip a bad deployment entry per job * chore(enterprise): record the poll-cycle bound on the managed-object IN lists * test(integration): cover background response poll retirement on the real proxy --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d5173e3d70
|
test: move 76 live top-level tests to offline unit and integration coverage (#45352)
* test: move 76 live top-level tests to offline unit and integration coverage * test: keep the mock-param opt-in guard valid once the top-level senders are gone * test: address review on offline replacements * docs(test): drop the deleted harness test file from the harness check command * test: make the key rebind, team member delete and routes integration tests exercise the legacy paths * test: make fallback, rpm and spend integration contracts deterministic and clean up their rows, assert the rpm limit in usage-based routing * test: expect the no-deployments error at the rpm limit and cover the strategy check without pre-call checks * test: hand member cleanups to the scenario instead of growing a budget list, flatten callback kinds * test: scope the admin health check to the test's own deployment --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
6e54dced8c
|
test: realign stale decisions and lens tests with merged behaviour (#45346)
Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
01166906fb
|
test(integration): run the response cache tool-call tests on a proxy without the message cap (#45350)
* test(integration): run the response cache tool-call tests on a proxy without the message cap Since #43878 the response cache skips any request past 4 messages by default, so the three providers-lane tests that post a seven-message tool conversation twice never saw a cache hit. They now boot one owned proxy from the lane config with max_messages set to null and keep their assertions; the shared proxy and the caching-lane cap test are unchanged * test(integration): give the uncapped-proxy cache tests the owned-proxy timeout budget The providers lane runs pytest with a 90 second per-test timeout that includes fixture setup, and the first of the three tests to run pays the owned proxy boot before the spend-log test can poll for up to 70 seconds. The sibling owned-proxy tests already carry pytest.mark.timeout(240), so these three take the same budget --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
0e26edfdb9
|
test: move 81 legacy live tests in pass-through, spend, batches, audio, search, guardrails, image and ocr dirs offline (#45288)
* test: make legacy live tests in spend, batches, openai endpoints and audio dirs offline (partial)
* test: migrate wave-1b live tests offline (guardrails, images, ocr, search, openai endpoints)
* test: fix wave-1b review items, add responses/ocr integration tests and firecrawl unit test
* test: anthropic messages router/bedrock/openai-bridge unit tests for wave-1b nodes
* test: anthropic messages logging, prompt-caching and tool-search unit tests; drop migrated base nodes
* test: finish pass_through_unit_tests nodes, logging drain fix and mutations
* test: migrate anthropic passthrough tests to integration wire tests
* test: fix passthrough migration wire spend row lookup and wildcard config
* test: migrate hosted vllm and openai file passthrough tests offline
* test: move assemblyai and vertex passthrough nodes to in-process unit tests
* test: restore unlisted router node and fix logging worker drain in passthrough unit tests
* test: drop spend-row BUG skip and sharpen non-streaming skip reason for anthropic messages
* test: use public presidio alias after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: restore the batch and file unit tests the migration rewrote away
* test: restore full legacy intent in anthropic messages router unit tests
Drop the false BUG skip on non-streaming aanthropic_messages logging (success
callbacks do fire; the skipped body filtered on the wrong model), assert the
logged model_group, messages, cost and usage, cover streaming logging for both
Anthropic and Bedrock invoke, assert dict content blocks for Anthropic, Bedrock
invoke and the OpenAI bridge, add the Bedrock invoke leg of the router test,
fall back from a real 401, test system-prompt caching and streaming
message_start cache fields on converse and invoke, send the legacy tool-search
tools and beta header, remove the type: ignore and bare dict helper, and add
in-process native /anthropic passthrough spend logging tests
* test: fix passthrough migration wire tests and drop false native spend BUG skip
The native spend row test read rows by the shared master-key digest, so it
matched other tests' rows; read each row by its own message id instead and
assert tokens, total, spend, tags, provider, api_base and end_user on both the
non-streaming and streaming native routes. Merge the streaming test that never
checked spend, assert exact tags from litellm_metadata, stop rebinding a Final
in a loop, require cost > 0, and add the chat-completions bridge cost case the
legacy test covered
* test: harden the spend, OCR, image, search and guardrail migration replacements
The OCR spend tests now build fresh kwargs per case instead of mutating a
shared fixture, and every payload case asserts the exact logged spend. The
OCR wire test reads its spend row by request id and checks exact page
pricing. Image edit, Nova Canvas, DuckDuckGo, Firecrawl, Bedrock guardrail
and Presidio replacements now fake only the provider HTTP boundary (respx or
an in-process aiohttp server) and assert the outbound request, so the
DuckDuckGo limit, Azure base_model pricing and guardrail masking are proven
rather than assumed
* test: cover Exa and Perplexity search structure and max_results offline and retire the two base search methods
* test: drive batch and file replacements through the provider HTTP boundary and real logging callback
Replace monkeypatched litellm.afile_content and AsyncHTTPHandler doubles with respx routes,
read batch logging metadata from a registered success callback instead of get_logging_payload,
use the real managed-files hook for the GEN-2166 regression, assert outbound request bodies,
pin poller ownership explicitly in the migrated DB-sync tests, and require the scripted
upstream to be hit in the responses error-status wire tests
* test: assert file content download headers pass through the proxy
* test: point spend coverage references at the tests that replaced the retired spend job
* test: assert passthrough identity, spend and route dispatch from the code under test
The AssemblyAI non-admin test asserted metadata it wrote itself and leaked a
background poll to the real AssemblyAI host. It now drives assemblyai_proxy_route
with a real Request and waits for the success callback for its own transcript id.
The Vertex spend test matches its log by call id instead of taking the first event.
The OpenAI files wire test hit the native /{provider}/v1/files route; it now calls
/openai/files so the passthrough is what forwards the upload and delete.
* test: mock only the HTTP boundary in the migrated audio tests
Vertex TTS tests no longer replace _ensure_access_token or AsyncHTTPHandler.post;
the token comes from a mocked Google OAuth endpoint and the synthesize call from
respx. Speech tests assert the outbound body, the transcription cache test polls
for the cache write instead of relying on test ordering, and the model pass-through
test checks the multipart model field per model.
* test: wait on a logger event instead of polling the clock in anthropic messages unit tests
Recorders keep payloads in a rebound tuple and set an asyncio.Event; tests
await it with asyncio.wait_for instead of a sleep-and-deadline poll loop
* test: freeze module-level batch and file response fixtures as Final MappingProxyType
* test: fake the presidio analyzer with an in-memory aiohttp connector
The blocked-entity tests started an aiohttp TestServer, which binds a local
socket. They now hand the guardrail a ClientSession whose connector answers
/analyze and /anonymize in process, so no socket is opened and the outbound
analyze text and entities are still asserted
* test: type anthropic messages router test helpers with LiteLLM's Anthropic TypedDicts
Messages, cached system blocks and tool-search tools now use
AnthropicMessagesUserMessageParam, AnthropicMessagesTextParam,
AnthropicToolSearchToolRegex and AnthropicMessagesTool instead of bare
dict shapes; tools are converted to plain dicts only at the acreate call,
whose tools parameter is list[dict]
* test: type batch limiter helpers with TypedDicts and wait on the logging callback event instead of polling
* test: give the migrated OCR, image and presidio helpers precise types
OCR spend helpers take ReadOnly TypedDicts for kwargs and responses and use
LiteLLM's OCRResponse/OCRUsageInfo instead of local pydantic stand-ins; spend
metadata is validated with a TypeAdapter. The presidio fake uses LiteLLM's
PresidioAnalyzeRequest/ResponseItem types, and the image-edit logger validates
the logged payload instead of storing an untyped dict
* test: signal callback and cache events instead of polling
Recorders keep tuples and set an asyncio.Event, thread-safely, when the payload
for this test's transcript id or upstream URL arrives. The transcription cache
test waits on a Cache subclass that signals after async_add_cache. No clock
polling or sleeps remain in these tests.
* test: assert the batch limiter hook updates the caller's request in place
* test: tolerate model-list probes and read native passthrough rows by owned key
The router's OpenAI-compatible model-info refresh (litellm/router.py:10710)
sends GET /v1/models to configured openai api_bases, so the wire answers it
with an empty list and excludes it from the provider-call assertions. Native
/anthropic spend rows are now read by a per-request virtual key digest and
call_type, then the row's request_id is checked against the message id
* test: expect the OCR alias in the proxy response model
The proxy restamps every OpenAI-compatible response model to the name the
client requested (_override_openai_response_model), so /v1/ocr returns the
scenario alias. The upstream model is now checked on the drained request body
instead of inside the peer, where a failed assert never reached the test
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
04d97abffb
|
test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs (#45298)
* test: make 77 legacy live tests offline in litellm_utils, router_unit and responses dirs * test: restore lost coverage in litellm_utils offline replacements Bound live health-check tasks to the concurrency limit so eager task creation behind a semaphore fails, route the Langfuse trace-id check through completion -> Logging.get_trace_id with all four metadata combinations, run the callback dedup checks through acompletion, pass the dynamic key via litellm_params, cover the env-inferred default model list, and add the per-provider audio transcription config lookup * test: restore router coverage lost in the offline move Add the missing non-stream LIT-3058 header/count-once test, pin the UTC minute on every usage-counter test, assert router-level client reuse for transcription, and replace the private selector-attribute checks with routing outcomes per strategy. Assert outbound bodies for speech, rerank, image, assistants and moderation, add timeouts to event waits, and drop the router_unit_tests husk files that no longer hold tests * test: restore sync stream, router sync stream, error event and field-type coverage for migrated responses tests Add the sync streaming logging and sync Router.responses streaming cases the offline replacements dropped, port the legacy per-event and response field-type validation, assert raw headers, the float created_at conversion, MCP tool headers, the search_context_size input, and add an offline replacement for the in-stream context-window error event. Remove helpers left dead in the legacy file. * test: run numpydoc-backed unit tests in GHA and drop a test that pinned a crash * test: restore sequence number, item id and content part checks in the responses stream validator * test: match router logging events to the test's own deployment so late events from other tests are ignored * test: restore the s3 cold storage history test to its original assertions --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
Priyansh Nandwana
|
7c7b0ea85b
|
fix(bedrock_mantle): send OpenAI explicit prompt cache breakpoints for GPT-5.6 and newer (#38729)
* fix(cache_control): let a hosted deployment opt into the OpenAI cache dialect _targets_openai_prompt_cache_breakpoint gated on custom_llm_provider == "openai" unconditionally, so an OpenAI-shaped model served by another provider could never qualify, even with supports_prompt_cache_breakpoint set explicitly on its own cost-map entry. bedrock_mantle honours prompt_cache_breakpoint end to end, and had no way to say so. model_cost is keyed per exact deployment string, so a flag on the deployment's own entry states the dialect more precisely than a provider name can. Consult it before the provider check, and require the entry's litellm_provider to match the serving provider so an openai entry cannot license another provider. Entries for the openai provider keep their api_base check, so an OpenAI-compatible third-party host is still not assumed to speak the dialect. Fixes #38666 * test: drop a laziness assertion the existing suite already makes test_provider_lookup_skipped_for_models_below_gpt_5_6 already pins that _resolve_provider is not called for an unflagged model, and it covers the new flag lookup unchanged. The duplicate patched a litellm internal for no added coverage, which the test-quality gate counts against TQ008. * fix(bedrock_mantle): flag GPT-5.6 and newer rows for OpenAI explicit prompt cache breakpoints * fix(responses): predict the chat-completions bridge with the model name the dispatch resolves * fix(cache_control): read a region-prefixed Mantle GPT deployment through its region-free price-map row * fix(cache_control): let a bare hosted deployment name read its own provider's price-map row * fix(responses): hand the hook the provider the router resolved for a Foundry GPT deployment * fix(cache_control): read the deployment breakpoint flag strictly and default a null prompt_cache_options A cost-map or deployment `supports_prompt_cache_breakpoint` that is not the boolean `true` (the string "true", the integer 1, "false", 0, a long string) no longer opts a deployment into the OpenAI prompt cache dialect; only `true` does, the same reading Bedrock Converse applies to its own flag. A client that sends `"prompt_cache_options": null` on `/v1/chat/completions` or `/v1/messages` now gets the implicit default the hook already applied on `/v1/responses` when it placed a breakpoint; before, the null suppressed the default and the request left with a breakpoint and no options. * test(integration): audit cells for the Mantle GPT prompt cache breakpoint dialect Deterministic cells for the configured-breakpoint path on Bedrock Mantle GPT and Azure AI Foundry GPT-6 deployments: every endpoint, streaming and not, sync and async SDKs and raw httpx, the hostile option shapes, flag precedence, the response cache twin, a two-worker burst, a worker kill and a graceful restart. * fix(cache_control): ignore a malformed cache_control_injection_points value instead of failing the request A deployment or client `cache_control_injection_points` that is not a list of points (a string, an integer, a bare point dict, a list of strings) raised inside the prompt hook (`'str' object has no attribute 'get'`, `'int' object is not iterable`) and turned every request to that deployment into a 500 on `/v1/chat/completions`, `/v1/messages` and `/v1/responses`. Every entry point now reads such a value as no configured points, the way a `null` already read, and the request leaves without a breakpoint; entries of a list that are not points are dropped and the point entries kept. * test(integration): cover int, dict and string-list injection point shapes in the Mantle audit cells The D9 cell now also sends an integer, a bare point dict and a list of strings as the deployment's `cache_control_injection_points`, which the merge base answered with a 500 on every request. * fix(cache_control): stamp the OpenAI dialect for a bare deployment name served by a flagged provider A deployment written as `model: openai.gpt-5.6-sol` with `custom_llm_provider: bedrock_mantle` has no cost-map row of its own and no openai row of the same name, so the dialect stamp's cheap gate (`supports_openai_prompt_cache_breakpoint`) returned early and the configured point was never stamped. On `/v1/responses` and on the chat seeding path the hook sees no provider, so it fell back to the Anthropic `cache_control` marker, which Bedrock Mantle strips. The gate now also passes a model whose serving provider is already known and whose provider-keyed row carries the flag, which is the same row the dialect resolution reads and costs no provider lookup. A bare name without a provider is still left alone, so models below GPT-5.6 keep their points untouched and resolve nothing. * test(integration): cover a bare Mantle deployment name with its provider in the audit cells A deployment configured as `model: openai.gpt-5.6-sol` plus `custom_llm_provider: bedrock_mantle` sends the breakpoint and the implicit options on all three endpoints; the merge base leaves it on the Anthropic dialect. * fix(cache_control): keep a configured point that sits beside junk entries on every chat seed path `configured_injection_points` kept the point dicts of a mixed `cache_control_injection_points` list as a tuple, but only read a `list` back. The chat seed and the `/v1/responses` dialect stamp write the normalized value back onto the request, and on a deployment the OpenAI dialect does not stamp (an Anthropic model, Claude on Bedrock Mantle) that value is the tuple itself, so the hook then read it as no configured points and the valid point was silently dropped. The stamped OpenAI dialect and the `/v1/messages` path kept it only because they build a new list. The normalizer now reads back the tuple it wrote, so the point reaches the wire on every path; an all-dict list still passes through as the same object. * test(integration): cover a configured point beside junk entries on a Claude Mantle deployment A deployment configured with `cache_control_injection_points: ["system", {system point}, 3]` marks the system block on the Anthropic dialect on all three endpoints; the merge base answers 500 on the junk entry. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
9df3d4c96b
|
test(integration): boot owned proxies with a readiness-sized worker healthcheck budget (#45292)
* test(integration): boot owned proxies with a readiness-sized worker healthcheck budget * test(integration): keep owned_proxy_process's extra_arguments for cells that pass their own flags --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
42d2158894
|
fix(responses): map input_audio blocks in the chat-to-Responses bridge (#45224)
* fix(responses): map input_audio blocks in the chat-to-Responses bridge * fix(responses): honor base_model when deciding to drop input_audio under drop_params * fix(responses): carry a dropped audio part's cache breakpoint to the preceding part Dropping an input_audio part under drop_params lost the prompt_cache_breakpoint riding on it, so an injected system marker landing on a trailing audio block left prompt_cache_options explicit with no breakpoint on the wire. The marker now moves to the nearest preceding kept part when that part has none. Update the CircleCI-only Responses bridge wire tests that pinned input_audio to its stringified input_text shape: video_url becomes the stringified kind so the marker-on-stringified-block coverage survives, and dedicated audio tests assert the forwarded part and the drop_params carry-over. * refactor(responses): carry dropped audio markers in one pass Replace the per-block slice that re-read the rest of the content list for every retained part with a single reverse scan, and skip the copy for a message that carries no audio part. Type the unit test helper's messages as a sequence of mappings instead of bare lists. * test(responses): add integration cells for input_audio parts through the bridge Cover the chat-to-Responses bridge's input_audio handling on the real proxy: the part is forwarded as input_audio without drop_params and dropped under drop_params unless the model or its base_model supports audio input, across httpx, the OpenAI SDK (sync stream and async), the Anthropic SDK, native /v1/responses, tool and assistant messages, hostile input_audio values, duplicate parts, an audio-only message, marker carry-over, a null drop_params, a global litellm_settings.drop_params, a mixed burst, and a burst with dropped upstream connections * test(responses): check every upstream attempt of a dropped connection The dropped-connection cell required exactly one upstream POST per call, which tied it to the retry policy instead of the fix. It now groups the recorded POSTs by marker, requires one attempt per answered call, and checks that every attempt of a doomed call carries the dropped audio shape. |
||
|
Mateo Wang
|
328f5a720c
|
fix(router): honor a per-request fallbacks list on a mid-stream fallback (#45227)
* fix(router): honor a per-request fallbacks list on a mid-stream fallback The completion entrypoints record the per-request fallbacks, context_window_fallbacks, and content_policy_fallbacks in the mid-stream controls carrier the Responses and Messages paths already use, and each completion attempt restores them into the kwargs its stream re-enters the fallback chain with, so a streaming request that dies before its first chunk fails over to the list the request named instead of the router-level list only. The sync vs async parity cell now asserts the backup's text on both twins. * test(router): annotate the new fallback test locals as Final and flatten the leak check * fix(proxy): skip null key and team router settings when merging per-request overrides * test(proxy): annotate the null router settings test locals and drop a redundant comment |
||
|
Mateo Wang
|
a7ee038592
|
fix(router): keep include_fallback_errors off the provider call on the sync Router path (#45218)
* fix(router): keep include_fallback_errors off the provider call on the sync Router path Router.completion spread include_fallback_errors into litellm.completion, so OpenAI answered 400 Unknown parameter and a fallback walk lost the backup's reply. Only Router._acompletion popped the router-only flags. Both paths now build the provider call through one shared helper that drops them, and a wire-level respx test covers each path. * test(router): pin include_fallback_errors off the wire across the sync, async, and proxy paths |
||
|
Mateo Wang
|
8fc31b1e4e
|
feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions (#45184)
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions The System One request format moves to /v1/systemone and /systemone. /v1/decisions and /decisions now accept the OpenAI Decisions API format, translate it into a System One request, route it through the same pipeline, and translate the answers back. * fix(decisions): import assert_never from typing_extensions for Python 3.10 * fix(decisions): cap OpenAI-format questions at the System One limit /v1/decisions accepted up to 200 questions, but the System One request it translates to takes at most 128, so 129 to 200 questions failed with an internal validation error. Both now share MAX_DECISION_QUESTIONS. * fix(decisions): return 400 for bodies that are not JSON * test(decisions): send System One bodies to /v1/systemone in integration tests * fix(decisions): return guardrail_information on OpenAI-format decisions when requested * refactor(decisions): validate proxy request data before reading guardrail settings |
||
|
devin-ai-integration[bot]
|
0e09e99146
|
refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler (#45130)
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): keep APIConnectionError for unreachable deployments on every provider `_map_upstream_exception` now maps a status `_handle_error` synthesized for a non-HTTP failure to `APIConnectionError` before `exception_type` runs, so a Perplexity (openai-compatible) deployment that cannot be reached no longer answers `InternalServerError` where main answers `APIConnectionError`. Audit cells: the unreachable-deployment integration cell is parametrized over the five providers, OpenRouter connection failures are pinned on chat, stream, embeddings, decisions and the SDK, a deployment whose provider has no decisions config gets the 400 naming every supported provider, and a unit cell pins the mapper alone. The first docstring sentence of the five translation bases is dropped. --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4f49f88172
|
fix(proxy): keep idle responses websockets open until a configurable session limit (#44433)
* add config-driven bound to /responses websocket connection * test(proxy): cover responses websocket session limit and idle first frame Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ci): sync API schema and Ruff formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): close responses websocket client before upstream cleanup at session limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert recorded websocket outcomes and simplify session reaping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): move responses websocket session limit timing coverage to integration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): run provider-close checks off the event loop Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover handshake auth, subprotocol, DB override and worker kill for the responses websocket session limit Adds the audit cells for the configurable session limit: a rejected bearer fails the handshake with 403, a requested Sec-WebSocket-Protocol is echoed back, a DB override through /config/field/update caps new sessions on every worker without a restart while sessions accepted before it keep their limit, deleting the override restores the default, out-of-range and wrong-type updates answer 400 and leave the stored row alone, and SIGKILLing the worker holding idle sockets drops only its sockets while the proxy keeps serving * test(integration): open override sockets until both proxy workers hold one uvicorn workers share one listening socket and a worker that wakes first accepts a whole simultaneous burst, so eight sockets opened at once can all land on one worker. The DB override cell now opens its sockets one at a time, alternating the two paths, until at least eight are open and every worker holds one, bounded at forty --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
efe0715831
|
fix(proxy): accept router-wide default_litellm_params in required-param validation (#44483)
* fix(proxy): accept router-wide default_litellm_params in required-param validation The required-body-param check added in #43787 only consulted request data and deployment-level litellm_params, so a param supplied solely by router_settings.default_litellm_params (e.g. max_tokens for /v1/messages, documents for /rerank) was rejected with a 400 at route entry even though the router would have injected it during dispatch. Consult the router-wide defaults in the same place, treating None-valued defaults as absent to mirror the setdefault merge. * fix(proxy): consult only the defaults the dispatching router will merge Review follow-up: router-wide defaults are now consulted only where dispatch actually applies them. user_config requests dispatch on their own throwaway Router, so they consult that config's defaults instead of the global router's; search, managed-agent and eval routes, and model-less direct dispatch never pass through the router's defaults merge, so they keep the route-entry 400. Also adds rerank router-default coverage (unit + e2e) and drops the redundant source comment. |
||
|
devin-ai-integration[bot]
|
02b8b5cc80
|
fix(proxy): traceparent/baggage fallback must not override caller metadata (#43688)
* fix(proxy): traceparent/baggage fallback must not override caller metadata
The W3C traceparent/baggage fallback in
add_litellm_metadata_from_request_headers documents itself as last-resort:
Lower priority than everything above - only fires when neither the
explicit litellm headers nor the Anthropic-metadata path found anything
But it guards on the top-level `litellm_trace_id` / `litellm_session_id` body
keys and never checks `metadata`, which is the documented way callers set
trace_id / session_id (`metadata: {"trace_id": ...}` on /chat/completions).
A caller that explicitly sets metadata.trace_id has it silently replaced by the
header value, so the implementation contradicts its own stated precedence.
This is not a corner case on managed platforms: GCP's front end injects a
traceparent into every inbound request, so the fallback fires on traffic whose
caller never sent the header. The request still returns 200 and the trace still
reaches the logging backend, just under an id the caller never chose, so any
caller correlating by its own id silently fails to find its trace.
Guard both fallbacks on the caller's request-body metadata as well.
Deliberately narrow:
- x-litellm-trace-id still outranks the body (documented priority #1)
- a request that steers neither field still adopts traceparent/baggage exactly
as before
- steering is per-field: setting only trace_id still lets session_id come from
baggage
- litellm_metadata is checked too, for LITELLM_METADATA_ROUTES (/v1/responses,
/v1/messages, batches, files)
Also corrects the comment, which understated the guard.
4 new tests; each fails without the source change. The existing traceparent and
baggage tests are unchanged and still pass.
* fix(proxy): check only the active metadata container and ignore empty values
Addresses review. The first version treated any value in either `metadata` or
`litellm_metadata` as caller steering. On LITELLM_METADATA_ROUTES the body
`metadata` is provider-facing and is only promoted later, so a session_id there
suppressed baggage before apply_missing_session_id_policy ran, and a request
with a usable baggage session id got a 400 under `missing_session_id: reject`.
An empty session_id did the same, since the policy treats "" as absent
Now only the active metadata container counts, and only a truthy value, which
matches how apply_missing_session_id_policy decides a session id is present.
The helper takes a typed `object` rather than a bare dict
Adds regression tests for both cases, including the end to end reject path,
and drops the test docstrings
* fix(proxy): count promoted caller trace ids on litellm_metadata routes
Addresses review. On LITELLM_METADATA_ROUTES the proxy promotes the caller's
trace control fields (trace_id, session_id, ...) from `metadata` into
`litellm_metadata`, but only after the header fallback runs. Checking only
`litellm_metadata` let traceparent and baggage fill those fields first, and the
promotion then skipped them because they were already set, so the trace was
recorded under the header ids
The check now also reads `metadata` for fields in
LITELLM_TRACE_CONTROL_METADATA_FIELDS on those routes, and
apply_missing_session_id_policy uses the same check, so `reject` no longer
refuses a request whose session id is about to be promoted
The earlier test asserting that a session_id in `metadata` must not block
baggage on /v1/responses had the premise wrong, since that value is promoted.
It is replaced by tests that assert the promoted caller ids win on
/v1/responses and /v1/messages, with no policy, reject, and generate
* fix(proxy): an empty trace field in litellm_metadata shadows the promoted one
Addresses review. Promotion copies a trace control field from `metadata` into
`litellm_metadata` only when the key is absent, so a key that is present but
empty in `litellm_metadata` wins and the `metadata` value is never used. The
check still looked at `metadata` in that case, which let
`missing_session_id: reject` accept a request that ends up with an empty
session id, and let an empty trace_id or session_id block the headers
The check now follows the same rule as promotion. If the active container has
the key, its value decides. Only when the key is absent does the promoted
`metadata` value count. This makes the conflicting-bucket cases behave exactly
as on main
* fix(proxy): stay within the LIT006 cast budget
The type-discipline gate failed: this branch added 4 unsuppressed `cast()`
calls and LIT006 was already at its limit. Each cast now carries a
`# cast-ok` reason. The two in the helper follow an `isinstance` check that
proves the Mapping. The two at the call sites stay because the method's
`data` parameter is a bare `dict`, and dropping those casts adds
basedpyright unknown-type errors instead. No behavior change
* test(proxy): cover caller metadata precedence over W3C trace headers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): generated session uses promoted caller trace id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move test context into assertion messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for w3c fallback precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tighten audit chaos cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(test): drop stray blank lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): ignore body-less model-info probes in langfuse precedence upstreams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep root trace/session fields on equal ids and usable caller values
The _caller_trace_field gating added for caller-metadata precedence was
presence-aware, which regressed three pre-call behaviors versus main:
- Equal ids in W3C headers and body metadata suppressed the
traceparent/baggage branch entirely, leaving the root
litellm_trace_id/litellm_session_id unset so router fallbacks minted a
fresh uuid4 per request. The header branch now also fires when the
caller value equals the header-derived id, so equal ids stamp the root
fields exactly like main.
- A truthy but non-string metadata value (e.g. session_id 4815162342)
counted as caller-supplied and suppressed the baggage fallback, so
downstream str-only consumers (code interpreter sandbox reuse) got a
new sandbox per turn. _caller_trace_field now counts only non-empty
string values, and the generate policy falls through to generation
when the caller value is not usable.
- On litellm_metadata routes a usable caller session id suppressed the
missing_session_id policies while never landing on the root field, so
the root session stayed unset. The policy now promotes the caller's
usable session id to litellm_session_id instead of leaving it stranded.
* test(integration): non-string caller ids fall back to W3C like empty ids
The audit cell compared a headers leg against a headerless leg for
equality, which only held while a truthy non-string id suppressed the
W3C fallback on both legs. With unusable values ignored again, the
headers leg resolves to the header ids while the headerless leg cannot,
so assert the headers leg's concrete outcome instead, mirroring the
empty/null cell.
* style(proxy): trim the session promotion comment to the non-obvious why
---------
Co-authored-by: Filipe Andujar <filipeandujar@gmail.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Mateo Wang
|
ed9c02d2d9
|
fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item (#45177)
* fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item The ollama/ text stream minted a fresh chatcmpl- id per chunk; every chunk now carries the one id of its response. A tool message with int content raised a TypeError inside ollama_pt and answered 500; it now answers 400 naming the message index and the type it carried. Merged user, tool, and function turns and the text parts of one message were joined with no separator; they are joined with a newline. A streamed tool call with no text rebuilds to content null instead of an empty string, and the Responses bridge no longer opens a message item on a leading empty delta, so a /v1/responses stream on a tool call completes with the function_call alone, as the non-stream call does * fix(ollama): defer the BadRequestError annotation so litellm imports on Python 3.10 to 3.13 * fix(ollama): read the user run lazily, type the prompt helpers, and 400 on a non-string text part * fix(ollama): 400 on a text or image_url part that carries no value * fix(responses): stream bridged output items at contiguous indexes and reject non-object ollama content parts The chat-completions bridge reserved output_index 0 for the message item, so a tool-only stream announced its function_call at index 1 with no item at 0 and the OpenAI SDK's responses.stream() accumulator raised IndexError. Items now take the next index in emission order, reasoning included, and response.completed lists the output in that streamed order. ollama_pt answers 400 for a content part that is not an object and for an image_url object without a url string, instead of a 500 or a silent drop. * fix(responses): list only announced items in the bridged completed snapshot The Responses bridge drops a message item from response.completed when the stream never announced one and its text is empty, so a tool-only stream and a reasoning-only stream end with the items the client saw. The chunk builder's rebuilt content goes back to what main does; aligning it with the non-stream shape is its own change for every provider's consumers * style(responses): sort the itertools import with the other stdlib imports * test(integration): add ollama prompt-tools and responses bridge audit cells |
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
9d3bf29d6d
|
fix(responses): keep cache breakpoints on blocks the bridge stringifies (#40032)
* fix(responses): keep cache breakpoints on requests that reach the Responses API The chat-completions bridge rebuilt every content block without the prompt_cache_breakpoint marker the cache-control hook had just placed on it, so the request went out with prompt_cache_options set to explicit mode and nothing actually marked. Explicit mode caches only what is marked, so those deployments lost the implicit caching they were getting before the injection point was added The native surface had a second miss. The bridging check looked the provider config up with the still-prefixed model while the config keys on the bare one, so every bedrock_mantle model read as having no native Responses support, and role-targeted injection points were deferred to a chat-completions pass that never runs for them * fix(responses): keep prompt cache breakpoint on stringified content blocks * test: cover the responses/ routing prefix in the native cache point case * fix(responses): keep only the fallback-branch marker carry, main already routes the hook #42281 landed the text, image_url and file branch carry and the implicit default on main, and the Responses routing lookup this branch changed has no observable effect there, so the PR shrinks to the unknown-block fallback branch and its regression test * fix(responses): drop a malformed prompt_cache_breakpoint under drop_params in the chat-to-Responses bridge * fix(responses): accept the 30m prompt_cache_breakpoint ttl under drop_params OpenAI's Responses API takes a ttl of 30m on an explicit breakpoint marker, so the drop_params validation keeps it instead of stripping it as an unknown key. Trims the bridge test docstrings to the dated vendor citation. * test(integration): audit cells for prompt_cache_breakpoint through the chat-to-Responses bridge Adds the /audit cells for the bridge's prompt_cache_breakpoint carry: valid and malformed markers on every carrying block kind and on tool and assistant content, the drop_params on and off contract, the hook-injected system marker, the Anthropic SDK path through the Responses adapter, upstream errors, idempotent spend rows, and the chaos cells (mixed burst, upstream outage, slow streams, worker SIGKILL and a proxy restart mid burst), all against a scripted Responses endpoint with the request it received read back by response id * test(integration): give every scripted responses reply its own id The prompt-cache-breakpoint audit's scripted upstream minted the response id from the request's marker, so three identical marked requests shared one id. The spend-log writer skips rows whose request_id already landed, and the idempotent-logging cell saw one row for three calls on every leg. The upstream now mints a unique id per reply like a real provider, and the cells match a response to its request by the marker inside that id. |
||
|
mrinal-berri
|
0e076008e1
|
fix(bedrock): map Scheduled batch jobs to in_progress on retrieve (#45157)
* fix(bedrock): map scheduled batch job status to in_progress Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): cover batch retrieve status lifecycle through the proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): drop legacy covers marker and type job payload as Mapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
c4b48f6b3f
|
fix(anthropic): fail closed when a token-file or inline-token federation credential is missing a rule or organization id (#45178)
* fix(anthropic): fail closed when a token-file or inline-token federation credential is missing a rule or organization id * refactor(anthropic): flatten the federation-request helper * test(anthropic): annotate the new WIF regression tests * test(anthropic): audit cells for legacy federation references missing an id Integration cells drive the seven clients, files, skills, health, Test Connect and a mixed burst against deployments whose token file or inline token reference lacks a rule id or an organization id, asserting the named 401 and no outbound call, with controls for the static key, the environment ids, the blank reference, the unauthenticated call and the complete shapes. The unit test covers the environment ids completing a token file param * test(anthropic): strip host federation ids from the audit proxies |
||
|
D41910
|
87e961fad0
|
fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary (#44800)
* fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary AnthropicStreamWrapper.async_anthropic_sse_wrapper swallowed every upstream exception and emitted its own Anthropic error frame, so a provider drop on a streamed /v1/messages call never reached the proxy's streaming boundary — no failure spend row, no failure callbacks. Re-raise when the stream is proxy-managed (detached failure hook armed) so async_streaming_data_generator runs post_call_failure_hook once and serializes the error frame; when consumed standalone (SDK litellm.messages path), run the logging object's async failure handler and keep the client-facing error frame. Fixes #44742 * test(anthropic): satisfy PT012 single-statement rule in mid-stream error test * fix(anthropic): dispatch both sync and async failure callbacks * fix(anthropic): dispatch both sync and async failure callbacks * test(anthropic): move the regression cases into the mapped test file * refactor(anthropic): read the public detached-failure hook and drop the explanatory comments * test(anthropic): drive the mid-stream failure tests through a real logging object * fix(anthropic): re-raise the provider error from the chat wrapper envelope so the router keeps its pre-content retry * test(integration): cover mid-stream failure bookkeeping of bridged /v1/messages streams Nine integration cells for the chat bridge: raw httpx and the Anthropic SDK (sync and async) streams that fail after the first content delta now land exactly one failure SpendLogs row carrying the provider's error class, the non-streaming 500, the response-cache twin and a client that leaves mid-stream keep their bookkeeping, a 24-stream burst lands every call id once, and the standalone SDK stream reports the provider error to failure callbacks once. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
70e6a2ff24
|
fix(vertex_ai): dial the multi-region Live API host for realtime sessions (#45166)
* fix(vertex_ai): dial the multi-region Live API host for realtime sessions
A realtime deployment with vertex_location us or eu dialed
{location}-aiplatform.googleapis.com, which Vertex does not serve for a
multi-region, so the WebSocket handshake came back 404. The URL builder now
resolves its host through the shared Vertex host resolver, which already
maps us and eu to aiplatform.{geo}.rep.googleapis.com and leaves regions
and global unchanged.
* test(vertex_ai): annotate the realtime multi-region URL tests
* fix(vertex_ai): skip the TLS argument when the realtime api_base is plain ws://
The Vertex realtime session and health-check connects passed the shared
TLS context to every websocket dial, so an api_base the transformation
already maps from http:// to ws:// failed with 'ssl argument is
incompatible with a ws:// URI' before any frame was sent. The OpenAI
realtime handler already skips the argument for ws://; the shared helper
now does the same for the Vertex path
* test(integration): cover the Vertex realtime multi-region host through the proxy
Fourteen cells on the scripted Gemini Live upstream: api_base overrides for
us, eu, us-central1 and global complete a turn; malformed locations are
refused before any dial; a missing location defaults to us-central1; the
realtime health check handshakes the override host and reports a malformed
location without dialing; a refused handshake reaches the client and the
next session connects; twelve concurrent sessions across locations each
reach the upstream once; an upstream outage closes every open session and
the proxy recovers
* test(realtime): type the health-check helpers and flatten the burst order
* test(http_handler): type the realtime TLS test signatures
|
||
|
RachelHuangZW
|
bf9f59d76b
|
fix(scheduler): remove a request's queue entry once it stops waiting (#43061)
* fix(scheduler): remove a request's queue entry once it stops waiting Requests admitted while a healthy deployment existed never left the priority queue, and neither did requests cancelled while waiting. The stale entries blocked later requests during cooldown and, with Redis, made add_request raise a TypeError on queues read back as JSON lists Both scheduling paths now share one polling helper that removes the entry in a finally block, whether the request was admitted, timed out or cancelled Related to #43059 * test(router): allowlist _wait_for_scheduler_turn in the router coverage check The coverage script only counts direct calls in test files. The helper is exercised through prioritized acompletion and atext_completion in test_router.py, like the other allowlisted entries * fix(scheduler): admit healthy requests before reading the queue, and clean up cancelled enqueues poll() raised on an empty queue before checking for healthy deployments. With the cleanup now rewriting the queue after every admission, a concurrent write from another replica can erase a waiting request's entry, and that request then failed while a deployment was healthy. poll() now admits as soon as a deployment is healthy and only reads the queue during cooldown add_request also moved inside the try block, so a request cancelled while its queue write is in flight still has its entry removed * refactor(scheduler): move the wait loop into Scheduler.wait_for_turn The router passes a healthy-deployments callable into the scheduler, so tests inject a Scheduler directly instead of replacing the router's scheduler attribute. The cancelled-mid-enqueue test moves to test_scheduler.py with the other scheduler tests, and poll() takes the deployments as a Sequence since it only checks whether any are healthy * test(scheduler): move scheduler tests into tests/unit * fix(scheduler): finish the queue removal when cancellation is delivered again during cleanup A second cancel, or an anyio cancel scope that re-cancels on every await, interrupted remove_request mid-write and left the entry in Redis. * fix(scheduler): read queue entries back from redis as tuples * fix(scheduler): admit a request whose queue entry vanished during cooldown * fix(scheduler): re-enqueue a request whose queue entry vanished during cooldown * test(integration): cover priority scheduler queue cleanup across instances Adds tests/integration/routing/test_priority_scheduler_queue_cleanup.py: 27 cells against the real proxy (two workers, Postgres, Redis, a scripted upstream) for every prioritized surface (/v1/chat/completions, /v1/completions, /queue/chat/completions, streaming and not, OpenAI SDK sync and async, raw httpx), the non-integer priority pass-through, the in-memory queue on one proxy, two proxies sharing a Redis queue (served requests leave no entry, a dead replica's entry is skipped or expires, a waiter that times out or disconnects removes only itself), a Redis outage mid burst and a SIGKILLed worker. Each cell asserts the caller's response, the upstream's requests by marker and the Redis queue contents. On the merge base the cross-instance cells fail with the list-of-lists TypeError and the in-memory cell with 408s behind the leaked entry; on this branch every cell passes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
6498bef8db
|
fix(mcp): honor scoped cache freshness (#45165)
* fix(mcp): honor upstream freshness and caller scope in discovery caches * test(mcp): type scoped freshness regression helpers * fix(mcp): age discovery freshness through cleanup * fix(mcp): preserve keyless discovery caller isolation --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
moe-berri
|
e9cfba2c17
|
feat(lens): isolate ingestion and investigations in a Rust service (#45148)
* feat(lens): isolate trace storage and investigation in a Rust service
* fix(lens): include Rust sources in the image build context
* feat(lens): wire service setup, scoped delivery receipts and lease attempts
* fix(lens): complete service routing and reject stale investigation results
* fix(lens): retry key propagation and validate isolated Compose setup
* fix(lens): seed through isolated ingestion and preserve upstream queue fixes
* chore: sync schema.prisma copies from root
* fix(lens): bind nullable due timestamps as text for Prisma
* chore(ui): remove stale lint suppressions
* fix(lens): address CI failures and review findings
* refactor(lens): remove retired Python worker and run evaluations in Rust
* fix(lens): reuse control connections and satisfy review checks
* test(lens): install and upgrade both Helm charts on Kubernetes
* test(lens): run connection reuse coverage as an integration test
* fix(ui): upgrade Next.js to 16.3.8 security release
* fix(lens): fence stale attempts and preserve reviewed evidence
* Revert "fix(ui): upgrade Next.js to 16.3.8 security release"
This reverts commit
|