mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
165 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
dfe4df8c8a
|
test: move whole-unit legacy test files into tests/unit and delete dead skips (#44807)
* test: delete unconditionally skipped legacy tests * test: move whole-unit legacy test files into tests/unit * test: keep moved legacy tests free of import-time global state * test: keep the module-level invocation scan pointed at tests/local_testing * ci: drop the agent_testing CircleCI job emptied by the move * test: fix moved-test isolation and router coverage * test: add Tinyfish search package marker * test: isolate moved tests from logger state leaks * test: isolate Helicone logging fixture state * test: isolate Vertex pass-through credentials between moved tests * test: cancel S3 periodic flush tasks started by moved tests * ci: restore CircleCI assistant test selection after move * test: prevent Datadog datetime import shadowing * ci: drop the litellm_assistants_api_testing CircleCI job emptied by the move * test: deduplicate imports in rebased unit tests * test: remove duplicate passthrough router patch import * test: remove moved legacy source files after rebase * test: align moved tests with rebased main * test: carry main's legacy-file edits into moved destinations * test: make the moved cost map fallback tests assert the fetch and the backup The four fallback cases only checked the result was non-empty, so they still passed with integrity validation disabled. They now inject a mock client, assert one fetch happened, and assert the result is exactly the local backup with the fallback reason recorded. --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
62dee3d730
|
feat(claude_code_gateway): issue rotating refresh tokens and a revocation endpoint (#44635)
* feat(claude_code_gateway): issue rotating refresh tokens and a revocation endpoint The Claude Code gateway's device-code grant now returns a refresh token alongside the session JWT, so a Claude Code session renews itself before the JWT expires instead of forcing the user back through the browser sign-in. grant_type=refresh_token re-mints the JWT from the live user row, rotates the refresh token, and refuses a replayed, foreign, or identity-only token with invalid_grant. The discovery document now advertises an RFC 7009 revocation endpoint, which Claude Code's /logout calls with both tokens, so sign-out burns the refresh token. The refresh token is the MCP gateway's sealed session refresh token bound to the fixed client id claude_code, so both front doors share one single-use record. * fix(claude_code_gateway): mint the whole credential before claiming the device code A refresh token that failed to mint answered 500 after the device code was already claimed and the login deleted, so the client could not redeem the completed sign-in again. The response is now built first, and the code is claimed only when it is a 200. * fix(claude-code-gateway): keep device sign-in working when session signing is unusable * feat(claude-code-gateway): end the whole refresh chain on a replay or a revocation * fix(mcp-gateway): fail the single-use peek closed on a Redis fault * fix(mcp-gateway): read the single-use marker under the cache namespace * fix(mcp-gateway): refuse a replayed refresh token without ending its chain A refresh token presented a second time is refused as already used and nothing else happens to the chain it was rotated from. Claude Code renews from its in-memory copy of the credential, so a second terminal on the same machine presents the token the first terminal already rotated and then recovers from the shared credential file; ending the chain there would sign both terminals out at every expiry. Only a revocation ends a chain, so the 503-on-unrecorded-chain-ending path and its tests go away. * fix(mcp-gateway): end the refresh chain before burning the revoked token --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
d1cfe17518
|
feat(mcp): add portable catalog pagination (#44446)
Some checks failed
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / Vertex AI (push) Blocked by required conditions
Unit Tests / misc (push) Blocked by required conditions
Unit Tests / Build the Rust bridge (push) Waiting to run
Unit Tests / caching-local (push) Blocked by required conditions
Unit Tests / core-utils (push) Blocked by required conditions
Unit Tests / enterprise-managed-files (push) Blocked by required conditions
Unit Tests / enterprise-package (push) Blocked by required conditions
Unit Tests / enterprise-routing (push) Blocked by required conditions
Unit Tests / integrations (push) Blocked by required conditions
Unit Tests / OpenAI and Meta Providers (push) Blocked by required conditions
Unit Tests / All Other Providers (push) Blocked by required conditions
Unit Tests / misc-dirs (push) Blocked by required conditions
Unit Tests / proxy-auth (push) Blocked by required conditions
Unit Tests / proxy-endpoints (push) Blocked by required conditions
Unit Tests / proxy-extras (push) Blocked by required conditions
Unit Tests / proxy-feature-endpoints (push) Blocked by required conditions
Unit Tests / proxy-hooks-client (push) Blocked by required conditions
Unit Tests / proxy-infra (push) Blocked by required conditions
Unit Tests / proxy-infra-root (push) Blocked by required conditions
Unit Tests / proxy-server (push) Blocked by required conditions
Unit Tests / responses-caching-types (push) Blocked by required conditions
Unit Tests / unit (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
* feat(mcp): add portable authorized catalog pagination * fix(mcp): preserve routing and isolation across catalog pages * fix(mcp): preserve bare calls after complete catalog pages * test(mcp): verify bare calls with cached database catalog revisions * test(mcp): cover refreshed authority and failed catalog continuations * refactor(mcp): narrow pagination interfaces and preserve listed metadata * fix(mcp): publish listed metadata only after successful pagination * fix(mcp): defer bare routes until aggregate listing succeeds * fix(mcp): refresh key access groups on catalog pages * fix(mcp): accept read-only optional catalog headers * fix(mcp): preserve consolidated listing metadata after restack * fix(mcp): propagate unexpected catalog continuation failures * fix(mcp): preserve key grants when a user record is absent --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
joshua-berri
|
950da7c2ec
|
fix(mcp): keep worker MCP configurations consistent via a catalog revision (#42568)
* fix(mcp): refresh server catalog for each gateway operation * fix(mcp): reject configuration changes during scoped dispatch * fix(mcp): refresh shared catalog state for each operation * fix(mcp): refresh catalog before native alias routing * docs(mcp): clarify native route resolution order * test(mcp): align streaming fixture and generated API documentation * fix(mcp): preserve discovery published during catalog refresh * fix(mcp): coalesce queued catalog reads without a stale window * fix(mcp): reconcile concurrent route changes when publishing catalog * test(mcp): provide catalog scope in post-call hook fixtures * fix(mcp): retain valid live routes and handlers during refresh * fix(mcp): keep refreshed OpenAPI operation membership authoritative * test(mcp): preserve logging fixtures after catalog integration * fix(mcp): isolate transport test admission state and exhaust OAuth outcomes * refactor(mcp): narrow catalog consistency change to ticket scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(mcp): gate catalog refresh on a database revision marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): refresh waiters that observed a newer catalog revision Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): awaitable catalog revision doubles in prisma mocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): satisfy catalog refresh lint and type gates * test(mcp): provide catalog scope in toolset fixtures * test(mcp): exercise temporary OAuth through catalog operations * fix(mcp): share active catalog snapshots across discovery tasks * fix(mcp): retain discovered routes across cached catalog operations * fix(mcp): preserve local tool ownership across discovery * refactor(mcp): satisfy tightened immutability lint ceiling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): authorize local handlers by registered server ownership * fix(mcp): check registered handler freshness and isolate fixtures * fix(mcp): read catalog revisions and snapshots from the writer * fix(mcp): resolve access groups from the operation catalog * fix(mcp): preserve empty access group restrictions * fix(mcp): retain rediscovered routes across concurrent updates * fix(mcp): enforce registered ownership for local tool dispatch * fix(mcp): preserve issuer discovery across catalog snapshots --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
b9ab1eee1f
|
fix(proxy): stop logging license values during verification (#44956)
* fix(proxy): stop logging license values during verification * fix(tests): address license logging review feedback --------- Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> |
||
|
tin-berri
|
9a3f000c9b
|
fix(cli): show full-session auto-router cost comparison (#44959) | ||
|
ishaan-berri
|
5fb23eabd8
|
feat(lens): add datasets built from real traces (#44765)
* feat(lens): add dataset case and size limits * feat(lens): add dataset, case and build models * feat(lens): build dataset cases from traces, findings and text * feat(lens): store dataset revisions insert-only * feat(lens): add dataset routes for build, save, export and eval cases * feat(lens): mount the dataset router before lens routes * feat(lens): add LiteLLM_LensDataset table * feat(lens): add LiteLLM_LensDataset table to proxy schema * feat(lens): add LiteLLM_LensDataset table to extras schema * feat(lens): add migration that creates the dataset table * test(lens): cover dataset case building, dedupe and limits * test(lens): cover dataset revisions, conflicts and eval cases * chore(ui): regenerate API types for lens datasets * feat(lens): add dataset UI types * feat(lens): add datasets API client * feat(lens): add dataset query and mutation hooks * feat(lens): add case selection and expected edit logic * test(lens): cover case selection and expected edits * feat(lens): add the add to dataset dialog * test(lens): cover saving picked cases from the dialog * feat(lens): add datasets list * feat(lens): add dataset detail with revisions and export * test(lens): cover editing, revisions and export in datasets tab * feat(lens): expose datasets on the lens API * feat(lens): add in-memory datasets for demo mode * feat(lens): wire demo datasets into the demo lens API * feat(lens): add optional lens API hook * feat(lens): add optional onboarding hook * feat(lens): add datasets tab and dataset routing * feat(lens): show the datasets tab * feat(lens): add to dataset from the trace header * feat(lens): add a single turn to a dataset from a step * feat(lens): add finding evidence to a dataset * docs(lens): add datasets screenshots for the PR * test(lens): cover dataset revision storage against Postgres * test(lens): cover dataset trace paging, findings and route errors * test(lens): cover dataset build fallbacks and no_content skips * fix(lens): register the dataset table for postgres span names * feat(lens): add invalid skip reason for unparseable case lines * fix(lens): keep valid JSONL cases, provenance and size limits; stop reading past the case cap * test(lens): cover malformed JSONL, re-import provenance, size fields and early cap * chore(ui): regenerate API types for the invalid skip reason * feat(lens): show text for the invalid skip reason * feat(lens): render datasets in the same inspector table as investigations * test(lens): open a dataset by clicking its table row * docs(lens): update the datasets list screenshot * style(lens): format the datasets table * fix(lens): normalize JSON text span input and output into messages when building cases * test(lens): cover JSON text span normalization for dataset cases * Revert "fix(lens): normalize JSON text span input and output into messages when building cases" This reverts commit 29c3e34962fa12427dbd516c08f2599db997a905. * fix(traces): normalize agent assistant summaries into UI messages * feat(lens): add dataset case view helpers built on the trace parsers * test(lens): cover dataset case view helpers * feat(lens): show dataset cases in an inspector table * feat(lens): open a dataset case in a side panel with trace message cards * feat(lens): rebuild the dataset page header and layout * feat(lens): keep the open dataset case in the URL * test(lens): drive dataset edits through the case table and panel * docs(lens): update dataset view screenshots * Revert "test(lens): cover JSON text span normalization for dataset cases" This reverts commit 30229ae8ae90265c09d755c43c1f1eeaa66050c7. * fix(lens): import StateMessage from shared in dataset detail * fix(lens): import StateMessage from shared in datasets list * fix(lens): give the datasets migration a unique timestamp after review checkpoints |
||
|
Sam Mobach
|
292fcb1c65
|
fix(proxy): resolve oidc/ pass-through credentials on every request (#44577)
* fix(proxy): resolve oidc/ pass-through credentials on every request A pass-through credential (a use_in_pass_through deployment's api_key or the provider's env var, e.g. TYPESAFE_API_KEY) was sent as a literal string, so an `oidc/...` reference such as `oidc/file//var/run/secrets/<name>/token` ended up on the wire as `Bearer oidc/file/...`, and a rotating projected Kubernetes service account token could not be used for pass-through auth. Resolve credentials that start with `oidc/` through get_secret_str() on every get_credentials() call, so the file is re-read and rotations apply without a restart. oidc/file/ keeps its credential-directory allowlist; other credentials are returned unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): only resolve local oidc/ pass-through credentials inline get_credentials() runs synchronously inside async pass-through routes, so resolving oidc/google/, oidc/github/ etc. on a cache miss would make a blocking HTTP call to the identity provider (timeout up to 600s) on the event loop. Limit per-request resolution to oidc/file/, oidc/env/ and oidc/env_path/, which need no network I/O; network-backed references are left unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(proxy): surface unreadable oidc/ pass-through credentials as 401 A missing or disallowed token file raised FileNotFoundError or OidcPathNotAllowedError out of the route, which the proxy turned into a generic 500 on every request. The resolver now raises an HTTPException 401 that names the credential reference and the reason, matching how the pass-through routes already report an unset key. oidc/env_path/ is no longer resolved inline because it reads any file named by an environment variable without the oidc/file/ credential directory allowlist; it is forwarded unchanged like the network-backed references. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> |
||
|
tin-berri
|
a193c67347
|
feat(otel): trace auto-router configuration and classifier failures (#44926) | ||
|
devin-ai-integration[bot]
|
2477635213
|
fix(proxy): bound daily spend rollup row-lock waits with lock_timeout and requeue 55P03 (#44450)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
acd2a7ccc5
|
fix(lens): use async-timeout on python 3.10 for budget reservation timeouts (#44911)
* fix(lens): use async-timeout on python 3.10 for budget reservation timeouts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(deps): keep uv.lock diff to the async-timeout entry Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(lens): cover real request deadline expiry in reserved_budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(lens): finish Python 3.10 timeout coverage and dependency checks * test(lens): control event-loop time for deadline regressions * test(tracing): include priced call count in trace fixture * fix(lens): limit timeout compatibility changes to PR scope --------- Co-authored-by: Moe Khalil <moe@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
a3e74a5223
|
fix(tracing): preserve optional provider evidence in fixture replay (#44930)
* fix(tracing): preserve optional provider evidence in fixture replay * fix(tracing): keep copied provider identities consistent |
||
|
tin-berri
|
2cecc7e33f
|
fix(auto-router): separate tuning and select chained heuristic (#44928) | ||
|
devin-ai-integration[bot]
|
282d733fb6
|
feat(auth): deny search tools by default when search_tool_deny_by_default is set (#44490)
Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
abc543e701
|
feat(proxy): add opt-in vector_store_deny_by_default for least-privilege vector store access (#44244)
* feat(proxy): add opt-in vector_store_deny_by_default for standalone virtual keys
Adds general_settings.vector_store_deny_by_default (typed bool, default false). When enabled, a virtual key
with no team must list the requested vector store in its object_permission.vector_stores; no permission
record, null, or an empty list is denied with key_vector_store_access_denied. Omitted or false keeps the
existing behavior, including nonempty allowlist enforcement. The master key is unchanged in both modes.
Team keys and keyless callers are deferred to later increments of LIT-6035
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): require key and team vector store grants for team keys under vector_store_deny_by_default
With the flag enabled, a virtual key on a team needs both its own grant and its team's grant for every requested vector store. A missing permission record, an empty list or an unresolved team grants nothing. Dashboard session keys and the master key keep their existing behavior, and flag-off behavior is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): require user or team vector store grants for keyless requests under vector_store_deny_by_default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover vector_store_deny_by_default through a real proxy with key, team and JWT identities
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for vector_store_deny_by_default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): cover path and file_search vector store ids under vector_store_deny_by_default
Strict mode now reads vector_store_ids and tools[].vector_store_ids from the request body without needing a vector store registry, so /v1/vector_stores/{id}/search and Responses file_search are checked. User grants load through the object permission cache, and the proxy admin user rebuild keeps object_permission_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast user entitlement cache eviction to every worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): reuse VectorStoreRegistry id extraction for vector_store_deny_by_default
Strict mode now uses get_vector_store_ids_to_run on an empty registry when none is loaded, instead of a parallel set of request-shape helpers, and vector_store_access_check documents the policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add strict vector store audit cells for routes, SDKs, workers and concurrency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): consolidate vector_store_deny_by_default coverage to core cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject invalid vector_store_deny_by_default at config load and return 400 for malformed vector_store_ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): read vector store key and team grants through the object permission cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): return typed permission rows in request flow vector store tests
Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): validate vector store ids and tools as immutable sequences
Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
26a9b02f7b
|
feat(otel): let team and key Arize callbacks choose the OTLP transport (#44492)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
5147aefa8b
|
fix(lens): reuse trace reviews and consolidate findings across runs (#44778)
* fix(lens): reuse trace reviews and consolidate findings across runs * chore: sync schema.prisma copies from root * fix(lens): preserve partial reviews and bound budget admission * fix(lens): resolve CI regressions and clarify reused runs * test(lens): exercise review reuse in worker container smoke * fix(lens): show actual reviews when a run finishes * fix(lens): distinguish reuse plans and stop blocked scans * fix(lens): retain partial findings when model requests stop * docs(lens): clarify budget edits during active runs * fix(lens): publish findings only after reconciliation completes * fix(lens): guide insufficient-budget runs to budget settings * fix(lens): renew budget holds and bound admission waits * fix(lens): preserve provider errors during budget cleanup * fix(ci): update PgBouncer and sharp for current builds * fix(lens): serialize settlement and fence review checkpoints * fix(lens): defer generated Prisma client type import * test(lens): verify checkpoint and progress rollback together --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
837c6a7481
|
fix(security): remove the publicly known master key from the repo (#44718)
* fix(security): hash the publicly known master key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: replace weak master key examples and regenerate artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: replace weak key fixtures with generated test keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: generate master keys for proxy startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve lens dev key entropy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore proxy key compatibility in scrub examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: scrub merged SSO fixture and refresh dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): stabilize test keys and metadata collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d4a791d650
|
refactor(types): replace Any with proven types in 157 files (#44798)
* refactor(types): replace Any with proven types in 299 files Clears 871 basedpyright Any errors (reportAny 6,703 to 6,240, reportExplicitAny 1,651 to 1,243) without adding a cast, an ignore or a suppression, and without touching any budget file Most edits are annotation-only: a parameter, return or local goes from Any to object, Mapping[str, object] or the concrete type the value always held. Fourteen files validate untyped JSON once where it enters, through a module-level pydantic TypeAdapter or model_validate, and then use real types No HTTP status, error type or response shape changes. Mistral speech and fal.ai Bria image generation now report a pydantic ValidationError instead of an AttributeError when the provider answers 2xx with a body that is not a JSON object * refactor(types): make the config locals fix effective and trim no-op edits The Mapping[str, object] annotation on `locals().copy()` removed no error, because the checker narrows the variable back to the dict[str, Any] the call returns. 21 provider config constructors now build the same copy with dict(locals()), which the checker infers as object values under that annotation, so each file loses one reportAny. The same annotation is reverted in 31 other config files where it stayed a no-op, together with the tests that were added only to cover those lines, and the one MCP server manager line that no CI coverage shard executes is reverted too. The pull request drops from 345 to 297 changed files. * refactor(types): accept only int in the proxy state setter get_proxy_state_variable is annotated to return int, but set_proxy_state_variable still took Any, so the checker could not hold callers to the type the getter promises. The setter now takes int, which is what its only caller already passes. * refactor(types): index the proxy state key so the getter returns int * refactor(types): keep public annotations and provider error text unchanged Restore every public return, public method parameter, public attribute and exported alias to its annotation on main so code that type-checks against the package keeps type-checking, and take the Mistral speech and fal.ai Bria changes back out so no provider error message differs from main * refactor(types): leave the Vertex RAG chunking read as it is on main Take the chunking format validation back out of the Vertex RAG ingestion path. It needs the vertexai SDK, a storage bucket and a RAG corpus to execute, so nothing here could run it end to end, and it cleared only two errors * test(integration): pin the validated provider boundaries on a live proxy * test(integration): give the held burst a client that outlasts the gate The fault cell holds a burst at the upstream for up to 60 seconds while it kills a worker, but sent the burst through the shared 15 second client, so a slow box could time the survivors out before the gate opened. The burst now goes through its own client whose timeout is twice the gate, and the gate length is one named constant. * refactor(types): keep the license reply handling and experimental MCP signatures as they were The license check validated the whole reply as a mapping, which changed the error text logged for a reply that is not an object. It now validates only the verify value, so every reply is handled and logged exactly as before while the value is still typed. Three files under the experimental MCP server changed annotations on public functions and methods (three returns and three parameters). They go back to their previous content so no public signature in the diff is narrowed. * test(integration): answer the proxy's model-list call in the OpenAI stand-ins Every 300 seconds each proxy worker asks an OpenAI deployment for GET /v1/models. Four new cells own an OpenAI stand-in that accepted only the call under test, so a refresh landing inside a cell failed it. The stand-ins now answer that call through the suite's own helper and the cells count only the provider calls they drive. |
||
|
devin-ai-integration[bot]
|
5455152913
|
refactor: remove fresh tech debt from the 2026-10-05 window (#44821)
* refactor: remove fresh tech debt from the 2026-10-05 window Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(lens): drop the review models left unused by the dead review helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c13746c951
|
fix(router): price a model group from the deployments that serve it (#44732)
* fix(router): price a model group from the deployments that serve it A model group's info read composed the group's own model_group_alias entry into its deployments, a hop the router never takes: an alias is resolved exactly once at request time, so a group reached as an alias target is served by its own deployments. For the chain X -> T -> U the price read for X included U's deployments too, and the free-model budget waiver refused a free request to X on an over-budget key, while GET /model_group/info reported U's providers and price for X. The group info read now prices a group from the deployments routing serves it with: the ones named after it, the routing group of that name, or the wildcard route matching it when neither exists. The budget waiver, GET /model_group/info, the rate limiters, and the response headers all read the same set as routing. get_model_list keeps its behavior for every other caller. * test(router): give the paid fixtures explicit per-token prices * test(router): call the routed-group read by name so the router coverage gate sees it * test(integration): audit the alias chain budget waiver on every route, shape, and outage Thirty-four cells under the management group prove an over-budget key is served through an alias chain entry at the price of the deployment that serves it, on chat, responses, and messages, sync and streamed, through the OpenAI and Anthropic SDKs and raw httpx on both replicas, with the chain middle, the reverse chain, a cost-map priced middle, a ghost middle, wildcard and routing-group targets, malformed and hostile inputs, a cached reply, a repointed alias, a provider failure, and two chaos bursts (a killed worker, a scripted outage) --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
402fa63366
|
feat(proxy): cap batch file records, daily batch uploads, and per-file downloads (#43632)
* feat(proxy): cap batch file records, daily batch uploads, and per-file downloads
Adds three opt-in limits for batch jobs, each settable in general_settings as a
per-key default and overridable in key or team metadata by a proxy admin:
max_batch_file_records rejects a purpose=batch upload with more request lines
than allowed with a 413 before it reaches the provider.
max_batch_file_uploads_per_day counts accepted batch uploads per key and per
team in a UTC day and returns 429 with Retry-After once the count is used.
max_file_downloads_per_minute counts GET /v1/files/{id}/content per key and per
team for each file in a one-minute window and returns 429 with Retry-After.
The existing admin-only guard for batch_enqueued_token_limit now covers all four
metadata keys.
* fix(proxy): keep file usage counters in their own store and gate batch limits on user creation
* fix(proxy): count keyless JWT callers per user, require positive file caps, and take the upload slot after request validation
* refactor(proxy): end file usage cap describers with an explicit return after the match
* fix(proxy): keep file usage counters when more than 200 are live without Redis
The file usage counter store used a default in-memory cache, which holds 200
entries and evicts the one that expires soonest. Without Redis, a caller got a
fresh per-file download allowance after touching about 200 other file ids in
the same minute, and a key got a fresh daily upload allowance once about 200
other keys had uploaded that day. The store now tracks up to 20,000 live
counters per worker, the same bound the login throttle uses
* test(proxy): move the file usage cap tests into the directory the proxy shard runs
Main's shard coverage check found tests/unit/proxy/openai_files_endpoints
claimed by no shard, so its tests would not run in CI. The file moves next to
the other files endpoint tests in tests/unit/proxy/openai_files_endpoint, which
the proxy-endpoints shard already runs
* fix(proxy): declare the file usage counters as rate limit calls
Main's redis producer gate requires every module that writes a shared cache to name its key family, and the file usage counters wrote theirs without one.
* test(files): audit batch file usage caps across processes, Redis outages, and config reloads
* test(files): guard the chaos cells against minute boundaries and open Redis breakers
Two chaos cells each failed once in the audit run. The restart check ran
three sequential downloads with no guard against straddling a UTC minute,
and the exact-cap probe after a Redis outage ran while both workers' Redis
circuit breakers were still open (60 s default recovery), so it counted in
per-process memory and the two workers split the cap
Every burst now carries a window guard, the chaos fixture lowers the
breaker recovery to 2 s, and the post-outage check drives a fresh key to its
cap through a one-worker sibling proxy and then expects the two-worker
candidate to refuse the whole burst, which only the shared Redis count can
produce, polled until the breakers close
* test(files): release the held uploads when the killed-worker cell fails early
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
devin-ai-integration[bot]
|
4a963a6c0b
|
fix(proxy): judge the free-model budget waiver by the group an alias routes to (#44638)
Some checks failed
Unit Tests / Build the Rust bridge (push) Waiting to run
Unit Tests / caching-local (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / jwt-and-keys (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / key-generation (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / logging-misc (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-runtime (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / core-utils (push) Blocked by required conditions
Unit Tests / enterprise-managed-files (push) Blocked by required conditions
Unit Tests / enterprise-package (push) Blocked by required conditions
Unit Tests / enterprise-routing (push) Blocked by required conditions
Unit Tests / integrations (push) Blocked by required conditions
Unit Tests / OpenAI and Meta Providers (push) Blocked by required conditions
Unit Tests / All Other Providers (push) Blocked by required conditions
Unit Tests / Vertex AI (push) Blocked by required conditions
Unit Tests / misc (push) Blocked by required conditions
Unit Tests / misc-dirs (push) Blocked by required conditions
Unit Tests / proxy-auth (push) Blocked by required conditions
Unit Tests / proxy-endpoints (push) Blocked by required conditions
Unit Tests / proxy-extras (push) Blocked by required conditions
Unit Tests / proxy-feature-endpoints (push) Blocked by required conditions
Unit Tests / proxy-hooks-client (push) Blocked by required conditions
Unit Tests / proxy-infra (push) Blocked by required conditions
Unit Tests / proxy-infra-root (push) Blocked by required conditions
Unit Tests / proxy-server (push) Blocked by required conditions
Unit Tests / responses-caching-types (push) Blocked by required conditions
Unit Tests / unit (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
VS Code Extension / vscode-extension (push) Has been cancelled
* fix(proxy): judge the free-model budget waiver by the group an alias routes to A hidden model_group_alias can reuse the name of a real model_name. The router serves that name from the alias target, but two of the three checks behind the free-model budget waiver read the deployments of both the alias target and the real model that shares the name. So an over-budget key was served through an alias whose name belongs to a model with an explicit $0 price when the target is $0 only by the cost map, which the same key is refused on by its own name. The mirror case refused a free target because the shadowed name belongs to a PTU-priced deployment. Resolve the alias once and have the explicit-cost and PTU checks read the routed group, the same group the price check already reads. * test(proxy): cover the plain alias form of a shadowed free model name The same wrong verdict exists for a plain string alias, so the unpriced-target case now runs for both alias shapes. * fix(proxy): judge an alias chain's budget waiver by the deployments it is served from The explicit-price and PTU checks read the alias target through Router.get_model_list(), which follows a second alias hop when the target is itself an alias key. The router never takes that hop, so an alias chain was judged by a deployment the request never reaches. The checks now take the deployments named after the routed group, or the wildcard deployment serving it when none carries its name. * fix(proxy): refuse the budget waiver when an alias chain is served by a priced wildcard route * test(integration): cover the shadowing alias budget gate end to end Adds the audit cells for a hidden alias whose name shadows an explicitly free group: streamed SDK refusals on chat, responses and messages, embeddings, the free wildcard and mixed-group paths, per-model budgets, JWT and custom auth callers, cache hits, alias removal under traffic, provider failure and fallback, a concurrent outage burst, and a worker kill on an owned two-worker proxy * test(integration): cite the cost-map rows the shadowing alias cells rely on --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
ff5084687c
|
fix(ci): namespace claude session ids in tracing seeds and allowlist /v1/logs on backend (#44761)
* fix(ci): namespace claude session ids in tracing seeds and allowlist /v1/logs on backend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): allowlist /v1/logs on the gateway alongside /v1/traces Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
6f5ca84f69
|
feat(prometheus): cap series per metric for every labeled metric (#44420)
* feat(prometheus): cap series per metric for every labeled metric Add prometheus_metrics_max_series_per_metric: per metric and per worker process, the first N label sets keep a series of their own. Counters and histograms record every later label set on one series whose labels are all "other", so totals stay exact, and gauges skip it. The cap holds with multiple workers because it never needs to remove a series. Add prometheus_metrics_ttl_seconds: a series idle for that long is removed and its slot is freed. The prometheus client cannot remove a series in multi-process mode, so the TTL is ignored there with a startup warning. Both settings are off by default. The end_user caps are unchanged. * fix(prometheus): share the series cap across workers of one proxy instance Workers writing to one PROMETHEUS_MULTIPROC_DIR now agree on which label sets get a series through an append-only admissions file per metric, so a merged scrape stays at the cap plus `other` instead of growing with every worker and every worker restart. The two fallback counters now pass their label names as a keyword so the cap and prometheus_exclude_labels apply to them, admission and child creation happen under one lock, the test fixture restores the shared registry, and the `other` label value lives in constants.py. * test(prometheus): check emitted labels instead of wrapper types, close the admission match The exclude-labels test now emits through the spend and provider budget metrics and checks the scrape keeps all their labels. The admission match arms end in assert_never so the match is exhaustive. * fix(prometheus): return the exhaustive-match fallback so every admission arm returns * fix(prometheus): pick the series tracker with isinstance so every path of _admits returns * fix(prometheus): skip an admissions line a worker could only write part of * fix(prometheus): frame each admissions record with newlines so a cut-off record cannot swallow the next A record a worker could only write part of used to merge with the next worker's record, and both were skipped for one request. Each record is now written between two newlines, so the fragment is a line of its own. The clock fixture in the series tests starts from a constant instead of reading the real clock * fix(prometheus): ignore a non-positive series cap or TTL with a warning instead of failing the logger A cap or TTL of 0 or less raised at logger init. The proxy logs that as a non-blocking error and keeps serving, so the result was a running proxy with no Prometheus metrics at all. The setting is now ignored with a startup warning naming it, the same rule the end_user cap already follows for a non-positive value * fix(prometheus): start the series cap over on a one-worker restart and audit it live A proxy with one worker and an operator-set PROMETHEUS_MULTIPROC_DIR now drops litellm's admission files at boot, so a restart frees every slot there the way it already does with several workers. A cap or TTL that is not a number greater than 0 (a bool, a non-numeric string, an empty value) is ignored with the startup warning instead of breaking the logger The integration cells drive the cap on every endpoint through the OpenAI and Anthropic SDKs and raw httpx, streaming and not, plus gauges, cache hits, failures, both workers of one instance, the TTL on one worker and its warning on two, ignored settings, excluded labels on the fallback counters, a null cap, /config/update, a concurrent burst scraped mid-flight, a provider outage, a killed worker, and restarts with one and two workers * fix(prometheus): wipe an operator-set multiprocess directory on a one-worker boot too * fix(prometheus): leave the multiprocess directory alone on a setup-only run A run with --skip_server_startup starts no worker, so it no longer creates or wipes PROMETHEUS_MULTIPROC_DIR. Wiping there deleted the samples of a proxy already running against the same directory * fix(prometheus): free the capped series slots when a gateway or backend container restarts The component image entrypoint starts uvicorn without the proxy CLI and wiped only the .db sample files at container start, so the admitted-series files of the previous container survived an in-place restart. Every label set seen after the restart was then counted on `other` once the previous container had filled the cap * test(prometheus): cover a setup-only run and a gateway image restart under the cap Two integration cells from the audit: a `--skip_server_startup` run pointed at a live two-worker proxy's operator directory leaves its samples alone, and the gateway image (`docker/component_entrypoint.sh` running `python -m gateway.launch`) restarted on a kept PROMETHEUS_MULTIPROC_DIR starts the cap over. The burst cells now wait for every counter they assert on, since the request and failure counters of one call increment at different points of the logging callback * test(prometheus): prove the cap reaches the fallback counters in the X1 cell * fix(prometheus): ignore a cleanup interval that is not a number of at least 0 A string or negative prometheus_metrics_cleanup_interval_seconds reached the series tracker unvalidated, so the first labeled emit with a TTL on raised TypeError inside the callback and recorded no series. The interval is now validated the way the cap and the TTL are: an invalid value is ignored with a warning and the default 60 seconds applies. The I2 integration cell drives a string interval through a live proxy and reads the warning from its log * test(prometheus): give the restart cells the boot budget of their siblings C4 and C5 boot two proxies each and hit the file's 240 s budget on a loaded box; C3 and D1 already carry 420 s --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
moe-berri
|
b58e2d7175
|
fix(lens): bound result recovery and preserve partial results (#44692)
* fix(lens): restate response contract during model repair * fix(lens): separate instructions and recover rejected results * fix(lens): correct loop type annotations and checks * fix(lens): preserve access to prior findings after compaction * fix(lens): cap result retries and preserve partial completion |
||
|
moe-berri
|
2190003bb8
|
fix(lens): reconstruct native coding agent conversations (#44711)
* fix(lens): reconstruct native coding agent conversations * fix(lens): display timestamps used for conversation ordering * fix(lens): preserve coding trace identity and message provenance * fix(lens): complete capture checks after trace pagination * fix(lens): show loaded replies during trace pagination |
||
|
moe-berri
|
c29b42a32b
|
fix(lens): preserve span timestamps in investigation evidence (#44702)
* fix(lens): preserve span timestamps in investigation evidence * style(lens): format chronology regression test |
||
|
moe-berri
|
c561372a02
|
fix(lens): show investigation findings for agent traces (#44696)
* fix(lens): show investigation findings for agent traces Replace child tool-error counts in the trace table with distinct investigation findings. Keep unassessed traces separate from completed clean investigations and use the root status for failure filters and timeline counts * fix(lens): stabilize findings updates and repair UI checks * fix(lens): allow viewer findings reads and index trace lookups * test(lens): cover viewer findings reads with Postgres |
||
|
moe-berri
|
d7210f95e6
|
fix(lens): persist final coverage with source diagnostics (#44682) | ||
|
devin-ai-integration[bot]
|
d9b1d57f07
|
fix(proxy): let a listed team alias win over a same-named key alias in the customer model check (#44677)
* fix(proxy): let a listed team alias win over a same-named key alias in the customer model check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): check each requested name in a plain loop in can_customer_access_model Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): only let a listed team alias skip the customer check when its target is live Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): move the per-name customer alias check into a local function Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a1a42768c1
|
fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses (#43560)
* fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(spend-tracking): share the short-lived miss cache write Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover key alias recovery after spend log lookup failures across usage routes * test(spend): bound outage alias lookups per miss window instead of a fixed count * fix(spend-tracking): treat any spend-log lookup failure as a short-lived miss The Prisma client raises a plain AttributeError when the database drops the connection mid-query, so the PrismaError catch let it through and the whole usage call answered 500. Any failure now keeps the 30 second backoff only, and the integration proxy patches its test entitlement at import so uvicorn's spawned workers inherit it * test(integration): audit spend-log metadata recovery under timeouts and dropped connections Cover the daily activity routes, the usage AI chat, the Vantage and CloudZero dry runs and exports under a locked spend-log table and under a database connection dropped mid-lookup, on a two-worker proxy, with the recovery after the outage asserted through the proxy's own miss TTL. Add a dropped_connection_relay that closes only the connection whose bytes carry a trigger, so a cell can drop the one connection the recovery query runs on while the rest of the pool keeps serving. Rewrite the sweep and JWT cells for the merged main: the export route reads metadata by SQL join and never calls the recovery, the search routes answer key rows and find deleted keys by alias, and the daily-spend owner recovery names the user while the alias stays blank. The sweep cell now times out a second lookup under the same lock, which pins the keys blank on the merge base and recovers on this branch. * test(integration): match a dropped-connection trigger split across two reads The dropped-connection relay checked each TCP read on its own, so a SQL marker that straddled two reads never tripped it and the outage cells would run without the outage they meant to exercise. Carry the tail of the previous read into the next check, as the held-statement relay already does, and pin that with a unit test that splits the trigger across two writes. * test(integration): scan relay triggers through an in-process helper The dropped-connection relay now matches its SQL trigger through a TriggerScanner that carries the previous read's tail, and the unit test exercises that scanner directly instead of opening loopback sockets, which tests/unit forbids. The relay's end to end behavior stays covered by the integration cells --------- Co-authored-by: gabriele <gabriele@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
tin-berri
|
9062fd3931
|
feat(proxy): embed enterprise LiteAdmin MCP in LiteLLM images (#44610) | ||
|
moe-berri
|
b69d744993
|
feat(lens): analyze trace workspaces with confined Python and compaction (#44640)
* feat(lens): add per-trace review models to jobs and progress
* feat(lens): append worker reviews to the job, capped, and count every review
* feat(lens): report a review with reasoning for each screened trace
* chore(ui): regenerate api types for lens job reviews
* feat(lens): type job reviews and fill them in lens fixtures
* feat(lens): add live review playback model
* feat(lens): pick the analysis model and slow single-review pacing
* feat(lens): add sample reviews for previewing the live run
* feat(lens): add live run layout with queue, reading trace and conclusions
* feat(lens): show the live run on investigations and open it from run now
* feat(lens): stream large review backlogs at 150ms or less and list newest first
* fix(lens): show the live run only for real reviews and keep fixtures test-only
* refactor(lens): restyle the live run as the native progress panel
* fix(lens): retry contended investigation updates with jittered backoff
* feat(lens): add a reading ticker line and replay for finished runs
* feat(lens): collapse the live run to an ambient line with show work
* fix(ui): crop the cerebras logo viewBox to its mark so it reads at icon size
* feat(lens): format review span previews as readable messages
* feat(lens): derive strip status, honest issue counts and drawer focus from a job
* feat(lens): track active jobs before their first review
* feat(lens): add a live trace results drawer with readable spans
* feat(lens): put the live strip under the progress bar and drop the inline panel
* feat(lens): add an ambient live strip that opens the drawer
* fix(lens): wait out provider rate limits and retry model calls four times
* style(lens): format repository contention tests
* feat(lens): read review spans as a conversation timeline
Turns spans into the user's ask, tool calls with args and results, and the agent's reply, dropping system prompts. Also handles a preview cut that lands inside the Output header.
* fix(lens): list recorded agents in the run now dialog
The run now agent field used a native datalist, whose suggestions do not show inside the modal dialog, so the agent list looked empty even though /lens/agents returned names. Use the same Combobox as investigation setup.
* feat(lens): pace live playback so each trace stays readable
Every trace now stays up for at least 1.5s. A backlog is cleared by skipping to the newest few instead of flickering through them. Conclusions count traces per check and kind, and new helpers cover share bars, group filters and flashes.
* feat(lens): keep the live run ambient until View run is clicked
The drawer no longer opens on Run or when entering a running investigation. LiveRun takes reviews as a prop so it can move to a dedicated reviews endpoint.
* feat(lens): show the live run as a two-pane trace and conclusions view
Left pane: the trace being reviewed as a readable timeline, followed by Lens's reasoning and the verdict. Right pane: ranked conclusion groups with share bars, plus a trace list you can filter.
* fix(lens): run several investigations per worker and poll every two seconds
* feat(lens): add worker slot and poll interval settings
* feat(lens): add list summaries and an incremental review filter
* perf(lens): strip reviews and run attributes from the lens list and serve reviews separately
* test(lens): cover list summaries, review polling and review access
* feat(lens): explain why a queued investigation is waiting
Works out whether no worker is connected, the worker is busy (with its running investigations and an estimated start time), or it is just being picked up.
* feat(lens): show the queue reason and what the worker is doing in the live strip
The progress header and the strip replace "Queued for your worker" with the concrete reason. While waiting, the strip lists the busy worker's investigations; click one to open it.
* feat(lens): add a review page model carrying the total reviewed count
* fix(lens): page live reviews by index so out-of-order reviews are never skipped
* feat(lens): take an index cursor on the reviews endpoint
* test(lens): cover index cursors across out-of-order and rolled-over reviews
* chore(ui): regenerate api types for the lens reviews endpoint
* feat(lens): page job reviews by index cursor
Adds api.reviews for GET /lens/{id}/runs/{job}/reviews?after=N, with a demo implementation. appendPage adds pages in arrival order and keeps the latest 200. liveJob now keys off reviewed, since the list no longer carries reviews.
* feat(ui): add a lens reviews query that polls the index cursor while live
* fix(lens): feed the live run from the reviews endpoint and keep View run open
LiveRun now gets its reviews from useJobReviews instead of the list, which no longer carries them. View run stays clickable while a run is queued or running, and before the first trace the opened view says what the worker is doing.
* fix(lens): split live conclusions into issues and patterns
A check could show up twice with the same label, once as an issue and once as a pattern.
* fix(lens): group live conclusions by check with short labels
There is now one group per check_id: issue traces are the main count and pattern traces a secondary note, so there are no duplicate red and grey cards. A long instruction falls back to the humanized check id. Adds briefReasoning and traceRows for the simplified trace list, and drops helpers nothing uses.
* feat(lens): simplify View run to traces and conclusions
The left pane is the trace list. A soft highlighter carrying the provider and model slides to the trace being reviewed, and clicking a row shows just Lens's reasoning and verdicts. The right pane keeps one conclusion card per check.
* refactor(lens): drop client-side replay in favour of real in-flight rows
Removes the playback reducer and its pacing. liveRows lists the traces the worker is reading, from job.reading, followed by completed reviews newest first, keyed by execution_id so a trace keeps its row when it finishes.
* feat(lens): show what the worker is reading and make View run obvious
Each trace in flight gets a highlighted row with the model and a live timer, and becomes its completed row in place. Completed rows show the real review time. View run is an outline button next to the progress line, and clicking anywhere on the strip opens it too.
* feat(lens): sum up a finished live run with time taken
doneLine reads like "Reviewed 30 traces in 31s with", measured from when reading started.
* feat(lens): slide one model rectangle over the traces being read
A single rounded rectangle carrying the provider logo and model wraps the real in-flight rows from job.reading. It translates and resizes over 250ms as traces finish in place. Before job.reading arrives it sits on a top slot showing the honest progress line, and when the run completes it fades out over 400ms. Rows have a fixed height and stable execution_id keys, so polls don't cause jumps or flicker.
* feat(lens): add in-flight runs to jobs and worker progress
* feat(lens): store in-flight runs from progress and clear them when a job ends
* refactor(lens): route progress, cancel and results through shared job transitions
* feat(lens): report each run as in flight when its review starts
* feat(lens): send in-flight runs with worker progress
* test(lens): cover in-flight runs across progress, old workers and terminal states
* test(lens): cover in-flight reporting under original run ids
* chore(ui): regenerate api types for lens in-flight runs
* feat(lens): model live reading lanes from in-flight runs and reviews
* feat(lens): show a now reading stage that types each trace's reasoning
* feat(lens): put the now reading stage above the trace list in View run
* fix(lens): resolve the analysis provider logo from the model catalog
* fix(lens): give demo jobs an empty in-flight list
* style(lens): format endpoint tests
* refactor(lens): name the run now handler in investigations view
* refactor(lens): name now reading conditions
* refactor(lens): name inline objects in the live run
* style(lens): format live run files
* fix(lens): keep worker settings inside the standalone worker package
* refactor(lens): keep update retry settings next to the repository
* fix(lens): start review history over when a run is reclaimed
* chore(lens): drop the unused review fixture
* refactor(lens): remove dead live helpers and use generated in-flight types
* fix(lens): keep polling a finished run until its last reviews arrive
* perf(lens): tick fast only while reasoning is typing
* fix(lens): isolate retried reviews and finding identities
* fix(lens): space the model name in run summary
* feat(lens): integrate confined workspace analysis with live reviews
* fix(lens): synchronize confined Python process monitoring
* Update review.md
* fix(lens): allow mixed context capacities and correct review assertions
* fix(lens): retrieve evidence on demand and isolate failed reviews
* fix(lens): isolate incomplete evidence reads from peer reviews
* test(lens): await trace status filter option
* test(lens): wait for reclaimed review state to settle
* fix(lens): recover from incomplete cross-session evidence
---------
Co-authored-by: Ishaan Jaff <ishaan@berri.ai>
|
||
|
devin-ai-integration[bot]
|
ee92183319
|
fix(sso): let CLI and Claude Code gateway sign-in through on DISABLE_ADMIN_UI nodes (#44620)
* fix(sso): let CLI and Claude Code gateway sign-in through on DISABLE_ADMIN_UI nodes * test(proxy): cover CLI SSO sign-in on a UI-disabled node --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
a3e15774ad
|
feat(proxy): limit which models an end user can call (#43904)
* feat(proxy): add models column to the end user table Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): enforce the end user models allowlist in model access checks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): accept and return models on the customer endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover the customer models allowlist Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): resolve team aliases before the end user model check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ishaan-berri
|
a3a52466a5
|
feat(lens): restore compact navigation with live investigation review (#44472)
* feat(lens): add per-trace review models to jobs and progress
* feat(lens): append worker reviews to the job, capped, and count every review
* feat(lens): report a review with reasoning for each screened trace
* chore(ui): regenerate api types for lens job reviews
* feat(lens): type job reviews and fill them in lens fixtures
* feat(lens): add live review playback model
* feat(lens): pick the analysis model and slow single-review pacing
* feat(lens): add sample reviews for previewing the live run
* feat(lens): add live run layout with queue, reading trace and conclusions
* feat(lens): show the live run on investigations and open it from run now
* feat(lens): stream large review backlogs at 150ms or less and list newest first
* fix(lens): show the live run only for real reviews and keep fixtures test-only
* refactor(lens): restyle the live run as the native progress panel
* fix(lens): retry contended investigation updates with jittered backoff
* feat(lens): add a reading ticker line and replay for finished runs
* feat(lens): collapse the live run to an ambient line with show work
* fix(ui): crop the cerebras logo viewBox to its mark so it reads at icon size
* feat(lens): format review span previews as readable messages
* feat(lens): derive strip status, honest issue counts and drawer focus from a job
* feat(lens): track active jobs before their first review
* feat(lens): add a live trace results drawer with readable spans
* feat(lens): put the live strip under the progress bar and drop the inline panel
* feat(lens): add an ambient live strip that opens the drawer
* fix(lens): wait out provider rate limits and retry model calls four times
* style(lens): format repository contention tests
* feat(lens): read review spans as a conversation timeline
Turns spans into the user's ask, tool calls with args and results, and the agent's reply, dropping system prompts. Also handles a preview cut that lands inside the Output header.
* fix(lens): list recorded agents in the run now dialog
The run now agent field used a native datalist, whose suggestions do not show inside the modal dialog, so the agent list looked empty even though /lens/agents returned names. Use the same Combobox as investigation setup.
* feat(lens): pace live playback so each trace stays readable
Every trace now stays up for at least 1.5s. A backlog is cleared by skipping to the newest few instead of flickering through them. Conclusions count traces per check and kind, and new helpers cover share bars, group filters and flashes.
* feat(lens): keep the live run ambient until View run is clicked
The drawer no longer opens on Run or when entering a running investigation. LiveRun takes reviews as a prop so it can move to a dedicated reviews endpoint.
* feat(lens): show the live run as a two-pane trace and conclusions view
Left pane: the trace being reviewed as a readable timeline, followed by Lens's reasoning and the verdict. Right pane: ranked conclusion groups with share bars, plus a trace list you can filter.
* fix(lens): run several investigations per worker and poll every two seconds
* feat(lens): add worker slot and poll interval settings
* feat(lens): add list summaries and an incremental review filter
* perf(lens): strip reviews and run attributes from the lens list and serve reviews separately
* test(lens): cover list summaries, review polling and review access
* feat(lens): explain why a queued investigation is waiting
Works out whether no worker is connected, the worker is busy (with its running investigations and an estimated start time), or it is just being picked up.
* feat(lens): show the queue reason and what the worker is doing in the live strip
The progress header and the strip replace "Queued for your worker" with the concrete reason. While waiting, the strip lists the busy worker's investigations; click one to open it.
* feat(lens): add a review page model carrying the total reviewed count
* fix(lens): page live reviews by index so out-of-order reviews are never skipped
* feat(lens): take an index cursor on the reviews endpoint
* test(lens): cover index cursors across out-of-order and rolled-over reviews
* chore(ui): regenerate api types for the lens reviews endpoint
* feat(lens): page job reviews by index cursor
Adds api.reviews for GET /lens/{id}/runs/{job}/reviews?after=N, with a demo implementation. appendPage adds pages in arrival order and keeps the latest 200. liveJob now keys off reviewed, since the list no longer carries reviews.
* feat(ui): add a lens reviews query that polls the index cursor while live
* fix(lens): feed the live run from the reviews endpoint and keep View run open
LiveRun now gets its reviews from useJobReviews instead of the list, which no longer carries them. View run stays clickable while a run is queued or running, and before the first trace the opened view says what the worker is doing.
* fix(lens): split live conclusions into issues and patterns
A check could show up twice with the same label, once as an issue and once as a pattern.
* fix(lens): group live conclusions by check with short labels
There is now one group per check_id: issue traces are the main count and pattern traces a secondary note, so there are no duplicate red and grey cards. A long instruction falls back to the humanized check id. Adds briefReasoning and traceRows for the simplified trace list, and drops helpers nothing uses.
* feat(lens): simplify View run to traces and conclusions
The left pane is the trace list. A soft highlighter carrying the provider and model slides to the trace being reviewed, and clicking a row shows just Lens's reasoning and verdicts. The right pane keeps one conclusion card per check.
* refactor(lens): drop client-side replay in favour of real in-flight rows
Removes the playback reducer and its pacing. liveRows lists the traces the worker is reading, from job.reading, followed by completed reviews newest first, keyed by execution_id so a trace keeps its row when it finishes.
* feat(lens): show what the worker is reading and make View run obvious
Each trace in flight gets a highlighted row with the model and a live timer, and becomes its completed row in place. Completed rows show the real review time. View run is an outline button next to the progress line, and clicking anywhere on the strip opens it too.
* feat(lens): sum up a finished live run with time taken
doneLine reads like "Reviewed 30 traces in 31s with", measured from when reading started.
* feat(lens): slide one model rectangle over the traces being read
A single rounded rectangle carrying the provider logo and model wraps the real in-flight rows from job.reading. It translates and resizes over 250ms as traces finish in place. Before job.reading arrives it sits on a top slot showing the honest progress line, and when the run completes it fades out over 400ms. Rows have a fixed height and stable execution_id keys, so polls don't cause jumps or flicker.
* feat(lens): add in-flight runs to jobs and worker progress
* feat(lens): store in-flight runs from progress and clear them when a job ends
* refactor(lens): route progress, cancel and results through shared job transitions
* feat(lens): report each run as in flight when its review starts
* feat(lens): send in-flight runs with worker progress
* test(lens): cover in-flight runs across progress, old workers and terminal states
* test(lens): cover in-flight reporting under original run ids
* chore(ui): regenerate api types for lens in-flight runs
* feat(lens): model live reading lanes from in-flight runs and reviews
* feat(lens): show a now reading stage that types each trace's reasoning
* feat(lens): put the now reading stage above the trace list in View run
* fix(lens): resolve the analysis provider logo from the model catalog
* fix(lens): give demo jobs an empty in-flight list
* style(lens): format endpoint tests
* refactor(lens): name the run now handler in investigations view
* refactor(lens): name now reading conditions
* refactor(lens): name inline objects in the live run
* style(lens): format live run files
* fix(lens): keep worker settings inside the standalone worker package
* refactor(lens): keep update retry settings next to the repository
* fix(lens): start review history over when a run is reclaimed
* chore(lens): drop the unused review fixture
* refactor(lens): remove dead live helpers and use generated in-flight types
* fix(lens): keep polling a finished run until its last reviews arrive
* perf(lens): tick fast only while reasoning is typing
* fix(lens): isolate retried reviews and finding identities
* fix(lens): space the model name in run summary
* Update review.md
* test(lens): await trace status filter option
---------
Co-authored-by: moe-berri <moe@berri.ai>
|
||
|
devin-ai-integration[bot]
|
45e7be1abc
|
feat(tracing)!: return only data from SQL queries (#44609)
* feat(tracing): add generated SQL response contract Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): return data-only trace SQL responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(rust): format trace response schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
9247826cdd
|
refactor(proxy): rename management/teams/access.py to authz.py (#44624)
access already means access groups in this codebase; authz names what the module decides (who may act on a team). Pure rename: every importer now uses authz, and access.py stays as a re-export because the published litellm-enterprise 0.1.73 wheel imports is_team_admin from it. |
||
|
moyai-devin-berriai[bot]
|
75b45e39c9
|
fix(proxy): enforce internal-user model creation prohibition (#44438)
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
d80f8c28ca
|
refactor(types): replace Any with proven types in 137 files (#44478)
* refactor(types): prove runtime types at harness, search, rag and client boundaries Replace Any with adapter-validated types in the litellm.agent() harness, the search provider transformations, RAG ingestion and query, the vector store pre-call hook and registry, the galileo and opik logging integrations and the proxy client CLI. Each boundary gets unit tests for well-formed and malformed payloads. * chore(typing): prove types at more provider boundaries and restore search transformations Second pass over a2a, embedding, rerank, image, audio and small provider modules. Search transformations go back to their previous form because validating their response bodies would change the proxy status for malformed upstream bodies from 400 to 500. * refactor(types): prove types at logging, files, rerank, image, audio and management boundaries Replace Any with validated or annotation-only types in 37 more files: logging integrations, token counters, provider files/rerank/image generation/audio transcription transformations, pass-through logging handlers and management endpoints. No proxy HTTP status or error type changes. * refactor(types): prove types at repository, spend, files and router boundaries Replace Any with repository table accessors, validated mappings and annotation-only types in 33 more files: Prisma repositories, the enterprise batch and responses cost checkers, budget reservation, files endpoints, management endpoints, the policy registry, the adaptive and complexity routers and the secret managers. No proxy HTTP status or error type changes. * test(types): run the aiohttp transformation test in-process and cover repository row conversion The aiohttp chat transformation test no longer starts a server. It feeds the transformation a response whose json() returns the body under test. The proxy unit shards now exercise stored model rows whose params are JSON strings and the object permission create and update paths. |
||
|
devin-ai-integration[bot]
|
9b6a6a0b71
|
refactor(tracing): generate existing HTTP request models from Rust schemas (#44591)
* test(tracing): pin HTTP request compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(tracing): generate existing HTTP request models from Rust schemas Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(tracing): bind trace query params to generated request models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci(rust): raise the native wheel size gate to 48 MB Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(tracing): read the trace list clock without a thread-pool dispatch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(traces): share the trace page-size bounds between schema and reader Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): type trace request queries against the generated OpenAPI schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: encode the trace contract boundary in AGENTS.md Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format trace request aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Ninad Phalak
|
d946706744
|
feat(guardrails): add llm shield pii redaction and rehydration guardrail (#42645)
* feat(guardrails): add llm shield pii redaction and rehydration guardrail
LLM Shield is a self-hosted PII gateway. This adds it as a guardrail so a
proxy operator can redact personal data out of outbound requests and have
the original values restored in the model's reply.
The substitution is reversible, which is the difference from a masking
guardrail. Outbound text is replaced with placeholders held in a session
vault inside the operator's own LLM Shield deployment, and the reply is
restored before it reaches the caller, so the end user still sees real
values while the provider never received them.
Streaming responses are restored incrementally. LLM Shield holds back only
the trailing characters that could still turn out to be part of a
placeholder, so tokens are forwarded as they arrive rather than the whole
response being collected first. A placeholder split across two chunks is
never emitted in fragments.
The integration talks to LLM Shield over HTTP and adds no dependency.
Notes for reviewers:
- The guardrail sets use_native_lifecycle_hooks, since redaction and
restoration need the native pre-call, post-call and streaming hooks
rather than the unified path.
- Per-request state lives on the request dict, never on the guardrail
instance, because the proxy registers a single instance process-wide.
The streaming carry-over is a local of the generator for the same reason.
- Every failure blocks the request. A redaction guardrail that fails open
would send the exact data it exists to protect to the provider.
* feat(ui): list llm shield in the guardrail garden
Adds the card, preset and logo so operators can pick LLM Shield from the
guardrails page the same way as the other partner guardrails.
* docs(guardrails): add llm shield example config
Shows both modes on one entry. Listing only pre_call redacts the request
and then hands the placeholders back to the end user, so the test asserts
both hooks are enabled.
* feat(ui): use the llm shield brand mark for the guardrail logo
* fix(guardrails): restore llm shield values in anthropic replies
The /v1/messages reply is a plain dict with a content block list and no
choices, so it fell through the restore path and went back to the caller
still carrying placeholders. The request was redacted correctly, which is
what made this easy to miss.
Found by running all three endpoints against a live provider; the mocked
tests all passed because they only built the OpenAI shape. Adds tests for
the message shape and for leaving non-text blocks alone.
* docs(guardrails): correct the llm shield start command
* fix(guardrails): redact every request shape and restore every reply shape
Three gaps, all of which let an enabled guardrail hand data to the provider
or hand placeholders to the caller.
Requests only walked `messages`. The Responses API `input` and tool call
`arguments` went out untouched. Measured against a live provider: a request
sent through `/v1/responses` reached the model with the real address in it
while the guardrail reported as enabled. Request traversal now covers chat
content (string and multimodal), tool call arguments, and `input` as a bare
string or a list of items.
Fixing that exposed the matching gap on the way back: the Responses API reply
carries `output` items rather than `choices`, so it returned to the caller
still holding placeholders. It now gets its own walk, handling text blocks as
dicts or objects.
The dashboard preset seeded only pre_call, so a guardrail created from the UI
would redact the request and return the placeholders to the user. Presets can
now seed both modes; the form already normalised either shape.
Adds tests for each request shape, for both Responses API reply forms, and
replaces a test that had asserted the `input` bypass as correct behaviour.
* fix(guardrails): narrow the stream delta before writing to it
basedpyright could not prove the delta was non-None on the write path, and
reportOptionalMemberAccess has a zero budget. The guard is also clearer than
relying on the text check to imply it.
* fix(guardrails): mint the vault id instead of trusting the caller's
The vault id was taken from caller-supplied session metadata, and every
caller shares one LLM Shield key. Someone who knew or guessed another
caller's session id could send a placeholder, have the model echo it back,
and get that caller's plaintext restored into their own reply.
Vault ids are now minted per request behind a per-process prefix, so a
caller cannot name a vault this process uses. Redaction mints, restoration
reads back, and a reply whose id does not match is left holding its
placeholders rather than resolved against some other vault.
Also covers two more request fields that were reaching the provider intact:
the Responses API `instructions`, and the legacy `function_call.arguments`
alongside `tool_calls`.
The collectors move to module level, which drops the traversal back under
the complexity limit and lets the code carry its own explanation instead of
the comments that were restating it.
* fix(guardrails): drop Final from a loop-assigned local
basedpyright rejects a Final assigned inside a loop, and
reportGeneralTypeIssues sits one over its budget ceiling.
* fix(guardrails): redact completion prompts and responses tool items
Two more provider-bound request shapes were reaching the model intact while
the guardrail reported as enabled.
/v1/completions carries its text in a top-level `prompt`, which the
traversal never looked at. It is handled as a string and as the array form,
where each entry is rewritten in place.
Responses input items hold tool data outside `content`: a function_call item
in `arguments`, a function_call_output item in `output`. Both are now
collected alongside the item's content.
Adds a test per shape.
* fix(guardrails): redact the anthropic system prompt and string-array input
Two more provider-bound shapes, found by walking the request types rather
than waiting for them to be reported.
/v1/messages carries its system prompt at the top level, as a string or a
list of text blocks. It is one of the endpoints this guardrail claims to
cover, and a system prompt is a natural place to put a customer's details.
`input` as an array of bare strings, the embeddings and moderations shape,
was skipped because the loop only handled item dicts.
Verified against a live provider: a system prompt holding an address now
reaches the model as a stand-in and is restored in the reply.
* fix(guardrails): narrow prompt and input to a list before iterating
Guarding with a conditional iterable left the value un-narrowed, so passing
it on was an argument-type error and the element checks read as unreachable.
An early return narrows it properly and reads better.
* fix(guardrails): restore every streaming choice, not just the first
Streaming rehydration read and rewrote choices[0] only, so with n>1 every
later choice went back to the caller still holding its placeholders.
Each choice is its own token stream, so the sliding window is now tracked
per choice index rather than once per stream. A single shared window would
have been worse than the bug: it would splice the characters held back for
one choice onto the next one's delta.
The final flush walks every choice the same way, and the two helpers that
only ever looked at choices[0] are gone.
Adds a test that both choices come back restored, and one that each choice
gets its own window handed back rather than its neighbour's.
* refactor(guardrails): name the guardrail llm_shield_proxy throughout
The integration was called llm_shield in code, llm-shield in the example
config, and LLM Shield in the dashboard, while the product and its PyPI
package are both llm-shield-proxy. An operator who saw the guardrail in
LiteLLM could not tell what to install.
One identifier now: llm_shield_proxy for the enum value, module, directory,
class, config model, logo and environment variables, with LLM Shield Proxy
as the display name. That matches `pip install llm-shield-proxy`.
Renames only; no behaviour change.
* feat(guardrails): redact the participant name on a message
`name` on a user or assistant turn identifies a person and was going to the
provider intact. The proxy this integrates with already redacts it, so the
integration was the weaker of the two.
On a tool or function turn the same field carries the function's name, which
has to arrive unchanged or the call stops routing. That case is skipped, and
a test asserts the value is never even sent to the shield.
* fix(guardrails): flush every held choice, and cover tool results and suffix
Three review findings.
The trailing flush walked the last chunk's choices, so a choice that finished
earlier and stopped appearing lost whatever text was still held for it and its
answer was truncated. It is now driven by the windows themselves and emits one
chunk per choice, synthesising the choice when the terminal chunk omits it.
That was data loss, not just under-redaction.
An Anthropic tool_result carries its own content, as a string or as further
blocks, and only each part's `text` was being collected. Handled recursively;
image and audio parts still fall through untouched.
The legacy completions `suffix` is forwarded to providers that support it and
was never collected. Note the placement: it has to be gathered before the
string-prompt early return, which is what the new test pins.
* fix(guardrails): walk nested tool results iteratively, with a depth bound
CI flagged _collect_content as recursive. It was, and worse, it was unbounded:
a tool_result nests its own content, the nesting is caller controlled, and the
descent had nothing to stop it. That is a JSON bomb, not a style issue.
Now an explicit queue with a depth bound of 8. Real payloads nest one or two
deep. The queue is walked in document order because the shield maps its replies
back by position, so collection order is part of the contract.
* fix(guardrails): redact Responses PromptObject variables
A Responses request can send `prompt` as a PromptObject rather than a string.
Its `variables` are substituted into the stored prompt on the provider side, so
they are caller text, and the dict shape was falling through untouched.
`id` and `version` pick which stored prompt to run and are left unchanged.
* test(guardrails): assert the depth bound instead of only reaching the end
The depth test asserted nothing, so it passed whether or not the bound held,
and the test-quality gate counted it as a zero-assert test. It now sends a
shallow value alongside a 200-deep chain and asserts the shallow one is
collected while the value past the bound is not.
* fix(guardrails): keep system-prompt values out of the restored reply
Redaction put every span of a request into one vault, and the reply was restored
against that same vault. System prompts are written by the application and the
caller never sees them, so a caller who got the model to echo a placeholder back
had its plaintext restored into their own reply -- a way to read a system prompt
they were never shown.
Server-authored spans now go into a vault of their own: system and developer
turns, Anthropic's top-level `system`, and the Responses API `instructions`.
Its id is deliberately never stored, so nothing restores against it. The reply
is restored against the caller's vault alone, and an echoed placeholder from a
system prompt comes back as the placeholder.
Values the caller also wrote themselves are unaffected -- they are in the
caller's vault too, and still restore. The extra round trip happens only when a
request actually carries server-authored text.
* style(guardrails): satisfy ruff format and annotate the new tests
`ruff format` wanted the widened `_collect_responses_fields` signature on one
line, and the three tests added with the split-vault fix needed return
annotations to keep ANN201 level with the base.
* fix(guardrails): restore tool calls in the LLM Shield guardrail
The request walk redacted a tool call's `arguments` -- plus the legacy `function_call`,
Anthropic `tool_use.input` leaves and the Responses API's `function_call` /
`function_call_output` fields -- while the response walk restored only `message.content`.
A placeholder therefore reached the caller inside a tool call, and nothing raised.
This is the same change as the out-of-tree example adapter this file is copied from, kept
body-identical on purpose: the response side now collects every restorable span in one
positional rehydrate batch, streaming keeps a window per (choice index, tool-call index)
and flushes each into the chunk carrying the finish_reason, and `apply_guardrail` restores
`inputs["tool_calls"]` on the response side. The declared limit on restoring values inside
a JSON string is documented in the module.
* fix(guardrails): import copy, keep the vault id off the provider, drop recursion
Three defects Greptile and veria-ai found on the reopened PR, all real:
- `copy.deepcopy` was called in `apply_guardrail` with no `import copy`, a
guaranteed NameError on every response carrying tool calls. It landed on
2026-09-13, ten days after the review that rated this branch safe, and no test
reached it: every tool-call test covered the request side. Adds the import and
a regression test on the response side.
- The vault session id was stored in `metadata`, which is forwarded to the
provider on /v1/responses. A provider holding the placeholders and the session
id can call the shield's rehydrate endpoint and read back the plaintext this
guardrail exists to withhold. Moves it to `litellm_metadata`, which is not
forwarded, and reads it back from there only.
- `_collect_json_leaves` recursed over model-controlled JSON; the repo's
recursive_detector gate rejects that. Rewritten with an explicit stack, same
depth bound.
52 tests pass. ruff format, ruff-strict and check_type_discipline all clean, with
LIT counts identical to the merge base.
* fix(guardrails): build llm_shield_proxy stream deltas without new mutable literals
The lint job's LIT002 budget gate failed on this PR: the file added 11
mutable-collection constructions and the tree sits at its limit. Build the
index-only tool-call continuation in one helper, keep read-only inputs as
tuples, and annotate the lists the delta and texts fields require.
Adds tests for the two tool-call flush paths the refactor touches, which
had no coverage: held arguments landing in the finish_reason chunk next to
that chunk's own fragment, and the trailing flush of a stream that ends
without a finish_reason.
* fix(guardrails): drop Final from loop-body locals in llm_shield_proxy
basedpyright rejects Final on a name assigned inside a loop, and the eleven
such locals put reportGeneralTypeIssues over its budget (112/101). The LIT010
Final rule already exempts loop-body assignments, so the annotations go.
* feat(guardrails): restore llm_shield_proxy placeholders on native streams
Anthropic /v1/messages and /v1/responses streams have no `choices`, so the
streaming hook passed them through with placeholders still in them. Both
are now restored incrementally, with the same per-stream windows as chat:
- /v1/messages arrives as raw SSE. Frames are cut at event boundaries,
text_delta and input_json_delta are restored per block index, and held
text is emitted as one more delta ahead of content_block_stop. Signed
thinking deltas, frames from other endpoints and non-SSE raw streams
pass through unchanged.
- /v1/responses events are restored per item and part. Held text goes out
as a copy of the stream's last delta before its .done event, and the
events that repeat the reply (.done, content_part.done, output_item.done,
response.completed) are restored in full.
The request side now also redacts Anthropic tool_use inputs and Responses
reasoning summaries, and sends tool and function descriptions (including
parameter schema descriptions) and the user / safety_identifier fields to
the non-restorable vault, like system prompts. Tool results stay
restorable: the model reads them to answer, so restoring them returns what
the caller would have seen without the guardrail.
* fix(guardrails): redact llm_shield_proxy predicted outputs and output schemas
`prediction.content` is the caller's own draft of the reply, so it is
redacted into the caller vault and restored with the reply. The
descriptions in a structured-output schema (Chat
response_format.json_schema, Responses text.format) are application
authored like tool schemas, so they go to the non-restorable vault.
* fix(guardrails): fail closed on deep llm_shield_proxy requests, widen coverage
- Request walks no longer skip what lies past their depth bound. Content
nested past it, and tool inputs or schemas past the new JSON bound, now
block the request instead of reaching the provider unredacted. The old
depth test asserted the skip; it now asserts the block.
- Tool and output schemas are walked by their JSON Schema structure, and
give up `title`, `examples` and `default` as well as `description`.
`enum` and `const` still go out as sent.
- Responses events are matched by shape: any `*.delta` with a string delta
is a token stream, and any `*.done` restores every non-identifier text
field plus the `part` or `item` it repeats. This covers
reasoning_summary_part.done and MCP arguments, and future families.
Audio deltas are left alone.
- An SSE stream whose first chunk ends partway through a field name
(`b"eve"`) is no longer taken for a non-SSE stream.
* fix(guardrails): scan llm_shield_proxy schemas by default
The schema walk collected an allowlist of keywords, so any keyword it did
not list -- draft-07 `dependencies`, `$comment`, vendor `x-` extensions --
went to the provider in clear. Invert it: every string is collected except
under keywords whose value must go out verbatim (types, formats, patterns,
references, required lists, enum, const). Name -> subschema maps still
treat their keys as property names, so a property called `type` is
walked, not skipped.
* fix(guardrails): redact llm_shield_proxy schema enum and const values
`enum` and `const` were skipped by the schema walk, so a value holding PII
went to the provider in clear. They now go to the caller's vault rather
than the non-restorable one: the model emits the stand-in in its tool
arguments or structured output, and restoring the reply turns it back into
the value the schema allows, so the call still routes.
* fix(guardrails): redact llm_shield_proxy web search user locations
Web search forwards the user's approximate location, and its free-text
`city` and `region` fields can hold an address. Collect them into the
non-restorable vault, from Chat `web_search_options.user_location` and
from the `user_location` of Responses and Anthropic web-search tools.
* fix(guardrails): drop unused llm_shield_proxy suppressions
Upstream added LIT013 (a *-ok marker that suppresses nothing) and LIT014
(at most one for and one if per comprehension). Remove the 34 markers
that no longer suppress anything and flatten the finished streams with
itertools.chain.from_iterable.
* fix(guardrails): type the llm_shield_proxy request and reply walks
Narrowing with isinstance(x, dict) leaves keys and values unknown, so
every call that passed a narrowed value counted against the
reportUnknownArgumentType budget. Parse into dict[str, object] and
list[object] once, in _as_object and _as_array, type the carry keys and
accumulators, and bind writers with functools.partial instead of lambdas.
The shield's batch reply is now also checked to hold only strings.
* fix(guardrails): keep restored llm_shield_proxy replies out of the cache, widen coverage
Addresses the open veria-ai and Cursor Bugbot findings on #42645.
- Restore a copy of the reply and of each stream chunk, never LiteLLM's own object.
LiteLLM caches and logs that object, and placeholders are numbered per request, so
two callers' redacted requests can share a cache key: restoring in place cached one
caller's plaintext for the next. The deployment hook no longer restores either,
since LiteLLM caches what it returns; the proxy's post-call hook restores
model-level guardrails after the cache write.
- Restore /v1/completions replies, streamed and not, which carry `choice.text`.
- Redact Responses replay fields the reply side already restores: tool output sent
as input_text parts, custom_tool_call `input`, code_interpreter_call `code`.
- Redact typed Responses prompt variables (`{"type": "input_text", "text": ...}`).
- Put Responses system and developer input items in the non-restorable vault, like
their Chat counterparts.
- Expose LLMShieldProxyGuardrailConfigModel through get_config_model, so the
dashboard can collect the Shield URL and key.
* fix(guardrails): redact llm_shield_proxy plain-text document blocks
An Anthropic document block carries text inline, in a text source's `data` or a
content source's `content`, and that text reached the provider unredacted. Collect
both, plus the block's `title` and `context`; base64, URL and file sources pass
untouched.
* fix(guardrails): redact llm_shield_proxy extra_body overrides
LiteLLM merges extra_body over the transformed request just before sending, so text
placed there (input, messages, system, ...) replaced the redacted field on the wire.
Walk extra_body with the same collectors as the request, keeping the caller /
application split.
* test(guardrails): import InMemoryCache directly in the llm_shield_proxy cache test
litellm keeps a deprecated module-level `caching` bool, so `litellm.caching.caching`
resolves to that bool once an earlier test in the same worker has set it, and the test
failed with AttributeError depending on test order.
* fix(guardrails): restore llm_shield_proxy replies for model-level use outside the proxy
|
||
|
devin-ai-integration[bot]
|
2d82915084
|
fix(mcp): bind OAuth clients to their upstream issuer (#37777)
* feat(mcp): advertise the SDK's latest spec revision and validate the RFC 9207 iss MCPSpecVersion stopped at 2025-06-18 while the pinned SDK negotiates 2025-11-25, and the version LiteLLM puts on its own outbound initialize was a hardcoded historical member. Add the missing revision, name the highest revision we speak once, and pin it to the SDK's LATEST_PROTOCOL_VERSION with a test so the two cannot drift apart silently. /authorize now seals the issuer it sent the user to into the OAuth state, and /callback holds the authorization response's RFC 9207 iss against it, refusing to forward a code that came back from an authorization server we never sent the user to. An absent iss, an unanchored server row and a state minted before the seal all keep their current behavior. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep params, query and fragment significant in issuer comparison The shared canonicalizer drops all three, so two issuers differing only outside the path compared equal and a response from another tenant's authorization server would have continued through the flow. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): refresh generated API snapshots Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): cover OAuth client isolation and lint checks * fix(mcp): preserve registered clients in the existing save payload * test(mcp): cover optional OAuth registration metadata * fix(mcp): preserve compatible OAuth registrations across edits * fix(mcp): retain OAuth state through pending authorization * fix(mcp): guard pending OAuth at form submission * fix(mcp): discard canceled OAuth edit snapshots * test(mcp): preserve complete OAuth registration assertions * refactor(mcp): construct OAuth credential updates without mutation * fix(mcp): simplify issuer binding and reject unverifiable callbacks * fix(mcp): preserve replacement clients and pending redirect bindings * fix(mcp): retain clients with replacement authentication methods * fix(mcp): preserve cached clients and pin manual OAuth issuers --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
moyai-devin-berriai[bot]
|
2e81db03b9
|
fix(auto-router): align preview and serving default resolution (#44436)
Co-authored-by: moyai-devin-berriai[bot] <336287033+moyai-devin-berriai[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3286782dea
|
fix(caching): key response cache by router model group in litellm_metadata (#44542)
* test: align integration fixtures with current behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the spend flush before asserting its trace placement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): key response-cache entries by model group from litellm_metadata /v1/responses routes through the router with model_group in litellm_metadata, which the cache key ignored, so identical requests to different model groups sharing one underlying model hit each other's cached responses Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
moe-berri
|
eeb192d4a7
|
chore!: retire the integrated ROI calculator (#44477)
* chore!: retire the integrated ROI calculator * chore(ui): remove unused ROI demo notice |
||
|
devin-ai-integration[bot]
|
254c2f6b3b
|
refactor(mcp): drop Sequence/list return-type mismatch and collapse record_listed_tools wrapper (#44556)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
7a7d27c550
|
fix(guardrails): run the end-of-stream post_call scan when the client disconnects mid-stream (#43839)
* fix(guardrails): run end-of-stream post_call scan when the client disconnects mid-stream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): close the guardrail stream chain in async_data_generator on client disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): leave the raw upstream response to the shielded finalizer on client disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep disconnect cleanup going when a streaming callback cleanup raises Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert the refund through a recorder instead of the mock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): inspect tool calls released before a disconnect under incremental_diff and record a failed scan marker The incremental_diff transform stream now scans tool calls it already released when the client disconnects, and a disconnect scan whose translation raises after the guardrail recorded success also records guardrail_failed_to_respond Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): pin that a text-only disconnect scan is not handed a tool_calls finish Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): cover disconnect scans on every streaming endpoint and client, plus outage, worker-kill and cache-hit cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): prove the cache-hit twin is served from the cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): type the disconnect-close streams so basedpyright stops reporting unknown arguments * fix(guardrails): give the guardrail metadata cast a reason so the type discipline gate accepts it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): scan released Messages and Responses tool calls on disconnect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(guardrails): format the disconnect scan unit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(guardrails): drop mutable-ok markers that no longer suppress a rule Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): scan released Responses output after a finished item and end only in-flight Chat choices Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): type the disconnect scan test helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(guardrails): type request_data in the disconnect scan helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): type request_data in the disconnect scan test doubles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): pin that chat streams with no tool call in flight end as released Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): scan only released chunks on disconnect and skip it once a block owns the verdict The disconnect scan now uses the chunks actually yielded to the client, copies them before scanning, skips when a mid-stream block or HTTP error already settled the verdict, and the iterator wrapper only closes hooks that are async generators so plain async iterator hooks keep working Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): pin that a delivered guardrail error or final chunk settles the disconnect verdict Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): close any hook iterator that exposes aclose when the stream ends early Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): accept a synchronous aclose on custom streaming hook iterators Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): swallow callback aclose errors at end of stream A custom callback whose async_post_call_streaming_iterator_hook returns a non-generator async iterator with a raising aclose() failed the finished stream: content plus usage reached the client and then the stream surfaced an error SSE with no [DONE], or aborted a post_call pipeline's buffering loop into a 500 with an empty body. Wrap the aclose invocation in _wrap_streaming_iterator_with_enrichment in try/except and log a warning naming the callback and the cleanup error, matching close_guarded_stream and _close_guarded_layers. Iteration-time hook exceptions still propagate. * fix(proxy): log only the error type when a callback aclose raises Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |