mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
407 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Mateo Wang
|
8fc31b1e4e
|
feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions (#45184)
* feat(decisions): serve System One format at /v1/systemone and OpenAI format at /v1/decisions The System One request format moves to /v1/systemone and /systemone. /v1/decisions and /decisions now accept the OpenAI Decisions API format, translate it into a System One request, route it through the same pipeline, and translate the answers back. * fix(decisions): import assert_never from typing_extensions for Python 3.10 * fix(decisions): cap OpenAI-format questions at the System One limit /v1/decisions accepted up to 200 questions, but the System One request it translates to takes at most 128, so 129 to 200 questions failed with an internal validation error. Both now share MAX_DECISION_QUESTIONS. * fix(decisions): return 400 for bodies that are not JSON * test(decisions): send System One bodies to /v1/systemone in integration tests * fix(decisions): return guardrail_information on OpenAI-format decisions when requested * refactor(decisions): validate proxy request data before reading guardrail settings |
||
|
devin-ai-integration[bot]
|
0e09e99146
|
refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler (#45130)
* refactor(decisions): dispatch /v1/decisions through provider configs and the shared HTTP handler Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): map OpenRouter connection failures to APIConnectionError and drop explanatory docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(decisions): keep APIConnectionError for unreachable deployments on every provider `_map_upstream_exception` now maps a status `_handle_error` synthesized for a non-HTTP failure to `APIConnectionError` before `exception_type` runs, so a Perplexity (openai-compatible) deployment that cannot be reached no longer answers `InternalServerError` where main answers `APIConnectionError`. Audit cells: the unreachable-deployment integration cell is parametrized over the five providers, OpenRouter connection failures are pinned on chat, stream, embeddings, decisions and the SDK, a deployment whose provider has no decisions config gets the 400 naming every supported provider, and a unit cell pins the mapper alone. The first docstring sentence of the five translation bases is dropped. --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
4f49f88172
|
fix(proxy): keep idle responses websockets open until a configurable session limit (#44433)
* add config-driven bound to /responses websocket connection * test(proxy): cover responses websocket session limit and idle first frame Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ci): sync API schema and Ruff formatting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): close responses websocket client before upstream cleanup at session limit Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): assert recorded websocket outcomes and simplify session reaping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): move responses websocket session limit timing coverage to integration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): run provider-close checks off the event loop Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover handshake auth, subprotocol, DB override and worker kill for the responses websocket session limit Adds the audit cells for the configurable session limit: a rejected bearer fails the handshake with 403, a requested Sec-WebSocket-Protocol is echoed back, a DB override through /config/field/update caps new sessions on every worker without a restart while sessions accepted before it keep their limit, deleting the override restores the default, out-of-range and wrong-type updates answer 400 and leave the stored row alone, and SIGKILLing the worker holding idle sockets drops only its sockets while the proxy keeps serving * test(integration): open override sockets until both proxy workers hold one uvicorn workers share one listening socket and a worker that wakes first accepts a whole simultaneous burst, so eight sockets opened at once can all land on one worker. The DB override cell now opens its sockets one at a time, alternating the two paths, until at least eight are open and every worker holds one, bounded at forty --------- Co-authored-by: Mrinal Chanshetty <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
yucheng-berri
|
efe0715831
|
fix(proxy): accept router-wide default_litellm_params in required-param validation (#44483)
* fix(proxy): accept router-wide default_litellm_params in required-param validation The required-body-param check added in #43787 only consulted request data and deployment-level litellm_params, so a param supplied solely by router_settings.default_litellm_params (e.g. max_tokens for /v1/messages, documents for /rerank) was rejected with a 400 at route entry even though the router would have injected it during dispatch. Consult the router-wide defaults in the same place, treating None-valued defaults as absent to mirror the setdefault merge. * fix(proxy): consult only the defaults the dispatching router will merge Review follow-up: router-wide defaults are now consulted only where dispatch actually applies them. user_config requests dispatch on their own throwaway Router, so they consult that config's defaults instead of the global router's; search, managed-agent and eval routes, and model-less direct dispatch never pass through the router's defaults merge, so they keep the route-entry 400. Also adds rerank router-default coverage (unit + e2e) and drops the redundant source comment. |
||
|
devin-ai-integration[bot]
|
02b8b5cc80
|
fix(proxy): traceparent/baggage fallback must not override caller metadata (#43688)
* fix(proxy): traceparent/baggage fallback must not override caller metadata
The W3C traceparent/baggage fallback in
add_litellm_metadata_from_request_headers documents itself as last-resort:
Lower priority than everything above - only fires when neither the
explicit litellm headers nor the Anthropic-metadata path found anything
But it guards on the top-level `litellm_trace_id` / `litellm_session_id` body
keys and never checks `metadata`, which is the documented way callers set
trace_id / session_id (`metadata: {"trace_id": ...}` on /chat/completions).
A caller that explicitly sets metadata.trace_id has it silently replaced by the
header value, so the implementation contradicts its own stated precedence.
This is not a corner case on managed platforms: GCP's front end injects a
traceparent into every inbound request, so the fallback fires on traffic whose
caller never sent the header. The request still returns 200 and the trace still
reaches the logging backend, just under an id the caller never chose, so any
caller correlating by its own id silently fails to find its trace.
Guard both fallbacks on the caller's request-body metadata as well.
Deliberately narrow:
- x-litellm-trace-id still outranks the body (documented priority #1)
- a request that steers neither field still adopts traceparent/baggage exactly
as before
- steering is per-field: setting only trace_id still lets session_id come from
baggage
- litellm_metadata is checked too, for LITELLM_METADATA_ROUTES (/v1/responses,
/v1/messages, batches, files)
Also corrects the comment, which understated the guard.
4 new tests; each fails without the source change. The existing traceparent and
baggage tests are unchanged and still pass.
* fix(proxy): check only the active metadata container and ignore empty values
Addresses review. The first version treated any value in either `metadata` or
`litellm_metadata` as caller steering. On LITELLM_METADATA_ROUTES the body
`metadata` is provider-facing and is only promoted later, so a session_id there
suppressed baggage before apply_missing_session_id_policy ran, and a request
with a usable baggage session id got a 400 under `missing_session_id: reject`.
An empty session_id did the same, since the policy treats "" as absent
Now only the active metadata container counts, and only a truthy value, which
matches how apply_missing_session_id_policy decides a session id is present.
The helper takes a typed `object` rather than a bare dict
Adds regression tests for both cases, including the end to end reject path,
and drops the test docstrings
* fix(proxy): count promoted caller trace ids on litellm_metadata routes
Addresses review. On LITELLM_METADATA_ROUTES the proxy promotes the caller's
trace control fields (trace_id, session_id, ...) from `metadata` into
`litellm_metadata`, but only after the header fallback runs. Checking only
`litellm_metadata` let traceparent and baggage fill those fields first, and the
promotion then skipped them because they were already set, so the trace was
recorded under the header ids
The check now also reads `metadata` for fields in
LITELLM_TRACE_CONTROL_METADATA_FIELDS on those routes, and
apply_missing_session_id_policy uses the same check, so `reject` no longer
refuses a request whose session id is about to be promoted
The earlier test asserting that a session_id in `metadata` must not block
baggage on /v1/responses had the premise wrong, since that value is promoted.
It is replaced by tests that assert the promoted caller ids win on
/v1/responses and /v1/messages, with no policy, reject, and generate
* fix(proxy): an empty trace field in litellm_metadata shadows the promoted one
Addresses review. Promotion copies a trace control field from `metadata` into
`litellm_metadata` only when the key is absent, so a key that is present but
empty in `litellm_metadata` wins and the `metadata` value is never used. The
check still looked at `metadata` in that case, which let
`missing_session_id: reject` accept a request that ends up with an empty
session id, and let an empty trace_id or session_id block the headers
The check now follows the same rule as promotion. If the active container has
the key, its value decides. Only when the key is absent does the promoted
`metadata` value count. This makes the conflicting-bucket cases behave exactly
as on main
* fix(proxy): stay within the LIT006 cast budget
The type-discipline gate failed: this branch added 4 unsuppressed `cast()`
calls and LIT006 was already at its limit. Each cast now carries a
`# cast-ok` reason. The two in the helper follow an `isinstance` check that
proves the Mapping. The two at the call sites stay because the method's
`data` parameter is a bare `dict`, and dropping those casts adds
basedpyright unknown-type errors instead. No behavior change
* test(proxy): cover caller metadata precedence over W3C trace headers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): generated session uses promoted caller trace id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move test context into assertion messages
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): audit cells for w3c fallback precedence
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): tighten audit chaos cells
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(test): drop stray blank lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): ignore body-less model-info probes in langfuse precedence upstreams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep root trace/session fields on equal ids and usable caller values
The _caller_trace_field gating added for caller-metadata precedence was
presence-aware, which regressed three pre-call behaviors versus main:
- Equal ids in W3C headers and body metadata suppressed the
traceparent/baggage branch entirely, leaving the root
litellm_trace_id/litellm_session_id unset so router fallbacks minted a
fresh uuid4 per request. The header branch now also fires when the
caller value equals the header-derived id, so equal ids stamp the root
fields exactly like main.
- A truthy but non-string metadata value (e.g. session_id 4815162342)
counted as caller-supplied and suppressed the baggage fallback, so
downstream str-only consumers (code interpreter sandbox reuse) got a
new sandbox per turn. _caller_trace_field now counts only non-empty
string values, and the generate policy falls through to generation
when the caller value is not usable.
- On litellm_metadata routes a usable caller session id suppressed the
missing_session_id policies while never landing on the root field, so
the root session stayed unset. The policy now promotes the caller's
usable session id to litellm_session_id instead of leaving it stranded.
* test(integration): non-string caller ids fall back to W3C like empty ids
The audit cell compared a headers leg against a headerless leg for
equality, which only held while a truthy non-string id suppressed the
W3C fallback on both legs. With unusable values ignored again, the
headers leg resolves to the header ids while the headerless leg cannot,
so assert the headers leg's concrete outcome instead, mirroring the
empty/null cell.
* style(proxy): trim the session promotion comment to the non-obvious why
---------
Co-authored-by: Filipe Andujar <filipeandujar@gmail.com>
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
Mateo Wang
|
ed9c02d2d9
|
fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item (#45177)
* fix(ollama): one stream id, 400 on non-text tool content, separated tool results, no empty message item The ollama/ text stream minted a fresh chatcmpl- id per chunk; every chunk now carries the one id of its response. A tool message with int content raised a TypeError inside ollama_pt and answered 500; it now answers 400 naming the message index and the type it carried. Merged user, tool, and function turns and the text parts of one message were joined with no separator; they are joined with a newline. A streamed tool call with no text rebuilds to content null instead of an empty string, and the Responses bridge no longer opens a message item on a leading empty delta, so a /v1/responses stream on a tool call completes with the function_call alone, as the non-stream call does * fix(ollama): defer the BadRequestError annotation so litellm imports on Python 3.10 to 3.13 * fix(ollama): read the user run lazily, type the prompt helpers, and 400 on a non-string text part * fix(ollama): 400 on a text or image_url part that carries no value * fix(responses): stream bridged output items at contiguous indexes and reject non-object ollama content parts The chat-completions bridge reserved output_index 0 for the message item, so a tool-only stream announced its function_call at index 1 with no item at 0 and the OpenAI SDK's responses.stream() accumulator raised IndexError. Items now take the next index in emission order, reasoning included, and response.completed lists the output in that streamed order. ollama_pt answers 400 for a content part that is not an object and for an image_url object without a url string, instead of a 500 or a silent drop. * fix(responses): list only announced items in the bridged completed snapshot The Responses bridge drops a message item from response.completed when the stream never announced one and its text is empty, so a tool-only stream and a reasoning-only stream end with the items the client saw. The chunk builder's rebuilt content goes back to what main does; aligning it with the non-stream shape is its own change for every provider's consumers * style(responses): sort the itertools import with the other stdlib imports * test(integration): add ollama prompt-tools and responses bridge audit cells |
||
|
Mateo Wang
|
58258409c9
|
refactor(proxy): expose public names for private proxy helpers (#45170)
* refactor(proxy): expose public names for private proxy helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep original class names behind public aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve internal callback filtering Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep _PROXY_ class names for managed files hooks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names in package exports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve recursive auth helper name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): align MCP limiter tests with server enforcement Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): match main's MCP limiter tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep old private names bound in importing modules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): preserve compatibility imports through strict lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): use exact pyright suppression in password helper test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): add reasons to compatibility import noqa comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): update IN-list baseline for renamed helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
9d3bf29d6d
|
fix(responses): keep cache breakpoints on blocks the bridge stringifies (#40032)
* fix(responses): keep cache breakpoints on requests that reach the Responses API The chat-completions bridge rebuilt every content block without the prompt_cache_breakpoint marker the cache-control hook had just placed on it, so the request went out with prompt_cache_options set to explicit mode and nothing actually marked. Explicit mode caches only what is marked, so those deployments lost the implicit caching they were getting before the injection point was added The native surface had a second miss. The bridging check looked the provider config up with the still-prefixed model while the config keys on the bare one, so every bedrock_mantle model read as having no native Responses support, and role-targeted injection points were deferred to a chat-completions pass that never runs for them * fix(responses): keep prompt cache breakpoint on stringified content blocks * test: cover the responses/ routing prefix in the native cache point case * fix(responses): keep only the fallback-branch marker carry, main already routes the hook #42281 landed the text, image_url and file branch carry and the implicit default on main, and the Responses routing lookup this branch changed has no observable effect there, so the PR shrinks to the unknown-block fallback branch and its regression test * fix(responses): drop a malformed prompt_cache_breakpoint under drop_params in the chat-to-Responses bridge * fix(responses): accept the 30m prompt_cache_breakpoint ttl under drop_params OpenAI's Responses API takes a ttl of 30m on an explicit breakpoint marker, so the drop_params validation keeps it instead of stripping it as an unknown key. Trims the bridge test docstrings to the dated vendor citation. * test(integration): audit cells for prompt_cache_breakpoint through the chat-to-Responses bridge Adds the /audit cells for the bridge's prompt_cache_breakpoint carry: valid and malformed markers on every carrying block kind and on tool and assistant content, the drop_params on and off contract, the hook-injected system marker, the Anthropic SDK path through the Responses adapter, upstream errors, idempotent spend rows, and the chaos cells (mixed burst, upstream outage, slow streams, worker SIGKILL and a proxy restart mid burst), all against a scripted Responses endpoint with the request it received read back by response id * test(integration): give every scripted responses reply its own id The prompt-cache-breakpoint audit's scripted upstream minted the response id from the request's marker, so three identical marked requests shared one id. The spend-log writer skips rows whose request_id already landed, and the idempotent-logging cell saw one row for three calls on every leg. The upstream now mints a unique id per reply like a real provider, and the cells match a response to its request by the marker inside that id. |
||
|
mrinal-berri
|
0e076008e1
|
fix(bedrock): map Scheduled batch jobs to in_progress on retrieve (#45157)
* fix(bedrock): map scheduled batch job status to in_progress Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): cover batch retrieve status lifecycle through the proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(bedrock): drop legacy covers marker and type job payload as Mapping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
c4b48f6b3f
|
fix(anthropic): fail closed when a token-file or inline-token federation credential is missing a rule or organization id (#45178)
* fix(anthropic): fail closed when a token-file or inline-token federation credential is missing a rule or organization id * refactor(anthropic): flatten the federation-request helper * test(anthropic): annotate the new WIF regression tests * test(anthropic): audit cells for legacy federation references missing an id Integration cells drive the seven clients, files, skills, health, Test Connect and a mixed burst against deployments whose token file or inline token reference lacks a rule id or an organization id, asserting the named 401 and no outbound call, with controls for the static key, the environment ids, the blank reference, the unauthenticated call and the complete shapes. The unit test covers the environment ids completing a token file param * test(anthropic): strip host federation ids from the audit proxies |
||
|
D41910
|
87e961fad0
|
fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary (#44800)
* fix(anthropic): let /v1/messages mid-stream failures reach the proxy failure boundary AnthropicStreamWrapper.async_anthropic_sse_wrapper swallowed every upstream exception and emitted its own Anthropic error frame, so a provider drop on a streamed /v1/messages call never reached the proxy's streaming boundary — no failure spend row, no failure callbacks. Re-raise when the stream is proxy-managed (detached failure hook armed) so async_streaming_data_generator runs post_call_failure_hook once and serializes the error frame; when consumed standalone (SDK litellm.messages path), run the logging object's async failure handler and keep the client-facing error frame. Fixes #44742 * test(anthropic): satisfy PT012 single-statement rule in mid-stream error test * fix(anthropic): dispatch both sync and async failure callbacks * fix(anthropic): dispatch both sync and async failure callbacks * test(anthropic): move the regression cases into the mapped test file * refactor(anthropic): read the public detached-failure hook and drop the explanatory comments * test(anthropic): drive the mid-stream failure tests through a real logging object * fix(anthropic): re-raise the provider error from the chat wrapper envelope so the router keeps its pre-content retry * test(integration): cover mid-stream failure bookkeeping of bridged /v1/messages streams Nine integration cells for the chat bridge: raw httpx and the Anthropic SDK (sync and async) streams that fail after the first content delta now land exactly one failure SpendLogs row carrying the provider's error class, the non-streaming 500, the response-cache twin and a client that leaves mid-stream keep their bookkeeping, a 24-stream burst lands every call id once, and the standalone SDK stream reports the provider error to failure callbacks once. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
70e6a2ff24
|
fix(vertex_ai): dial the multi-region Live API host for realtime sessions (#45166)
* fix(vertex_ai): dial the multi-region Live API host for realtime sessions
A realtime deployment with vertex_location us or eu dialed
{location}-aiplatform.googleapis.com, which Vertex does not serve for a
multi-region, so the WebSocket handshake came back 404. The URL builder now
resolves its host through the shared Vertex host resolver, which already
maps us and eu to aiplatform.{geo}.rep.googleapis.com and leaves regions
and global unchanged.
* test(vertex_ai): annotate the realtime multi-region URL tests
* fix(vertex_ai): skip the TLS argument when the realtime api_base is plain ws://
The Vertex realtime session and health-check connects passed the shared
TLS context to every websocket dial, so an api_base the transformation
already maps from http:// to ws:// failed with 'ssl argument is
incompatible with a ws:// URI' before any frame was sent. The OpenAI
realtime handler already skips the argument for ws://; the shared helper
now does the same for the Vertex path
* test(integration): cover the Vertex realtime multi-region host through the proxy
Fourteen cells on the scripted Gemini Live upstream: api_base overrides for
us, eu, us-central1 and global complete a turn; malformed locations are
refused before any dial; a missing location defaults to us-central1; the
realtime health check handshakes the override host and reports a malformed
location without dialing; a refused handshake reaches the client and the
next session connects; twelve concurrent sessions across locations each
reach the upstream once; an upstream outage closes every open session and
the proxy recovers
* test(realtime): type the health-check helpers and flatten the burst order
* test(http_handler): type the realtime TLS test signatures
|
||
|
RachelHuangZW
|
bf9f59d76b
|
fix(scheduler): remove a request's queue entry once it stops waiting (#43061)
* fix(scheduler): remove a request's queue entry once it stops waiting Requests admitted while a healthy deployment existed never left the priority queue, and neither did requests cancelled while waiting. The stale entries blocked later requests during cooldown and, with Redis, made add_request raise a TypeError on queues read back as JSON lists Both scheduling paths now share one polling helper that removes the entry in a finally block, whether the request was admitted, timed out or cancelled Related to #43059 * test(router): allowlist _wait_for_scheduler_turn in the router coverage check The coverage script only counts direct calls in test files. The helper is exercised through prioritized acompletion and atext_completion in test_router.py, like the other allowlisted entries * fix(scheduler): admit healthy requests before reading the queue, and clean up cancelled enqueues poll() raised on an empty queue before checking for healthy deployments. With the cleanup now rewriting the queue after every admission, a concurrent write from another replica can erase a waiting request's entry, and that request then failed while a deployment was healthy. poll() now admits as soon as a deployment is healthy and only reads the queue during cooldown add_request also moved inside the try block, so a request cancelled while its queue write is in flight still has its entry removed * refactor(scheduler): move the wait loop into Scheduler.wait_for_turn The router passes a healthy-deployments callable into the scheduler, so tests inject a Scheduler directly instead of replacing the router's scheduler attribute. The cancelled-mid-enqueue test moves to test_scheduler.py with the other scheduler tests, and poll() takes the deployments as a Sequence since it only checks whether any are healthy * test(scheduler): move scheduler tests into tests/unit * fix(scheduler): finish the queue removal when cancellation is delivered again during cleanup A second cancel, or an anyio cancel scope that re-cancels on every await, interrupted remove_request mid-write and left the entry in Redis. * fix(scheduler): read queue entries back from redis as tuples * fix(scheduler): admit a request whose queue entry vanished during cooldown * fix(scheduler): re-enqueue a request whose queue entry vanished during cooldown * test(integration): cover priority scheduler queue cleanup across instances Adds tests/integration/routing/test_priority_scheduler_queue_cleanup.py: 27 cells against the real proxy (two workers, Postgres, Redis, a scripted upstream) for every prioritized surface (/v1/chat/completions, /v1/completions, /queue/chat/completions, streaming and not, OpenAI SDK sync and async, raw httpx), the non-integer priority pass-through, the in-memory queue on one proxy, two proxies sharing a Redis queue (served requests leave no entry, a dead replica's entry is skipped or expires, a waiter that times out or disconnects removes only itself), a Redis outage mid burst and a SIGKILLed worker. Each cell asserts the caller's response, the upstream's requests by marker and the Redis queue contents. On the merge base the cross-instance cells fail with the list-of-lists TypeError and the in-memory cell with 408s behind the leaked entry; on this branch every cell passes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
6498bef8db
|
fix(mcp): honor scoped cache freshness (#45165)
* fix(mcp): honor upstream freshness and caller scope in discovery caches * test(mcp): type scoped freshness regression helpers * fix(mcp): age discovery freshness through cleanup * fix(mcp): preserve keyless discovery caller isolation --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
moe-berri
|
e9cfba2c17
|
feat(lens): isolate ingestion and investigations in a Rust service (#45148)
* feat(lens): isolate trace storage and investigation in a Rust service
* fix(lens): include Rust sources in the image build context
* feat(lens): wire service setup, scoped delivery receipts and lease attempts
* fix(lens): complete service routing and reject stale investigation results
* fix(lens): retry key propagation and validate isolated Compose setup
* fix(lens): seed through isolated ingestion and preserve upstream queue fixes
* chore: sync schema.prisma copies from root
* fix(lens): bind nullable due timestamps as text for Prisma
* chore(ui): remove stale lint suppressions
* fix(lens): address CI failures and review findings
* refactor(lens): remove retired Python worker and run evaluations in Rust
* fix(lens): reuse control connections and satisfy review checks
* test(lens): install and upgrade both Helm charts on Kubernetes
* test(lens): run connection reuse coverage as an integration test
* fix(ui): upgrade Next.js to 16.3.8 security release
* fix(lens): fence stale attempts and preserve reviewed evidence
* Revert "fix(ui): upgrade Next.js to 16.3.8 security release"
This reverts commit
|
||
|
devin-ai-integration[bot]
|
e48f8d928d
|
feat(mcp): rate limit all MCP operations and add server-level rpm (#44600)
* feat(mcp): rate limit all MCP operations and add server-level rpm Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): drop comments copied onto list fallbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): count discovery once per server and rate limit REST tools listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): cover rate-limited catalog error propagation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): avoid fastapi import in mcp operations test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): apply server rate limits to paginated catalog listings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): restore main's unused prompt and resource listing helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): add server rate limit coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): harden rate-limit integration setup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): prevent rejected calls from consuming shared limits Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d910653b33
|
fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge (#44119)
* fix(responses): keep prompt_cache_breakpoint markers in the chat to responses bridge Preserve cache-breakpoint markers through the bridge for supported models and drop them for models without breakpoint support Co-authored-by: Simon Sorg <simonsorg13@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): honor base_model when gating bridge cache breakpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): avoid recursive cache-breakpoint stripping Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): justify bridge stripping casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep cache breakpoints out of non-bridge converter callers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): read prompt_cache_breakpoint without validating content blocks The marker read ran every chat content block through TypeAdapter(dict[str, object]).validate_python, which rejects dict blocks with non-string keys that chat completion callers passing Python dicts could previously send; the request then failed with a pydantic ValidationError on the bridge keep path, the strip path, and the image/file conversions alike. item is already isinstance-narrowed to a dict at every read site, so read the marker with dict.get directly and drop the adapter. Adds a regression test covering text/image_url/file blocks carrying non-string keys on both the keep (gpt-5.6) and strip (gpt-4o) paths. * fix(responses): cast content block before reading prompt_cache_breakpoint basedpyright flags the raw dict.get read as reportUnknownArgumentType (+2 against the error budget); cast the isinstance-narrowed block to dict[str, object] first, matching the strip helpers' cast-ok idiom. * style(responses): ruff-format the marker-read cast * fix(responses): hoist one cast-ok content block read for the type gates A Final assignment inside the conversion loop trips reportGeneralTypeIssues, and per-site casts trip the LIT006 budget; read the marker through one cast-narrowed local instead. --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Simon Sorg <simonsorg13@gmail.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
225fd3bcdf
|
feat(proxy): add denied_passthrough_routes deny list for custom pass-through endpoints (#44924)
* feat(proxy): add denied_passthrough_routes deny list for custom pass-through endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): harden denied_passthrough_routes against non-admin clears and dot-segment paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for typed denied_passthrough_routes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): match trailing-slash deny entries, block null metadata from dropping denies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): close bulk update, encoded ?/# and ordering gaps in passthrough deny list Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): check denied pass-through routes against the path the forwarder sends upstream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type matching_denied_passthrough_route metadata mappings * fix(proxy): treat a `/` deny entry as denying every pass-through route Also types the new deny-list tests and drops their get_server_root_path mock in favour of unsetting SERVER_ROOT_PATH. * test(proxy): type the deny-list tests and drop unrelated test reformatting --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Yucheng He <yucheng@berri.ai> |
||
|
devin-ai-integration[bot]
|
ec0af8d5f8
|
fix(proxy): evict cached user on every proxy for tpm/rpm updates and edit limits in the users UI (#44130)
* test(proxy): cover cross-worker cache eviction for user tpm/rpm limit updates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): broadcast user cache eviction when tpm_limit or rpm_limit changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): return tpm_limit and rpm_limit from /v2/user/info Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): edit user tpm and rpm limits from the user edit form Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): clear user tpm_limit and rpm_limit when sent as null Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover user tpm/rpm limit updates across proxies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover user rate-limit routes and bulk updates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): reject unsafe rate limit integers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): cover user rate limit seed and saved state Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): type user endpoint test doubles Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert: drop rate limit upper bound Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): validate user rate limits with zod and clear user edit lint warnings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): rename user edit schema and name its input and output types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop redundant event loop yields in rate limit eviction test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3ca3e1c686
|
fix(streaming): stop re-wrapping a bridged stream's MidStreamFallbackError (#44989)
* fix(streaming): stop re-wrapping a bridged stream's MidStreamFallbackError A chat completion served over the Responses API already gets its mid-stream error wrapped by the Responses iterator. The chat stream wrapper wrapped it a second time, and the router unwraps one layer, so the client saw the inner sentinel (message prefixed litellm.MidStreamFallbackError, type null) instead of the provider's RateLimitError. The chat wrapper now re-raises an already wrapped error untouched. * test(streaming): type the bridged stream regression test's locals * fix(streaming): rebuild a bridged mid-stream error with the outer wrapper's bookkeeping * test(integration): cover the bridged stream error typing on every surface Checked-in audit cells for the Responses bridge: chat completions through the OpenAI SDK and httpx, /v1/messages through httpx and the Anthropic SDK, native /v1/responses, the litellm and Router SDK stream paths, and a chaos file with a mixed burst, a worker SIGKILL and a proxy SIGTERM against an owned two-worker proxy. Every cell scripts the provider through a wire server and asserts the caller's body, the upstream's received requests and the spend row by id. * test(integration): read bridged chat content through one helper * test(integration): build the bridged fallback config without mutating the loaded yaml --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
amarrtech
|
88f15e7572
|
fix(router): resume sync streaming fallbacks without retrying primary (#43959)
* fix(router): resume sync streaming fallback chain Signed-off-by: amarrtech <272048731+amarrtech@users.noreply.github.com> * test(router): cover the sync mid-stream fallback walking every configured target * test(router): cover the sync stream fallback walk against a wire upstream --------- Signed-off-by: amarrtech <272048731+amarrtech@users.noreply.github.com> Co-authored-by: amarrtech <272048731+amarrtech@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
127278f951
|
feat(guardrails): add logging_only_scope to observe input, output, or both (#43695)
* feat(guardrails): add logging_only_scope to observe one direction Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(guardrails): remove unrelated test churn Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): allow native lifecycle logging-only scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate api types for logging_only_scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): remove callbacks when scope validation fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep output-only scans when request copy fails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): configure logging_only_scope on guardrails Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep guardrails enforcing when logging_only_scope is invalid at load Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): avoid inline guardrail test fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): support directional scope in PATCH and provider UI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): format guardrail files for frontend lint Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): reset unsupported directional scope selections Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): normalize logging-only scope and sanitize warning logs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): normalize scope in shared guardrail field Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(api): sync guardrail schema artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): tolerate invalid stored logging-only scopes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): add logging_only_scope integration audit cells Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): fix scope test lint issues Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): extend K4 chaos test timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): restore stored guardrail row verbatim on rejected patch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): satisfy collection lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): validate masked params through TypeAdapter Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): tolerate invalid stored params on reads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep constructor coercion in tolerant params parser Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(ui): edit logging_only_scope in the custom code guardrail modal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): extract custom code logging-only scope control Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep no-op PUT and logging_only scan-failure semantics stable Three regressions from the logging_only_scope feature, fixed while keeping the input/output/both selection working: 1. PUT with byte-identical litellm_params no longer forces a teardown + re-init of the live callback. reject_invalid_logging_only_scope now only validates; re-init still happens exactly when params/name change. Previously a description-only PUT re-appended the callback at the END of litellm.callbacks, reordering guardrails: with a BLOCK guardrail created before a MASK one, the mask started winning and blocked requests started succeeding (400 -> PUT 200 -> 200). Invalid unchanged scopes are still rejected with 422 without touching the live instance. 2. CustomGuardrail._scan_logged_call no longer swallows input-scan exceptions per branch: a raising or BLOCKED input scan aborts the logging_only hook again (one policy call, one verdict) for guardrails that never selected a logging_only_scope. Explicit input/output/both selection keeps selecting which scans run. 3. A failed PATCH now rolls the DB row back to exactly the stored raw litellm_params (a legacy 4-key row stays 4 keys) instead of expanding it to a full LitellmParams dump; pinned with a test. This matches the PUT rollback shape and is the faithful rollback. * refactor(guardrails): type directional-scope provider list as a tuple * fix(lint): stay within the basedpyright budget - drop a Final annotation assigned inside the validation loop (reportGeneralTypeIssues over ceiling by one) - rename the PR-introduced _configured_event_hooks to public configured_event_hooks; its cross-module import added the two reportPrivateUsage errors that pushed the rule over its ceiling * fix(guardrails): keep the response verdict for explicit logging_only_scope=both on input-scan failure Greptile P1: an explicitly configured both-direction observer asked for a verdict on each direction, so a failed request scan must not silently drop the response verdict. The implicit default (logging_only_scope None) keeps the abort semantics of a logging_only hook whose scan raised, which is the base behavior the earlier fix restored. Also drops two test comments that restated their assertions (Greptile P2). * test(guardrails): align logging_only_scope integration rows with abort-on-input-failure and encrypted params Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep the response scan when the request copy fails for explicit both scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
eb9355d11a
|
feat(ui): configure Anthropic workload identity federation from the dashboard (#44889)
* feat(ui): configure Anthropic workload identity federation from the dashboard Add Credential and Edit Credential offer workload identity federation for Anthropic, Add Model creates a federated credential and attaches it, and the credentials table marks federated credentials. Editing a credential now sends only the values the admin changed, and switching provider no longer leaves the previous provider's default base URL on screen * fix(ui): keep a credential edit to what the admin set in the federation form * fix(ui): lock the provider in the federation dialog opened from Add Model * fix(ui): require one federation id when the identity source is the proxy environment * test(credentials): cover the Anthropic federation dashboard and credential routes Integration cells for the credential routes every dashboard shape writes (round trips, PATCH set and delete, malformed bodies, non-admin refusals, the token-file allowlist and exchange-host checks, every identity source through chat and messages against a scripted exchange, concurrent writes across two workers and a worker kill mid burst), plus Playwright specs for the Add Credential, Edit Credential and Add Model federation flows and the team-admin view. The owned proxies boot with a 2 s config reload so both workers serve a stored credential inside the fixture budget. * test(e2e): type the federation spec's captured bodies and clean up the Add Model deployment by its created id captureRequestBody and postAsMaster take a type parameter instead of returning Record<string, any>, the spec names the credential and model write shapes it captures, and the Add Model cell reads the deployment id from the /model/new response right after the click so a later failing check no longer leaves the deployment behind. --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
dbdf555da2
|
test(e2e): cover ollama and ollama_chat on chat completions, responses and messages (#45079)
* test(e2e): cover ollama and ollama_chat on chat completions, responses and messages Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): declare Subject metadata and check streamed tool call ids in the ollama suite Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
65dcf43257
|
perf(lens): claim worker jobs from an indexed due queue instead of scanning every lens (#45095)
* perf(lens): claim worker jobs from an indexed due queue instead of scanning every lens Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(lens): apply the due_at index concurrently on its own and default legacy rows to due Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * test(lens): pass repository to claim lifecycle tests Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(lens): page past unsupported due lenses and declare the full due_at index Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> * fix(lens): claim due lenses in a loop instead of recursion Co-Authored-By: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <155045088+ishaan-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
810a3106e1
|
test(integration): hold the wire barrier until the test releases it or the wire closes (#45124)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
1c6f714187
|
fix(realtime): run transcript guardrails on raw-path transcription sessions with a transcription-safe block (#44844)
* fix(realtime): skip guardrail VAD session.update injection for transcription sessions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(realtime): run transcript guardrails on raw-path transcription sessions with a transcription-safe block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(realtime): assert transcription session keeps transcribing after a guardrail block Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(realtime): only expect a follow-up transcript when the block keeps the session open Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(realtime): type the transcription block regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(realtime): flag transcription sessions from the route intent and backend events only A client session.update declaring session.type transcription on a voice session no longer sets the transcription flag, so it cannot switch off the guardrail's create_response gate or skip the transcript guardrail * fix(realtime): flag transcription sessions from provider-transformed session events * test(realtime): cover transcript guardrail blocks on transcription sessions --------- Co-authored-by: gabriele <gabriele@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
8283033c0e
|
test(integration): turn the model-info refresh off in every integration proxy config (#45104)
The shared integration proxy configs left the five-minute /v1/models refresh on, so every proxy booted from them sent a GET /v1/models to each openai-compatible deployment at boot and every 300 s after, including the per-test deployments other cells register against their own scripted wires. A scripted provider that asserts on the exact requests it receives then saw a GET it never scripted, mid-test. Set disable_model_info_refresh in proxy_config.yaml, coordination_redis_proxy_config.yaml, and oci_proxy_test_config.yaml, and drop the three per-test copies of that setting from the observability cells that each loaded the stock config and set it themselves. Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
31b90ebb47
|
test(observability): wait for warm-up spend rows in capped and count only the post-wipe half in X4 (#45006)
Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
0586289817
|
test(mcp): fix stale bridge-hook, applied-guardrails and pagination-revoke integration tests (#45008)
* test(mcp): fix stale bridge-hook, applied-guardrails and pagination-revoke integration tests * test(mcp): clear the spare direct grant when restoring the access-group policy --------- Co-authored-by: yuneng <yuneng@berri.ai> |
||
|
devin-ai-integration[bot]
|
740d0435a8
|
test(integration): scope the team-scoped models upstream check to its own model (#45106)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
77fc3315e5
|
fix(caching): skip the cache past max_messages and keep tool_result text in semantic prompts (#43878)
* fix(caching): keep tool calls and tool results in semantic cache prompts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): keep semantic tool prompt helpers within lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): keep structured function_call_output text in semantic prompts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): split Responses text-field collection to stay within complexity budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): tag each tool result with the position of the call it answers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): encode tool result position and output together so tool text cannot forge result tags Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): expect encoded tool result record in qdrant semantic prompt parity case Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(caching): cover tool result arrangements, SDK clients, concurrency and qdrant outage for semantic cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(caching): embed every semantic cache prompt field except volatile ones Replace the per-shape allowlist in the Python and Rust semantic cache prompt walkers with one include-by-default walker. Plain text keeps its old concatenation; any other block or message is embedded as compact JSON with call ids mapped to ordinals, cache_control dropped, and signatures, encrypted content and base64 data replaced with a short sha256 digest. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(code-quality): allow the bounded semantic cache prompt walkers in the recursion check Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(rust): expect structured JSON for unknown fields in redis and valkey semantic prompts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * Revert "test(rust): expect structured JSON for unknown fields in redis and valkey semantic prompts" This reverts commit |
||
|
devin-ai-integration[bot]
|
736da28ac1
|
fix(tests): match the lowercased bind error in the owned-proxy port-race retry (#45097)
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
91e9b1f06b
|
fix(auth): clear the recent-miss user memo when /user/new creates the user (#45020)
* fix(auth): clear the recent-miss user memo when /user/new creates the user Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(auth): pin the miss memo window and drop the class patch in the new_user regression test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): pin a first-time SSO user's first message and second sign-in on one worker * test(integration): cover /user/new clearing the user-miss memo on the creating worker The fake IdP now signs in the subject a login_hint names, so cells on the shared one-worker proxy can each use a fresh user. New cells: a plain-key miss followed by /user/new is budgeted at once (same on both legs, the auth prefetch loads the row) and the admin-created user's first SSO sign-in inside the window completes (500 at the callback before the fix). The first-sign-in cell moved onto the shared one-worker proxy fixture --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
d364d5e7ca
|
refactor: expose public hidden_params accessors (#44668)
* refactor: expose public hidden_params accessors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: keep hidden params writes on duck-typed responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor: move hidden params helpers to core utils
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve hidden params in duck-typed cost responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: cover hidden params accessors across response types
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: preserve hidden params for dynamic responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: expose logs through backend component
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "fix: expose logs through backend component"
This reverts commit
|
||
|
devin-ai-integration[bot]
|
68c0a972a7
|
fix(realtime): skip guardrail VAD session.update injection for transcription sessions (#44843)
* fix(realtime): skip guardrail VAD session.update injection for transcription sessions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(realtime): flag transcription sessions from the route intent and backend events only A client session.update declaring session.type transcription on a voice session no longer sets the transcription flag, so it cannot switch off the guardrail's create_response gate or skip the transcript guardrail * fix(realtime): flag transcription sessions from provider-transformed session events * test(integration): cover transcription sessions skipping the VAD auto-response injection Adds the realtime transcript guardrail audit cells: transcription sessions on all three realtime routes, the OpenAI SDK, the beta protocol, the whisper default deployment, the Azure GA path over a TLS scripted upstream, and Meta Muse push-to-talk sessions keep the client's session.update verbatim and get their transcript, while voice sessions keep the injected create_response gate and a client-declared transcription type no longer bypasses it. Sad, edge, and chaos cells cover malformed session fields, duplicate and older backend session events, unauthenticated upgrades, repeated sessions, an upstream outage under open sessions, and a worker kill with a proxy restart. The scripted upstream now answers session.update the way the vendor does (session.updated, or the missing turn_detection.type and session-type errors), records every websocket frame, serves the Azure and Muse realtime paths, and can run over TLS from an owned upstream. --------- Co-authored-by: gabriele <gabriele@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
e98bbc2f8e
|
fix(exceptions): map upstream 402 to PaymentRequiredError and cool down 402 deployments (#44879)
* fix(exceptions): keep upstream 402 status and cool down 402 deployments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): map 402 to PaymentRequiredError subclass of BadRequestError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): annotate PaymentRequiredError methods Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): skip 402 cooldown on single-deployment model groups Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): single prefix and 402 fallback response for PaymentRequiredError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic): map billing_error to PaymentRequiredError regardless of status Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): honor explicit allowed-fails policy for single-deployment 402s Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * revert(anthropic): drop billing_error body mapping to PaymentRequiredError Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(exceptions): type the PaymentRequiredError constructor parameters * test(integration): cover 402 PaymentRequiredError mapping and cooldown --------- Co-authored-by: Mubashir Osmani <mubashir@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
joshua-berri
|
d1cfe17518
|
feat(mcp): add portable catalog pagination (#44446)
Some checks failed
Unit Tests: Proxy DB Operations / proxy-server-core (push) Blocked by required conditions
Unit Tests: Proxy DB Operations / proxy-utils (push) Blocked by required conditions
Unit Tests / Vertex AI (push) Blocked by required conditions
Unit Tests / misc (push) Blocked by required conditions
Unit Tests / Build the Rust bridge (push) Waiting to run
Unit Tests / caching-local (push) Blocked by required conditions
Unit Tests / core-utils (push) Blocked by required conditions
Unit Tests / enterprise-managed-files (push) Blocked by required conditions
Unit Tests / enterprise-package (push) Blocked by required conditions
Unit Tests / enterprise-routing (push) Blocked by required conditions
Unit Tests / integrations (push) Blocked by required conditions
Unit Tests / OpenAI and Meta Providers (push) Blocked by required conditions
Unit Tests / All Other Providers (push) Blocked by required conditions
Unit Tests / misc-dirs (push) Blocked by required conditions
Unit Tests / proxy-auth (push) Blocked by required conditions
Unit Tests / proxy-endpoints (push) Blocked by required conditions
Unit Tests / proxy-extras (push) Blocked by required conditions
Unit Tests / proxy-feature-endpoints (push) Blocked by required conditions
Unit Tests / proxy-hooks-client (push) Blocked by required conditions
Unit Tests / proxy-infra (push) Blocked by required conditions
Unit Tests / proxy-infra-root (push) Blocked by required conditions
Unit Tests / proxy-server (push) Blocked by required conditions
Unit Tests / responses-caching-types (push) Blocked by required conditions
Unit Tests / unit (push) Blocked by required conditions
GitHub Actions Security Analysis / zizmor (push) Waiting to run
CodeQL / Analyze (actions) (push) Has been cancelled
CodeQL / Analyze (javascript-typescript) (push) Has been cancelled
CodeQL / Analyze (python) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
* feat(mcp): add portable authorized catalog pagination * fix(mcp): preserve routing and isolation across catalog pages * fix(mcp): preserve bare calls after complete catalog pages * test(mcp): verify bare calls with cached database catalog revisions * test(mcp): cover refreshed authority and failed catalog continuations * refactor(mcp): narrow pagination interfaces and preserve listed metadata * fix(mcp): publish listed metadata only after successful pagination * fix(mcp): defer bare routes until aggregate listing succeeds * fix(mcp): refresh key access groups on catalog pages * fix(mcp): accept read-only optional catalog headers * fix(mcp): preserve consolidated listing metadata after restack * fix(mcp): propagate unexpected catalog continuation failures * fix(mcp): preserve key grants when a user record is absent --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> |
||
|
joshua-berri
|
950da7c2ec
|
fix(mcp): keep worker MCP configurations consistent via a catalog revision (#42568)
* fix(mcp): refresh server catalog for each gateway operation * fix(mcp): reject configuration changes during scoped dispatch * fix(mcp): refresh shared catalog state for each operation * fix(mcp): refresh catalog before native alias routing * docs(mcp): clarify native route resolution order * test(mcp): align streaming fixture and generated API documentation * fix(mcp): preserve discovery published during catalog refresh * fix(mcp): coalesce queued catalog reads without a stale window * fix(mcp): reconcile concurrent route changes when publishing catalog * test(mcp): provide catalog scope in post-call hook fixtures * fix(mcp): retain valid live routes and handlers during refresh * fix(mcp): keep refreshed OpenAPI operation membership authoritative * test(mcp): preserve logging fixtures after catalog integration * fix(mcp): isolate transport test admission state and exhaust OAuth outcomes * refactor(mcp): narrow catalog consistency change to ticket scope Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(mcp): gate catalog refresh on a database revision marker Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): refresh waiters that observed a newer catalog revision Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): awaitable catalog revision doubles in prisma mocks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): satisfy catalog refresh lint and type gates * test(mcp): provide catalog scope in toolset fixtures * test(mcp): exercise temporary OAuth through catalog operations * fix(mcp): share active catalog snapshots across discovery tasks * fix(mcp): retain discovered routes across cached catalog operations * fix(mcp): preserve local tool ownership across discovery * refactor(mcp): satisfy tightened immutability lint ceiling Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): authorize local handlers by registered server ownership * fix(mcp): check registered handler freshness and isolate fixtures * fix(mcp): read catalog revisions and snapshots from the writer * fix(mcp): resolve access groups from the operation catalog * fix(mcp): preserve empty access group restrictions * fix(mcp): retain rediscovered routes across concurrent updates * fix(mcp): enforce registered ownership for local tool dispatch * fix(mcp): preserve issuer discovery across catalog snapshots --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
47afd1ba0b
|
test: fix shared-provider discovery, Codex catalog size and generated master key mismatches in CircleCI suites (#44905)
* test(integration): isolate Codex catalog and provider discovery fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): keep model discovery constant import-safe Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align test keys and CI env with generated master keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
c3a23fe499
|
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * types: narrow core migration diagnostics Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve runtime behavior in core symbol migration Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve core private usage migration behavior Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve private value rebinding compatibility Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore optional imports and cover public helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: exempt router property from call coverage Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: align recursive detector ignore names Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ishaan-berri
|
5fb23eabd8
|
feat(lens): add datasets built from real traces (#44765)
* feat(lens): add dataset case and size limits * feat(lens): add dataset, case and build models * feat(lens): build dataset cases from traces, findings and text * feat(lens): store dataset revisions insert-only * feat(lens): add dataset routes for build, save, export and eval cases * feat(lens): mount the dataset router before lens routes * feat(lens): add LiteLLM_LensDataset table * feat(lens): add LiteLLM_LensDataset table to proxy schema * feat(lens): add LiteLLM_LensDataset table to extras schema * feat(lens): add migration that creates the dataset table * test(lens): cover dataset case building, dedupe and limits * test(lens): cover dataset revisions, conflicts and eval cases * chore(ui): regenerate API types for lens datasets * feat(lens): add dataset UI types * feat(lens): add datasets API client * feat(lens): add dataset query and mutation hooks * feat(lens): add case selection and expected edit logic * test(lens): cover case selection and expected edits * feat(lens): add the add to dataset dialog * test(lens): cover saving picked cases from the dialog * feat(lens): add datasets list * feat(lens): add dataset detail with revisions and export * test(lens): cover editing, revisions and export in datasets tab * feat(lens): expose datasets on the lens API * feat(lens): add in-memory datasets for demo mode * feat(lens): wire demo datasets into the demo lens API * feat(lens): add optional lens API hook * feat(lens): add optional onboarding hook * feat(lens): add datasets tab and dataset routing * feat(lens): show the datasets tab * feat(lens): add to dataset from the trace header * feat(lens): add a single turn to a dataset from a step * feat(lens): add finding evidence to a dataset * docs(lens): add datasets screenshots for the PR * test(lens): cover dataset revision storage against Postgres * test(lens): cover dataset trace paging, findings and route errors * test(lens): cover dataset build fallbacks and no_content skips * fix(lens): register the dataset table for postgres span names * feat(lens): add invalid skip reason for unparseable case lines * fix(lens): keep valid JSONL cases, provenance and size limits; stop reading past the case cap * test(lens): cover malformed JSONL, re-import provenance, size fields and early cap * chore(ui): regenerate API types for the invalid skip reason * feat(lens): show text for the invalid skip reason * feat(lens): render datasets in the same inspector table as investigations * test(lens): open a dataset by clicking its table row * docs(lens): update the datasets list screenshot * style(lens): format the datasets table * fix(lens): normalize JSON text span input and output into messages when building cases * test(lens): cover JSON text span normalization for dataset cases * Revert "fix(lens): normalize JSON text span input and output into messages when building cases" This reverts commit 29c3e34962fa12427dbd516c08f2599db997a905. * fix(traces): normalize agent assistant summaries into UI messages * feat(lens): add dataset case view helpers built on the trace parsers * test(lens): cover dataset case view helpers * feat(lens): show dataset cases in an inspector table * feat(lens): open a dataset case in a side panel with trace message cards * feat(lens): rebuild the dataset page header and layout * feat(lens): keep the open dataset case in the URL * test(lens): drive dataset edits through the case table and panel * docs(lens): update dataset view screenshots * Revert "test(lens): cover JSON text span normalization for dataset cases" This reverts commit 30229ae8ae90265c09d755c43c1f1eeaa66050c7. * fix(lens): import StateMessage from shared in dataset detail * fix(lens): import StateMessage from shared in datasets list * fix(lens): give the datasets migration a unique timestamp after review checkpoints |
||
|
devin-ai-integration[bot]
|
191967c207
|
fix(cost): price batch image output tokens at the batch image rate (#44897)
* fix(cost): price batch image completion tokens at image batch rate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover default vertex batch output transformation for image cost Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): sync model prices schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): simplify batch completion cost with rate helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): keep global batch pricing fallback for image-only deployment rates Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(test): rename connect tunnel helper to https redirect Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): merge deployment batch rates over global pricing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model_prices): put flash-image batch image rate on the GA row Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(cost): drop redundant comment in batch fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): use _batch_or_half for the batch image rate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): price nano banana 2.1 fixtures off the real cost map rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(cost): use the vertex 4k image token count in the batch cost test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Hektor Jacynycz Garcia
|
157d96dcd0
|
fix(logging): log spend once for large non-streaming requests on the chat to Responses bridge (#44508)
* fix(logging): claim the async success dedup flag before the first await Nested @client wrappers on the chat -> Responses bridge schedule two async success tasks on one logging object. The dedup flag was checked at the top of _async_success_handler_body but only set after several awaits; the worker-thread base64 offload for messages >= 256 KiB yields there, so the second task passed the same check and the request was logged twice (two SpendLogs rows, key/team/daily spend counted twice). Fixes #44500 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(logging): serialise async success handlers per logging object instead of claiming early Claiming the dedup flag before the first await (previous commit) let a first handler that is cancelled before its callbacks run (e.g. by the logging worker's per-coroutine timeout) suppress the second one, losing the request's success accounting. Serialise non-streaming handlers on the same Logging object with an asyncio.Lock kept in a module-level WeakKeyDictionary: the second handler waits, then skips if the first logged, or logs itself if the first raised or was cancelled first. Streaming chunk logging is not locked. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(integration): pin one spend row and one key charge for large bridged chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): wait for the large request's spend row before asserting Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): use fixed timestamps in async success dedup tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(logging): trim the async success dedup lock comment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(logging): reset background interaction dedupe under the success dedup lock Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): use a fixed start time in the interactions logging helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: claim the async success log at schedule time instead of locking the handler Nested @client wrappers (chat over the Responses bridge, Anthropic Messages over the chat adapter) exit with one shared Logging object and each queued their own async success handler. The handler checked has_logged_async_success before its awaited work and set it after, so both could pass the check and the request was logged and billed twice. _schedule_async_success_logging now claims the log on the Logging object synchronously through claim_async_success_log, and a later wrapper returns without enqueuing. There is no await between the check and the decision, so no lock is needed. The per-object asyncio.Lock, the WeakKeyDictionary that held it and the locked block in async_log_background_interaction_completion are gone. Tests: the regression test moves to tests/unit/test_utils.py and drives two _dispatch_success_logging exits with one Logging object through the logging worker, asserting one log with the inner result. It fails without the claim. The cancelled-handler test is dropped since a second wrapper no longer retries. The background interaction completion test stays. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(logging): pin the background completion event order and name the innermost wrapper in the claim docstrings Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): send one bridged request and assert the inner Responses result is the logged one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
abd3af422d
|
fix(caching): never store or serve a chat completion with no choices (#44709)
* fix(caching): never store or serve a chat completion with no choices A provider response with empty choices was written to the response cache and served on every identical request until the TTL ended, with no provider call in between. The cache now skips storing such a response and treats an already stored one as a miss, so the next request goes back to the provider and its answer replaces the entry. * fix(caching): skip responses with no output on the Responses API and Anthropic Messages too * fix(caching): skip streams and stored entries that carry no output A chat or text completion stream whose chunks carried no choice is closed by the stream wrapper with one empty choice of its own, so the assembled response passed the choices check and was cached. The assembled stream is now judged on its content: a stream with no text, tool call, or other output in any choice is never stored, on the async and sync writers alike. The Responses API stream writer and the Anthropic Messages stream writer apply the same no-output check before storing. A stored entry with no output read through the worker memory tier is now evicted from that tier on the miss, so the next read reaches Redis where the refill lands; the text completion and messages writers only write to Redis, and the memory copy otherwise kept missing until its own TTL. * test(integration): response cache cells for answers without output Deterministic cells for the response cache on every unified endpoint, streamed and not, through the OpenAI and Anthropic SDKs and raw httpx, plus the sync SDK paths, stale entries, malformed answers, per-request TTLs, cache delete, and chaos (Redis stopped or paused mid burst, a worker killed, in-memory cache mode). The scripted upstream counts only POSTs as deployment calls, since the proxy's boot-time GET /v1/models discovery of a config deployment is not one. * test(caching): pin the stored entry timestamp in the worker-copy test * test(integration): drop the restating comments from the chaos cells * test(integration): close the breaker on the first call after the Redis restart --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
282d733fb6
|
feat(auth): deny search tools by default when search_tool_deny_by_default is set (#44490)
Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
abc543e701
|
feat(proxy): add opt-in vector_store_deny_by_default for least-privilege vector store access (#44244)
* feat(proxy): add opt-in vector_store_deny_by_default for standalone virtual keys
Adds general_settings.vector_store_deny_by_default (typed bool, default false). When enabled, a virtual key
with no team must list the requested vector store in its object_permission.vector_stores; no permission
record, null, or an empty list is denied with key_vector_store_access_denied. Omitted or false keeps the
existing behavior, including nonempty allowlist enforcement. The master key is unchanged in both modes.
Team keys and keyless callers are deferred to later increments of LIT-6035
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): require key and team vector store grants for team keys under vector_store_deny_by_default
With the flag enabled, a virtual key on a team needs both its own grant and its team's grant for every requested vector store. A missing permission record, an empty list or an unresolved team grants nothing. Dashboard session keys and the master key keep their existing behavior, and flag-off behavior is unchanged
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): require user or team vector store grants for keyless requests under vector_store_deny_by_default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover vector_store_deny_by_default through a real proxy with key, team and JWT identities
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for vector_store_deny_by_default
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): cover path and file_search vector store ids under vector_store_deny_by_default
Strict mode now reads vector_store_ids and tools[].vector_store_ids from the request body without needing a vector store registry, so /v1/vector_stores/{id}/search and Responses file_search are checked. User grants load through the object permission cache, and the proxy admin user rebuild keeps object_permission_id
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): broadcast user entitlement cache eviction to every worker
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): reuse VectorStoreRegistry id extraction for vector_store_deny_by_default
Strict mode now uses get_vector_store_ids_to_run on an empty registry when none is loaded, instead of a parallel set of request-shape helpers, and vector_store_access_check documents the policy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): add strict vector store audit cells for routes, SDKs, workers and concurrency
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): consolidate vector_store_deny_by_default coverage to core cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): reject invalid vector_store_deny_by_default at config load and return 400 for malformed vector_store_ids
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(auth): read vector store key and team grants through the object permission cache
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(auth): return typed permission rows in request flow vector store tests
Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(auth): validate vector store ids and tools as immutable sequences
Co-authored-by: mrinal <mrinal@berri.ai>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: mrinal <mrinal@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
|
||
|
moe-berri
|
5147aefa8b
|
fix(lens): reuse trace reviews and consolidate findings across runs (#44778)
* fix(lens): reuse trace reviews and consolidate findings across runs * chore: sync schema.prisma copies from root * fix(lens): preserve partial reviews and bound budget admission * fix(lens): resolve CI regressions and clarify reused runs * test(lens): exercise review reuse in worker container smoke * fix(lens): show actual reviews when a run finishes * fix(lens): distinguish reuse plans and stop blocked scans * fix(lens): retain partial findings when model requests stop * docs(lens): clarify budget edits during active runs * fix(lens): publish findings only after reconciliation completes * fix(lens): guide insufficient-budget runs to budget settings * fix(lens): renew budget holds and bound admission waits * fix(lens): preserve provider errors during budget cleanup * fix(ci): update PgBouncer and sharp for current builds * fix(lens): serialize settlement and fence review checkpoints * fix(lens): defer generated Prisma client type import * test(lens): verify checkpoint and progress rollback together --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
837c6a7481
|
fix(security): remove the publicly known master key from the repo (#44718)
* fix(security): hash the publicly known master key Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: replace weak master key examples and regenerate artifacts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: replace weak key fixtures with generated test keys Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: generate master keys for proxy startup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: preserve lens dev key entropy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: restore proxy key compatibility in scrub examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: scrub merged SSO fixture and refresh dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ci): stabilize test keys and metadata collection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: drop the rebuilt dashboard bundle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
090a4c3f24
|
fix(ui): read MCP submission rules from bare-array /config/list response (#44648)
* fix(ui): read MCP submission rules from bare-array /config/list response Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): cover submission rules contract and dashboard preload Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): disable MCP submission rules editor until rules load Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): format MCP submission integration test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): restore prior MCP submission rules after the rules spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): move MCP submission rules setup and cleanup into fixtures Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): use Promise.withResolvers in MCP rules loading test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(ui): clear lint warnings in MCPSubmissionsTab Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(ui): drop redundant JSX comments in MCPSubmissionsTab Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mrinal <mrinal@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |