* fix(ui): polish Lens runs loading, reload, and time range menu
Port the dashboard-only parts of a0a275e486, 1325389624, 5e7b0afd5c, 03bec959bf, 93e6ccff8d, 3d35d9f920, dc1a2b5b60, 1d0397f1c0, 773bfef066, f3387221ea and 3d049cd4a8 from litellm_lens_server_search onto main
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep the Lens timeline on the shown runs' window during a reload
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(ui): drop a redundant fixture comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): lead the Lens investigation detail with a run report
Port of the dashboard changes from a43f111abe and 29a1f3175d onto main. The detail view now opens with a run report for the selected run: status, a headline, progress or the failure, then cost, duration, coverage and issues, plus a collapsed activity log. An ordered situation table picks the report's one next action (Run now, Stop run, Retry, Raise budget, Connect worker, Review issues, Monitor this), and Run now moves into the investigation actions menu
Main's live review stays as is below the report, the queue reason still shows under queued progress, and partial results keep their own warning state with the run details folded away
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): keep Stop run on older Lens runs while another run is active
Also drop doc comments that restate the code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(vertex_ai): add regional endpoint uplift to gemini-3.1-flash-image
Google prices Gemini 3.1 Flash Image at 1.1x on non-global endpoints for
input, text output and image output, but the cost map row had no
regional_endpoint_uplift_multiplier, so regional calls billed at the
global rate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(integration): cover regional uplift spend for gemini-3.1-flash-image
* test(integration): read the proxy salt from the environment in the uplift spend cells
---------
Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
The native response-cache runtime attached through Cache._native_cache is
unreachable since the V2 cache replaced it. Drop the Python branches and
wrapper, the _ResponseCacheRuntime pyclass and its backend/activation/
semantic modules, the python-bridge deps only they used, and the tests and
fixtures dedicated to that path.
Co-authored-by: Yujong Lee <yujong@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): honor caller stream flag when provider forces SSE
The Responses handlers decided whether to hand back a streaming iterator
from the provider payload's `stream` field, which chatgpt sets
unconditionally because the Codex backend only serves SSE. A caller that
sent `stream: false` therefore received a raw SSE stream on /v1/responses,
and the chat-completions bridge failed with "Unknown items in responses
API response: []" once its recovery path lost the raw SSE it reads from
Transport streaming still follows the provider payload; only the caller's
own `stream` value now decides the response shape. When the provider
forces SSE for a non-streaming caller the body is read and aggregated
through the existing path
* test(chatgpt): inject the authenticator into the responses config so handler tests never log in
* fix(responses): treat an extra_body stream flag as the caller's own and drop a redundant comment
* test(integration): audit the chatgpt caller stream flag across responses, chat and messages
---------
Co-authored-by: SeongWoon Cho <coffee@soylatte.kr>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(chatgpt,github_copilot): refuse device-code login when an event loop is running
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(github_copilot): drop stray whitespace change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt): bound token refresh timeout and drop placeholder assignment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(chatgpt,github_copilot): keep the token file path out of the event-loop 401 message
* fix(auth): refuse device-code login from worker threads too
/v1/messages runs its handler in an executor thread, where the
running-loop check never fires, so a chatgpt or github_copilot model
still started the interactive device-code login there and the request
hung for up to 15 minutes. The guard now also requires the main thread,
so the login only runs where a human can actually answer it.
* test(chatgpt): keep authenticator tests out of the real token directory
* fix(chatgpt): keep the 5 second connect timeout and the operator's request_timeout on the token refresh call
* test(integration): cover the device-code login guard on the proxy and the SDK
---------
Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): bedrock_mantle-route basic translation cases on messages, chat completions and responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): rename translation runner run to assert_translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): use assert_translation and drop the LIT-9196 skips in the bedrock_mantle basic cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(lens): add traceShareUrl helper for shareable trace links
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat(lens): add copy link button to trace header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(lens): cover copy link on trace header
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(proxy): cap batch file records, daily batch uploads, and per-file downloads
Adds three opt-in limits for batch jobs, each settable in general_settings as a
per-key default and overridable in key or team metadata by a proxy admin:
max_batch_file_records rejects a purpose=batch upload with more request lines
than allowed with a 413 before it reaches the provider.
max_batch_file_uploads_per_day counts accepted batch uploads per key and per
team in a UTC day and returns 429 with Retry-After once the count is used.
max_file_downloads_per_minute counts GET /v1/files/{id}/content per key and per
team for each file in a one-minute window and returns 429 with Retry-After.
The existing admin-only guard for batch_enqueued_token_limit now covers all four
metadata keys.
* fix(proxy): keep file usage counters in their own store and gate batch limits on user creation
* fix(proxy): count keyless JWT callers per user, require positive file caps, and take the upload slot after request validation
* refactor(proxy): end file usage cap describers with an explicit return after the match
* fix(proxy): keep file usage counters when more than 200 are live without Redis
The file usage counter store used a default in-memory cache, which holds 200
entries and evicts the one that expires soonest. Without Redis, a caller got a
fresh per-file download allowance after touching about 200 other file ids in
the same minute, and a key got a fresh daily upload allowance once about 200
other keys had uploaded that day. The store now tracks up to 20,000 live
counters per worker, the same bound the login throttle uses
* test(proxy): move the file usage cap tests into the directory the proxy shard runs
Main's shard coverage check found tests/unit/proxy/openai_files_endpoints
claimed by no shard, so its tests would not run in CI. The file moves next to
the other files endpoint tests in tests/unit/proxy/openai_files_endpoint, which
the proxy-endpoints shard already runs
* fix(proxy): declare the file usage counters as rate limit calls
Main's redis producer gate requires every module that writes a shared cache to name its key family, and the file usage counters wrote theirs without one.
* test(files): audit batch file usage caps across processes, Redis outages, and config reloads
* test(files): guard the chaos cells against minute boundaries and open Redis breakers
Two chaos cells each failed once in the audit run. The restart check ran
three sequential downloads with no guard against straddling a UTC minute,
and the exact-cap probe after a Redis outage ran while both workers' Redis
circuit breakers were still open (60 s default recovery), so it counted in
per-process memory and the two workers split the cap
Every burst now carries a window guard, the chaos fixture lowers the
breaker recovery to 2 s, and the post-outage check drives a fresh key to its
cap through a one-worker sibling proxy and then expects the two-worker
candidate to refuse the whole burst, which only the shared Redis count can
produce, polled until the breakers close
* test(files): release the held uploads when the killed-worker cell fails early
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(vertex_ai): apply regional endpoint uplift on image generation cost path
completion_cost had vertex_location but never passed it to the image
generation cost router, so a regional_endpoint_uplift_multiplier on a
Vertex image row would be ignored. No image row carries the multiplier
yet, so nothing is misbilled today. Pass the location through to the
Vertex image calculator for both the token-based price and the
per-image fallback.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(vertex_ai): cover regional image cost through the proxy logging path
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(images): hand image_edit's vertex_location to its cost resolver
* test(integration): audit Vertex image regional uplift billing
* test(integration): require every spend row after a proxy restart
* test(integration): reject extra spend rows after a proxy restart
---------
Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): vertex_ai-route basic translation cases on messages, chat completions and responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): rename translation runner run to assert_translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): use assert_translation in vertex_ai-route basic translation cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): judge the free-model budget waiver by the group an alias routes to
A hidden model_group_alias can reuse the name of a real model_name. The
router serves that name from the alias target, but two of the three checks
behind the free-model budget waiver read the deployments of both the alias
target and the real model that shares the name.
So an over-budget key was served through an alias whose name belongs to a
model with an explicit $0 price when the target is $0 only by the cost map,
which the same key is refused on by its own name. The mirror case refused a
free target because the shadowed name belongs to a PTU-priced deployment.
Resolve the alias once and have the explicit-cost and PTU checks read the
routed group, the same group the price check already reads.
* test(proxy): cover the plain alias form of a shadowed free model name
The same wrong verdict exists for a plain string alias, so the unpriced-target case now runs for both alias shapes.
* fix(proxy): judge an alias chain's budget waiver by the deployments it is served from
The explicit-price and PTU checks read the alias target through
Router.get_model_list(), which follows a second alias hop when the target is
itself an alias key. The router never takes that hop, so an alias chain was
judged by a deployment the request never reaches. The checks now take the
deployments named after the routed group, or the wildcard deployment serving
it when none carries its name.
* fix(proxy): refuse the budget waiver when an alias chain is served by a priced wildcard route
* test(integration): cover the shadowing alias budget gate end to end
Adds the audit cells for a hidden alias whose name shadows an explicitly
free group: streamed SDK refusals on chat, responses and messages,
embeddings, the free wildcard and mixed-group paths, per-model budgets,
JWT and custom auth callers, cache hits, alias removal under traffic,
provider failure and fallback, a concurrent outage burst, and a worker
kill on an owned two-worker proxy
* test(integration): cite the cost-map rows the shadowing alias cells rely on
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(ci): namespace claude session ids in tracing seeds and allowlist /v1/logs on backend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): allowlist /v1/logs on the gateway alongside /v1/traces
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(gemini): stop replaying thinking block signatures to Gemini
A thinking block's signature has no provenance, and LiteLLM never fills it
from a Gemini response (Google signs text and functionCall parts, which ride
provider_specific_fields and the tool call id), so a Claude signature replayed
through a mixed model group reached Gemini as a thoughtSignature and Google
answered 400 Invalid thought signature on every later Gemini-served turn. The
same replay also sent the thinking text a second time as a plain text part.
The thinking text now goes out once, as the thought part built from
reasoning_content, and no part is built from thinking_blocks
* test(gemini): type the parts helper and split its comprehension
* test(gemini): cover thinking signature replay on the integration rig
Two integration files from the audit of the foreign thought signature fix: 62 wire cells asserting the model turn Google receives on chat, messages and responses across gemini and vertex_ai, streaming and not, SDK and httpx clients, the sad shapes of thinking_blocks, context caching through cachedContents, and 3 chaos cells (a concurrent burst across endpoints, upstream stream drops, a worker SIGKILL mid burst)
* test(gemini): read the integration salt from the environment
The wire test decrypted Responses ids with a literal salt; tests/integration/_support/process.py boots the proxy with LITELLM_SALT_KEY when it is set, so the test now reads the same variable with the same default
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(integration): bedrock_invoke-route basic translation cases on messages, chat completions and responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): rename translation runner run to assert_translation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): call assert_translation in the bedrock_invoke basic cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): assert the sibling-replica cooldown through the router
* test(e2e): warm the cooldown reads concurrently so every pod's read lands just before the trip
* test(e2e): send the trip right behind the warm so every pod's cooldown read is pinned to it
* test(e2e): warm every pod with a canned-answer group and trip only after every warm call answered
* test(e2e): trim the sibling cell's module docstring to what the design needs
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(prometheus): cap series per metric for every labeled metric
Add prometheus_metrics_max_series_per_metric: per metric and per worker process, the first N label
sets keep a series of their own. Counters and histograms record every later label set on one series
whose labels are all "other", so totals stay exact, and gauges skip it. The cap holds with multiple
workers because it never needs to remove a series.
Add prometheus_metrics_ttl_seconds: a series idle for that long is removed and its slot is freed.
The prometheus client cannot remove a series in multi-process mode, so the TTL is ignored there with
a startup warning.
Both settings are off by default. The end_user caps are unchanged.
* fix(prometheus): share the series cap across workers of one proxy instance
Workers writing to one PROMETHEUS_MULTIPROC_DIR now agree on which label
sets get a series through an append-only admissions file per metric, so a
merged scrape stays at the cap plus `other` instead of growing with every
worker and every worker restart. The two fallback counters now pass their
label names as a keyword so the cap and prometheus_exclude_labels apply to
them, admission and child creation happen under one lock, the test fixture
restores the shared registry, and the `other` label value lives in
constants.py.
* test(prometheus): check emitted labels instead of wrapper types, close the admission match
The exclude-labels test now emits through the spend and provider budget
metrics and checks the scrape keeps all their labels. The admission match
arms end in assert_never so the match is exhaustive.
* fix(prometheus): return the exhaustive-match fallback so every admission arm returns
* fix(prometheus): pick the series tracker with isinstance so every path of _admits returns
* fix(prometheus): skip an admissions line a worker could only write part of
* fix(prometheus): frame each admissions record with newlines so a cut-off record cannot swallow the next
A record a worker could only write part of used to merge with the next worker's record, and both were skipped for one request. Each record is now written between two newlines, so the fragment is a line of its own. The clock fixture in the series tests starts from a constant instead of reading the real clock
* fix(prometheus): ignore a non-positive series cap or TTL with a warning instead of failing the logger
A cap or TTL of 0 or less raised at logger init. The proxy logs that as a non-blocking error and keeps serving, so the result was a running proxy with no Prometheus metrics at all. The setting is now ignored with a startup warning naming it, the same rule the end_user cap already follows for a non-positive value
* fix(prometheus): start the series cap over on a one-worker restart and audit it live
A proxy with one worker and an operator-set PROMETHEUS_MULTIPROC_DIR now drops litellm's admission files at boot, so a restart frees every slot there the way it already does with several workers. A cap or TTL that is not a number greater than 0 (a bool, a non-numeric string, an empty value) is ignored with the startup warning instead of breaking the logger
The integration cells drive the cap on every endpoint through the OpenAI and Anthropic SDKs and raw httpx, streaming and not, plus gauges, cache hits, failures, both workers of one instance, the TTL on one worker and its warning on two, ignored settings, excluded labels on the fallback counters, a null cap, /config/update, a concurrent burst scraped mid-flight, a provider outage, a killed worker, and restarts with one and two workers
* fix(prometheus): wipe an operator-set multiprocess directory on a one-worker boot too
* fix(prometheus): leave the multiprocess directory alone on a setup-only run
A run with --skip_server_startup starts no worker, so it no longer creates or
wipes PROMETHEUS_MULTIPROC_DIR. Wiping there deleted the samples of a proxy
already running against the same directory
* fix(prometheus): free the capped series slots when a gateway or backend container restarts
The component image entrypoint starts uvicorn without the proxy CLI and wiped only the .db sample files at container start, so the admitted-series files of the previous container survived an in-place restart. Every label set seen after the restart was then counted on `other` once the previous container had filled the cap
* test(prometheus): cover a setup-only run and a gateway image restart under the cap
Two integration cells from the audit: a `--skip_server_startup` run pointed at a live
two-worker proxy's operator directory leaves its samples alone, and the gateway image
(`docker/component_entrypoint.sh` running `python -m gateway.launch`) restarted on a kept
PROMETHEUS_MULTIPROC_DIR starts the cap over. The burst cells now wait for every counter
they assert on, since the request and failure counters of one call increment at different
points of the logging callback
* test(prometheus): prove the cap reaches the fallback counters in the X1 cell
* fix(prometheus): ignore a cleanup interval that is not a number of at least 0
A string or negative prometheus_metrics_cleanup_interval_seconds reached the
series tracker unvalidated, so the first labeled emit with a TTL on raised
TypeError inside the callback and recorded no series. The interval is now
validated the way the cap and the TTL are: an invalid value is ignored with a
warning and the default 60 seconds applies. The I2 integration cell drives a
string interval through a live proxy and reads the warning from its log
* test(prometheus): give the restart cells the boot budget of their siblings
C4 and C5 boot two proxies each and hit the file's 240 s budget on a loaded
box; C3 and D1 already carry 420 s
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(lens): restate response contract during model repair
* fix(lens): separate instructions and recover rejected results
* fix(lens): correct loop type annotations and checks
* fix(lens): preserve access to prior findings after compaction
* fix(lens): cap result retries and preserve partial completion
* fix(responses): honor request cache controls on chat completions bridged to the Responses API
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): assert spend and cache-hit status on bridged no-cache rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): unskip LIT-9196 openai_responses basic translation cases
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): restore azure attribution check on bridged no-cache rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): exercise bridged cache controls through a real local cache instead of patched responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Add Reka as an OpenAI-compatible provider
* Add Reka supported endpoints
* fix: remove stray fragment after reka entry in provider_endpoints_support.json
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(reka): register provider identity and cover routing, credentials, and bridged endpoints
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: split slow unit shards and build the Rust bridge once per run
* ci: key the Rust bridge cache on source files only
* ci: keep the unit setup ceiling unchanged with the shared Rust bridge
* ci: fall back to the Cargo cache when the Rust bridge artifact is missing
* ci: keep reruns on enterprise-routing for the prompt caching flake
---------
Co-authored-by: yuneng <yuneng@berri.ai>
uv only rebuilds the editable litellm package when its cache keys change, and
the default keys are pyproject.toml, setup.py and setup.cfg. Editing or
switching to a branch with different Rust code left the old
litellm/rust_bridge/_native.abi3.so installed. Key the build on the Rust
toolchain pin, Cargo config, lockfile, workspace manifest and every file
under litellm-rust/crates, since crates embed .sql, .json and .jinja files at
compile time.
Co-authored-by: Nate Armstrong <narmstrong@Nates-MacBook-Pro.local>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(lens): show investigation findings for agent traces
Replace child tool-error counts in the trace table with distinct investigation findings. Keep unassessed traces separate from completed clean investigations and use the root status for failure filters and timeline counts
* fix(lens): stabilize findings updates and repair UI checks
* fix(lens): allow viewer findings reads and index trace lookups
* test(lens): cover viewer findings reads with Postgres
* test(integration): point the scratch upgraded proxy's read replica at the scratch database
* test(integration): check the scratch upgraded proxy's reader role is connected to the scratch database
* test(integration): assert every configured proxy role holds a scratch connection without a mode branch
* test(integration): skip backend workers without a role in the scratch connection scan
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* fix(router): retry a /v1/messages stream the provider drops before the first content chunk
A /v1/messages stream that the upstream closed before any content reached the
client answered an error event after a single attempt, so the router's
num_retries never applied to that drop. The pre-content failure is now retried
within the model group before the fallback chain runs, with the budget resolved
the way a failure raised before the stream opened resolves it: a retry policy
that names the error class, then the request's num_retries, then the
deployment's, then the router's. A drop after content reached the client keeps
surfacing the provider's error after one attempt.
Fixes#44238
* fix(router): hand a retry's non-retriable error to the fallback chain and type the retry helpers
A retry that failed before its stream opened with an error no retry covers raised straight to the
client, skipping a fallback the first attempt would have used. assert_never now comes from
typing_extensions so the router imports on Python 3.10, and the retry helpers read their kwargs
through typed narrowing instead of Mapping[str, Any]
* fix(router): cast the untyped router fallback defaults the stream retry gate reads
The retry gate passed the router's fallback attributes, declared without element types, to the
typed request override helper, which basedpyright counted as new unknown-argument errors
* fix(router): consult context_window_fallbacks when a retried /v1/messages stream overflows
A retry attempt raising ContextWindowExceededError reached the fallback chain inside its
mid-stream envelope, so only the regular fallbacks list matched. The fallback attempt now
unwraps it the way it unwraps a content policy error. The new router helpers are covered for
the router code coverage check with two direct-call tests and named covering tests
* fix(router): retry a 408 raised by a /v1/messages retry and honor deployment num_retries before the stream opens
* fix(router): attribute a retried /v1/messages stream to the deployment that served it and bound the retry-policy hold
* fix(router): retry /v1/messages error frames under their retry-policy class and keep the first drop's committed budget
An `event: error` frame that arrives before the first content delta now raises the exception class the pre-stream mapping gives an HTTP answer with the same status (429 RateLimitError, 500 and 529 InternalServerError, 503 ServiceUnavailableError, 504 Timeout), so a retry policy's per-class budget governs it the way it governs the error before the stream opened. The status the client sees is unchanged
A retry that lands on a sibling deployment keeps the budget the first drop committed to, read back from the request's attempted_retries and max_retries, instead of recomputing it from the new deployment's num_retries, matching the pre-stream retry loop
* refactor(anthropic): keep the error-frame exception mapping under llms and type the retry test helper
The status-to-exception mapping an `event: error` frame gets before the retry policy is consulted now lives next to the Anthropic error status map in llms/anthropic/common_utils.py, with its own unit test, and the two-deployment retry test helper takes explicit typed parameters instead of a bare dict and untyped kwargs
* refactor(anthropic): map an error frame's status with explicit returns on every path
* fix(router): map stream error frames through the pre-stream exception mapping
An overloaded `event: error` frame on a /v1/messages stream now raises the InternalServerError a 529 answer maps to, built by exception_type from the frame's own body, so one retry policy class governs the error before and after the first byte; a failed fallback after such a frame answers 500 like every other litellm path instead of the frame map's 503
A model_group_retry_policy that does not parse (a non-integer budget, an entry that is not a mapping) no longer fails every healthy stream of that group before its first attempt: the stream runs with no policy and the plain num_retries budget, with a warning naming the group
* fix(router): forward an error frame nothing can take over for as the provider sent it
A pre-content error frame whose class the retry policy grants no retry, with no fallback configured, raised an HTTP error only on the first attempt while the same frame after exhausted retries reached the client verbatim. Both now pass through as sent, the way the merge base forwarded every frame.
* test(integration): audit /v1/messages pre-content retry across routes and budgets
Adds the /audit cells for the pre-content stream retry: the native Anthropic route
(drops and error frames before content, HTTP rejections before the stream opens, SDK
sync and async, after-content and non-retriable controls, budget exhaustion, cache
twin, spend row and headers), the chat and responses bridges, the generic routes
(responses, chat, vllm pass-through, Gemini generateContent, fine-tuning jobs list),
owned two-worker proxies for router-level budgets, retry policies and fallbacks, and
two chaos cells (a worker killed mid burst, an outage on every first attempt). Shared
helpers for scripted Anthropic SSE upstreams and OpenAI-compatible wire replies live
in tests/integration/_support
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* fix(proxy): let a listed team alias win over a same-named key alias in the customer model check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): check each requested name in a plain loop in can_customer_access_model
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): only let a listed team alias skip the customer check when its target is live
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): move the per-name customer alias check into a local function
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend-tracking): stop caching failed spend-log metadata lookups as confirmed misses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(spend-tracking): share the short-lived miss cache write
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): cover key alias recovery after spend log lookup failures across usage routes
* test(spend): bound outage alias lookups per miss window instead of a fixed count
* fix(spend-tracking): treat any spend-log lookup failure as a short-lived miss
The Prisma client raises a plain AttributeError when the database drops
the connection mid-query, so the PrismaError catch let it through and
the whole usage call answered 500. Any failure now keeps the 30 second
backoff only, and the integration proxy patches its test entitlement at
import so uvicorn's spawned workers inherit it
* test(integration): audit spend-log metadata recovery under timeouts and dropped connections
Cover the daily activity routes, the usage AI chat, the Vantage and
CloudZero dry runs and exports under a locked spend-log table and under
a database connection dropped mid-lookup, on a two-worker proxy, with
the recovery after the outage asserted through the proxy's own miss TTL.
Add a dropped_connection_relay that closes only the connection whose
bytes carry a trigger, so a cell can drop the one connection the
recovery query runs on while the rest of the pool keeps serving. Rewrite
the sweep and JWT cells for the merged main: the export route reads
metadata by SQL join and never calls the recovery, the search routes
answer key rows and find deleted keys by alias, and the daily-spend
owner recovery names the user while the alias stays blank. The sweep
cell now times out a second lookup under the same lock, which pins the
keys blank on the merge base and recovers on this branch.
* test(integration): match a dropped-connection trigger split across two reads
The dropped-connection relay checked each TCP read on its own, so a SQL
marker that straddled two reads never tripped it and the outage cells
would run without the outage they meant to exercise. Carry the tail of
the previous read into the next check, as the held-statement relay
already does, and pin that with a unit test that splits the trigger
across two writes.
* test(integration): scan relay triggers through an in-process helper
The dropped-connection relay now matches its SQL trigger through a TriggerScanner that carries the previous read's tail, and the unit test exercises that scanner directly instead of opening loopback sockets, which tests/unit forbids. The relay's end to end behavior stays covered by the integration cells
---------
Co-authored-by: gabriele <gabriele@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
