* feat(pass-through): add eu/jev decision API pass-through with spend tracking
Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted
decision API that speaks the same POST /v1/systemone contract as TypeSafe.
It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to
https://jev.bevel.software), books spend through the shared decision logging
handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the
cost map
* fix(pass-through): default eu_jev to its canonical eu-jev.com origin
* fix(pass-through): reject custom_body on the eu_jev route
A key limited to other models could send an eu/jev request through
custom_body, because auth checks the outer body while the shared
pass-through forwards custom_body upstream. The route now answers 400
before anything is forwarded, the same way validate_oss_request guards
the laya and bespoke routes
* fix(pass-through): give the eu_jev route request and 200 response schemas
The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and
OA003. The route now documents the eu/jev System One request body and
its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI
document. The lazy OpenAPI snapshot and the dashboard API types are
regenerated with python -m litellm.proxy._lazy_openapi_snapshot and
npm run gen:api
* fix(pass-through): reject streaming on the eu_jev route
A body with stream=true switched the route to streaming, which skips the
decision spend tracking. Reject it with a 400, as validate_oss_request does
for laya and bespoke.
* fix(pass-through): type the eu_jev request body like the other decision routes
* fix(pass-through): check eu_jev decisions against the model eu/jev runs
Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted
model and the legacy names to eu-jeff-latest for key model checks and pricing,
check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when
EU_JEV_API_KEY is unset
* fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases
Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so
model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and
upstream still gets the client's exact body. eu_jev is a registered decisions
provider, so access groups, config deployments, /v1/systemone and /v1/decisions
accept it, and the Admin UI can add it. A release without its own cost map row
is priced as eu-jeff-latest instead of $0
* fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models
* fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it
* fix(eu_jev): take pass-through credentials only from proxy-wide deployments
The eu/jev route now ignores team-scoped deployments when it picks the
upstream key and base, so one team's deployment can no longer redirect
another team's calls, and a base ending in /v1 is not doubled
* fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it
Scope the mapped pass-through and gateway allowlist entries to the decision
route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep
their auth, metering, and gateway behavior from before this PR
* fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it
Drop the decision path from the mapped pass-through list and classify it with one
check that asks the pass-through registry, so an operator's own pass-through at that
path keeps its auth, metering, LLM-route status and litellm_user_api_key handling
from before this PR. The built-in route now yields on a per-request root path and
on any configured method, and the gateway keeps only the exact decision path
* fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding
* test(eu_jev): annotate new test variables with Final
* fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches
Swap an updated pass-through's routes with no await between removal and
re-registration, so the built-in eu/jev route never sees the path as
unconfigured mid-update. The built-in eu/jev handler no longer takes a
configured route's target, and the built-in route also yields on a
method-only match, so GET and PUT on the no-slash form of a slash route
redirect again.
* test(pass-through): annotate the route-swap test's locals as Final
* fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker
* fix(eu_jev): leave a null model out of the upstream decision request
A JSON model of null is limited and counted as eu-jeff-latest, so send the
request upstream without a model field, the way an omitted model goes, since
eu/jev answers a null model with 400 Unknown model
* fix(pass-through): keep route checks on a path another auth endpoint still serves after an update
Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one
* test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it
* refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* test(proxy): cover gzip and zstd request bodies on /v1/systemone
* fix(proxy): decode zstd and gzip request bodies before auth reads them
* fix(proxy): bound request body decompression and keep it inside CORS
Limit the decoded size by an expansion ratio and the operator's max_request_size_mb, pass bodies without the coding's magic number through, answer 415 when zstd is unavailable, and register the middleware innermost so errors carry CORS and security headers
* fix(proxy): budget decoded request bodies per worker and forward plugin bodies as sent
Each worker holds at most MAX_DECOMPRESSED_REQUEST_BODIES_IN_FLIGHT_MB of
decoded bodies and answers 503 past it. /plugin-proxy/ bodies reach the
plugin as sent, a zstd skippable frame counts as zstd, the decoded-size 413
names the decoded limit, and an integer max_request_size_mb stays an integer
in the 413 text.
* fix(proxy): read the magic number before size limits and reserve declared decoded sizes
* test(proxy): pin gzip mtime so parametrized test ids match across xdist workers
* fix(proxy): parse max_request_size_mb the same way in the auth size check
* fix(proxy): pass Claude Code OTLP export bodies through undecoded and skip the size parse without premium
* test(proxy): cover client disconnects, chunked bodies past the cap, and body-less request messages in the decompression middleware
* test(proxy): name the decompression test cases so CI logs stay small
* fix(proxy): gzip responses over 16 KiB off the event loop
GZipBufferedResponseMiddleware compressed every body under 1 MiB inline.
gzip.compress holds the event loop for its whole run, and its cost follows
how well the body compresses: an 824 KB base64 embeddings response (100
inputs, the OpenAI SDK default encoding) takes about 165 ms, so every other
request on the worker waits that long, and with several in flight the
compressions run back to back.
Lower OFF_LOOP_SIZE_BYTES to 16 KiB so only bodies that compress in well
under a millisecond stay on the loop. zlib releases the GIL, so the loop
keeps serving while a worker thread compresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): pin the gzip off-loop boundary at exactly OFF_LOOP_SIZE_BYTES
A body of exactly OFF_LOOP_SIZE_BYTES compresses in a worker thread, as the
`<` in _compress implies. Suggested in review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(proxy): pin an embeddings-sized gzip body off the event loop and drop the source comment
The off-loop test was relative to OFF_LOOP_SIZE_BYTES, so it still passed with the old
1 MiB line. An 824 KB body, the size of the 100-input base64 embeddings response in the
issue, now has to compress in a worker thread. The recorder no longer mutates a list, and
the threshold rationale stays in the commit history instead of a source comment.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* perf(ci): distribute unit cases and reuse security sweep connections
* revert(ci): reject first speed trial after new failures
* perf(ci): avoid repeated test discovery and parsing
* perf(ci): isolate parallel security workers
* revert(ci): restore measured security execution isolation
* perf(tests): reuse source scans and deduplicate cache cleanup
* perf(ci): restore parallel security with safe log batching
* test(ci): isolate fixture state and schema comparisons
* test(ci): make schema and scheduler fixtures deterministic
* test(ci): drain queued logging before fixture teardown
* test(ci): preserve logging delivery without idle worker restarts
* test(ci): keep guardrail sinks available during parallel bursts
* test(ci): isolate full proxy lifecycle from gateway imports
* test(ci): preserve first failure responses and sweep evidence
* test(ci): restore documentation URL settings between cases
* test(ci): isolate open MCP tools in private gateways
* test(ci): group private MCP cases and handle exited workers
* test(ci): retain proxy exits and MCP failure details
* test(ci): repair merge interactions with main
Key the schema shard's full clone on artifact-name, since main removed the
unit-flag input. Give each owned Redis its own log file again. Hand the
realtime transcription tests main added a logging object whose dispatch
handlers are awaitable, so the per-test logging drain no longer waits on a
MagicMock task forever
* perf(ci): shard owned integration cases and close lifecycle leaks
* fix(ci): qualify shard execution and harden interrupted cleanup
* fix(ci): bound worker supervision and own fault traffic
* perf(ci): reuse unparametrized fixtures across bounded case groups
* fix(ci): preserve boot fault allowances and complete exit records
* test(ci): tolerate startup retries and retain readiness details
* perf(ci): balance integration shards by recorded duration
* fix(ci): load migration database helpers only for workers
Counts failed gateway requests by HTTP status at the ASGI edge
(LiteLLM_DailyGatewayFailedRequests) and rolls failures up per virtual key,
team, user and model group with the status the logging callbacks recorded
(LiteLLM_DailyRequestErrors, fed from the spend writer, flushed like the
gateway counters, directly or through Redis with a pod lease, and on
shutdown). GET /gateway/daily/activity gains by_status_code; the new
admin-only GET /gateway/errors/activity returns per-day totals with
per-status counts, failures by status code, and keys, teams, users and
model groups ranked by failures with their top status. Dashboard schema.d.ts
regenerated.
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): read the UTC clock once per operation in gateway tracking, PTU rollup and Mavvrik export
* refactor(proxy): default the injected clocks to get_utc_datetime instead of three private copies
---------
Co-authored-by: yuneng <yuneng@berri.ai>
* feat(proxy): gzip buffered responses for clients that accept it
Large JSON reads like /user/daily/activity/aggregated shipped tens of MB
uncompressed. Compress single-message bodies of 500B or more when the
client's Accept-Encoding allows gzip (q-values and the wildcard honored).
Streamed and etagged responses pass through untouched, every negotiable
response carries Vary: Accept-Encoding, and bodies of 1MB or more are
compressed in a worker thread
* fix(proxy): skip partial and no-transform responses in gzip and always release the held start
The gzip gate now also skips 206 Partial Content and Cache-Control: no-transform,
since compressing either breaks byte ranges or ignores an explicit ban on transforms.
A response start without a headers key no longer raises, and a start the app never
follows with a body message is forwarded when the app returns instead of being dropped.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* build: point the local proxy unit targets at the nested tests/unit/proxy tree
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
