* feat(pass-through): add eu/jev decision API pass-through with spend tracking
Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted
decision API that speaks the same POST /v1/systemone contract as TypeSafe.
It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to
https://jev.bevel.software), books spend through the shared decision logging
handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the
cost map
* fix(pass-through): default eu_jev to its canonical eu-jev.com origin
* fix(pass-through): reject custom_body on the eu_jev route
A key limited to other models could send an eu/jev request through
custom_body, because auth checks the outer body while the shared
pass-through forwards custom_body upstream. The route now answers 400
before anything is forwarded, the same way validate_oss_request guards
the laya and bespoke routes
* fix(pass-through): give the eu_jev route request and 200 response schemas
The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and
OA003. The route now documents the eu/jev System One request body and
its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI
document. The lazy OpenAPI snapshot and the dashboard API types are
regenerated with python -m litellm.proxy._lazy_openapi_snapshot and
npm run gen:api
* fix(pass-through): reject streaming on the eu_jev route
A body with stream=true switched the route to streaming, which skips the
decision spend tracking. Reject it with a 400, as validate_oss_request does
for laya and bespoke.
* fix(pass-through): type the eu_jev request body like the other decision routes
* fix(pass-through): check eu_jev decisions against the model eu/jev runs
Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted
model and the legacy names to eu-jeff-latest for key model checks and pricing,
check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when
EU_JEV_API_KEY is unset
* fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases
Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so
model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and
upstream still gets the client's exact body. eu_jev is a registered decisions
provider, so access groups, config deployments, /v1/systemone and /v1/decisions
accept it, and the Admin UI can add it. A release without its own cost map row
is priced as eu-jeff-latest instead of $0
* fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models
* fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it
* fix(eu_jev): take pass-through credentials only from proxy-wide deployments
The eu/jev route now ignores team-scoped deployments when it picks the
upstream key and base, so one team's deployment can no longer redirect
another team's calls, and a base ending in /v1 is not doubled
* fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it
Scope the mapped pass-through and gateway allowlist entries to the decision
route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep
their auth, metering, and gateway behavior from before this PR
* fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it
Drop the decision path from the mapped pass-through list and classify it with one
check that asks the pass-through registry, so an operator's own pass-through at that
path keeps its auth, metering, LLM-route status and litellm_user_api_key handling
from before this PR. The built-in route now yields on a per-request root path and
on any configured method, and the gateway keeps only the exact decision path
* fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding
* test(eu_jev): annotate new test variables with Final
* fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches
Swap an updated pass-through's routes with no await between removal and
re-registration, so the built-in eu/jev route never sees the path as
unconfigured mid-update. The built-in eu/jev handler no longer takes a
configured route's target, and the built-in route also yields on a
method-only match, so GET and PUT on the no-slash form of a slash route
redirect again.
* test(pass-through): annotate the route-swap test's locals as Final
* fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker
* fix(eu_jev): leave a null model out of the upstream decision request
A JSON model of null is limited and counted as eu-jeff-latest, so send the
request upstream without a model field, the way an omitted model goes, since
eu/jev answers a null model with 400 Unknown model
* fix(pass-through): keep route checks on a path another auth endpoint still serves after an update
Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one
* test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it
* refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
* feat(proxy): log one line per token count call
Each count call on /v1/messages/count_tokens, /utils/token_counter, /v1/responses/input_tokens and the Gemini countTokens routes now logs one INFO line with the litellm call id, the requested model, the model the count ran against, the count, the count source and the status code, so it joins the inference call that sends the same x-litellm-call-id
* test(google_endpoints): give the count test's fake request a headers list
* fix(proxy): keep token count log lines one per call and trusted in JSON logs, and log Bedrock native counts
Quote log values that carry whitespace, quotes, equals, or control characters so a model name can't split or forge a line. Let JsonFormatter record extras win over a dict parsed from the message. Log the Bedrock native count-tokens route through the same line.
* fix(proxy): escape braces in token count log values, log the Bedrock model AWS counts, and skip the line when there is nothing to count
* fix(proxy): log the deployment id the count routed to on every token count line
* fix(proxy): log count requests rejected before counting with the status they answer
A count route that rejects a request before the shared count function runs now logs the same token count line with its 400. An unsupported content block on the Messages count route becomes its 400 inside the logged wrapper, so its line says 400 instead of 500. A count that ends in a BaseException other than cancellation logs a 500.
* fix(proxy): compute the Bedrock counted model id only after a successful count
* revert(logging): keep the JSON log formatter's field order as on main
The count line already escapes braces in its values, so the formatter never parses a dict from it. Changing the order for every logger is a separate decision.
* perf(ci): distribute unit cases and reuse security sweep connections
* revert(ci): reject first speed trial after new failures
* perf(ci): avoid repeated test discovery and parsing
* perf(ci): isolate parallel security workers
* revert(ci): restore measured security execution isolation
* perf(tests): reuse source scans and deduplicate cache cleanup
* perf(ci): restore parallel security with safe log batching
* test(ci): isolate fixture state and schema comparisons
* test(ci): make schema and scheduler fixtures deterministic
* test(ci): drain queued logging before fixture teardown
* test(ci): preserve logging delivery without idle worker restarts
* test(ci): keep guardrail sinks available during parallel bursts
* test(ci): isolate full proxy lifecycle from gateway imports
* test(ci): preserve first failure responses and sweep evidence
* test(ci): restore documentation URL settings between cases
* test(ci): isolate open MCP tools in private gateways
* test(ci): group private MCP cases and handle exited workers
* test(ci): retain proxy exits and MCP failure details
* test(ci): repair merge interactions with main
Key the schema shard's full clone on artifact-name, since main removed the
unit-flag input. Give each owned Redis its own log file again. Hand the
realtime transcription tests main added a logging object whose dispatch
handlers are awaitable, so the per-test logging drain no longer waits on a
MagicMock task forever
* perf(ci): shard owned integration cases and close lifecycle leaks
* fix(ci): qualify shard execution and harden interrupted cleanup
* fix(ci): bound worker supervision and own fault traffic
* perf(ci): reuse unparametrized fixtures across bounded case groups
* fix(ci): preserve boot fault allowances and complete exit records
* test(ci): tolerate startup retries and retain readiness details
* perf(ci): balance integration shards by recorded duration
* fix(ci): load migration database helpers only for workers
* fix(proxy): stop queued registry read-throughs spending the resync budget
RegistryReadThrough.attempt serializes misses behind one lock, but every
request that was queued behind the first one still spent a unit of the
20-per-5s resync budget and re-ran the DB resync, even though the first
request had already loaded the object. A burst of more than 20 requests
for a model created on another worker therefore exhausted the budget and
the rest got 400 "Invalid model name".
attempt now checks whether the key is already loaded once it holds the
lock and returns early without touching the budget. Models check the
router's model names and deployment ids, guardrails and agents reuse
their existing registry lookups.
* test(proxy): gate the queued read-through test on events and cover each registry's loaded check
The queued-requests test now holds the first resync on an asyncio.Event instead of
a timed sleep and records calls in a recorder with tuple and frozenset state. New
tests show the model, guardrail and agent read-throughs each answer an object that
is already loaded without reading the database, so rewiring any registry's loaded
check now fails a test
* test(proxy): keep the queued read-through recorder inside its test and type the agent registry fixture
* test(proxy): move proxy_server, _experimental and db tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep tuple identity in proxy state restore and fix misc target paths
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move utils, agent_endpoints and endpoint tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): package moved unit test directories
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): exclude proxy-db-owned files from the misc target
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): drop the redundant fixture docstrings in the proxy conftest
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): move auth, hooks, policy_engine and client tests into tests/unit/proxy
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): stub HIBP through respx by disabling the aiohttp transport
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): share the httpx transport fixture across proxy unit tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): restore proxy globals without a missing-value sentinel
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): package moved dirs and stub the login breach check at the HTTP boundary
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): isolate the mcp server manager per test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yuneng <yuneng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: fix the litellm-tests unit job with sysmon coverage, an env allowlist and coverage upload on failure
* test: replace key-dependent proxy, enterprise and mcp unit tests with synthetic values and integration and e2e coverage
* test: drop key reads at the legacy proxy, enterprise and mcp paths and wire the gemini pass-through split
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move caching, proxy-extras, gateway and enterprise tests into tests/unit and run them from litellm-tests under their legacy flags
* ci: move tests/proxy_unit_tests to tests/unit/proxy and run the proxy-db shards from litellm-tests
* ci: fail the unit shard when circleci tests split errors
* test: drop restating comments from the gemini pass-through split
* build: point the local proxy unit targets at the nested tests/unit/proxy tree
* ci: exit the unit shard cleanly when circleci tests split assigns it no files
---------
Co-authored-by: yuneng <yuneng@berri.ai>
