mirror of
https://github.com/BerriAI/litellm.git
synced 2026-11-03 07:23:33 +00:00
60601ca995
* feat(pass-through): add eu/jev decision API pass-through with spend tracking
Adds a /eu_jev/{endpoint:path} pass-through route for eu/jev, an EU-hosted
decision API that speaks the same POST /v1/systemone contract as TypeSafe.
It reads EU_JEV_API_KEY and EU_JEV_API_BASE (defaults to
https://jev.bevel.software), books spend through the shared decision logging
handler, and prices eu_jev/eu-jeff-1.0.0 and eu_jev/eu-jeff-latest from the
cost map
* fix(pass-through): default eu_jev to its canonical eu-jev.com origin
* fix(pass-through): reject custom_body on the eu_jev route
A key limited to other models could send an eu/jev request through
custom_body, because auth checks the outer body while the shared
pass-through forwards custom_body upstream. The route now answers 400
before anything is forwarded, the same way validate_oss_request guards
the laya and bespoke routes
* fix(pass-through): give the eu_jev route request and 200 response schemas
The OpenAPI docs gate flagged POST /eu_jev/{endpoint} for OA001 and
OA003. The route now documents the eu/jev System One request body and
its 200 answer, usage and meta shape, taken from the eu/jev OpenAPI
document. The lazy OpenAPI snapshot and the dashboard API types are
regenerated with python -m litellm.proxy._lazy_openapi_snapshot and
npm run gen:api
* fix(pass-through): reject streaming on the eu_jev route
A body with stream=true switched the route to streaming, which skips the
decision spend tracking. Reject it with a 400, as validate_oss_request does
for laya and bespoke.
* fix(pass-through): type the eu_jev request body like the other decision routes
* fix(pass-through): check eu_jev decisions against the model eu/jev runs
Serve only POST /eu_jev/v1/systemone, eu/jev's one endpoint, resolve an omitted
model and the legacy names to eu-jeff-latest for key model checks and pricing,
check scopes under eu_jev/<model> like Laya and Bespoke, and answer 503 when
EU_JEV_API_KEY is unset
* fix(pass-through): apply eu_jev key limits, register eu_jev, price unlisted releases
Pre-call hooks and guardrails now see the body model as eu_jev/<release>, so
model_rpm_limit and model_tpm_limit set on eu_jev/eu-jeff-latest apply, and
upstream still gets the client's exact body. eu_jev is a registered decisions
provider, so access groups, config deployments, /v1/systemone and /v1/decisions
accept it, and the Admin UI can add it. A release without its own cost map row
is priced as eu-jeff-latest instead of $0
* fix(eu_jev): use the pass-through deployment's api_base and accept provider-prefixed models
* fix(eu_jev): declare /v1/systemone on the eu/jev cost map entries so the generated endpoint table keeps it
* fix(eu_jev): take pass-through credentials only from proxy-wide deployments
The eu/jev route now ignores team-scoped deployments when it picks the
upstream key and base, so one team's deployment can no longer redirect
another team's calls, and a base ending in /v1 is not doubled
* fix(eu_jev): let a configured pass-through at /eu_jev/v1/systemone keep serving it
Scope the mapped pass-through and gateway allowlist entries to the decision
route instead of the whole /eu_jev prefix, so custom routes under /eu_jev keep
their auth, metering, and gateway behavior from before this PR
* fix(eu_jev): treat /eu_jev/v1/systemone as a provider route only while the built-in route serves it
Drop the decision path from the mapped pass-through list and classify it with one
check that asks the pass-through registry, so an operator's own pass-through at that
path keeps its auth, metering, LLM-route status and litellm_user_api_key handling
from before this PR. The built-in route now yields on a per-request root path and
on any configured method, and the gateway keeps only the exact decision path
* fix(eu_jev): find a configured route at the decision path with or without a trailing slash, and stop a deleted one forwarding
* test(eu_jev): annotate new test variables with Final
* fix(proxy): keep eu/jev pass-through routing stable during updates and method mismatches
Swap an updated pass-through's routes with no await between removal and
re-registration, so the built-in eu/jev route never sees the path as
unconfigured mid-update. The built-in eu/jev handler no longer takes a
configured route's target, and the built-in route also yields on a
method-only match, so GET and PUT on the no-slash form of a slash route
redirect again.
* test(pass-through): annotate the route-swap test's locals as Final
* fix(pass-through): keep an updated auth route's LLM API route checks on the updating worker
* fix(eu_jev): leave a null model out of the upstream decision request
A JSON model of null is limited and counted as eu-jeff-latest, so send the
request upstream without a model field, the way an omitted model goes, since
eu/jev answers a null model with 400 Unknown model
* fix(pass-through): keep route checks on a path another auth endpoint still serves after an update
Forward a non-string eu/jev model unchanged so eu/jev rejects it, and drop only a null or missing one
* test(eu_jev): keep the non-string model test in the eu/jev route test file so a CI shard runs it
* refactor(eu_jev): build the scoped pass-through body as a new dict instead of updating it in place
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
153 lines
3.4 KiB
Python
153 lines
3.4 KiB
Python
"""Path allowlist for the gateway component.
|
|
|
|
The gateway exposes the LLM data-plane surface: chat/completions, embeddings,
|
|
audio, batches, files, fine-tuning, rerank, decisions, extraction, ocr, rag, video, search, image,
|
|
responses, vector stores, passthrough providers, realtime websockets, MCP
|
|
tool-call endpoints, and operational endpoints (/health, /metrics, and the
|
|
/debug/memory/summary read of the serving worker's RSS).
|
|
|
|
Any path not listed here is dropped from the gateway process so management/UI
|
|
endpoints don't ride on the same pods.
|
|
|
|
Versioned data-plane paths are enumerated explicitly rather than allowing a
|
|
blanket `/v1/` or `/v2/` prefix — those broad prefixes would otherwise also
|
|
match management routes like `/v1/access_group`, `/v1/tool/{tool_name}/logs`,
|
|
`/v2/key/info`, etc.
|
|
"""
|
|
|
|
GATEWAY_PATH_PREFIXES: tuple[str, ...] = (
|
|
# OpenAI-compatible data-plane surface (versioned + unversioned)
|
|
"/v1/chat/",
|
|
"/chat/",
|
|
"/v1/completions",
|
|
"/completions",
|
|
"/v1/embeddings",
|
|
"/embeddings",
|
|
"/v1/moderations",
|
|
"/moderations",
|
|
"/v1/audio/",
|
|
"/audio/",
|
|
"/v1/images/",
|
|
"/images/",
|
|
"/v1/files",
|
|
"/files",
|
|
"/v1/batches",
|
|
"/batches",
|
|
"/v1/fine_tuning/",
|
|
"/fine_tuning/",
|
|
"/v1/fine-tuning/",
|
|
"/fine-tuning/",
|
|
"/v1/responses",
|
|
"/responses",
|
|
"/v1/threads",
|
|
"/threads",
|
|
"/v1/assistants",
|
|
"/assistants",
|
|
"/v1/vector_stores",
|
|
"/vector_stores",
|
|
"/v1/indexes",
|
|
"/v1/models",
|
|
"/models",
|
|
"/openai/",
|
|
"/engines/",
|
|
# Anthropic / agentic data-plane surface
|
|
"/v1/messages",
|
|
"/messages",
|
|
"/v1/skills",
|
|
"/v1/a2a/",
|
|
"/a2a/",
|
|
# LiteLLM-native LLM surface
|
|
"/v1/rerank",
|
|
"/v2/rerank",
|
|
"/rerank",
|
|
"/v1/decisions",
|
|
"/decisions",
|
|
"/v1/systemone",
|
|
"/systemone",
|
|
"/v1/extraction",
|
|
"/extraction",
|
|
"/v1/ocr",
|
|
"/ocr",
|
|
"/v1/rag/",
|
|
"/rag/",
|
|
"/v1/video",
|
|
"/v1/videos",
|
|
"/video/",
|
|
"/videos",
|
|
"/v1/search",
|
|
"/search",
|
|
"/v1/containers",
|
|
"/containers",
|
|
"/v1/evals",
|
|
"/v1/traces",
|
|
"/v1/logs",
|
|
"/v1/memory",
|
|
"/queue/chat/",
|
|
# Google data plane (v1beta is the Google AI Studio version)
|
|
"/v1beta/",
|
|
"/interactions",
|
|
# Provider passthrough
|
|
"/anthropic/",
|
|
"/azure/",
|
|
"/azure_ai/",
|
|
"/azure_speech/",
|
|
"/aws/",
|
|
"/bedrock/",
|
|
"/comprehendmedical",
|
|
"/transcribe",
|
|
"/cohere/",
|
|
"/gemini/",
|
|
"/gigachat/",
|
|
"/google/",
|
|
"/vertex_ai/",
|
|
"/vertex-ai/",
|
|
"/assemblyai/",
|
|
"/eu.assemblyai/",
|
|
"/deepgram/",
|
|
"/fal_ai/",
|
|
"/langfuse/",
|
|
"/vllm/",
|
|
"/mistral/",
|
|
"/typesafe/",
|
|
"/openrouter/",
|
|
"/nvidia_nim/",
|
|
"/groq/",
|
|
"/voyage/",
|
|
"/cursor/",
|
|
"/milvus/",
|
|
"/openai_passthrough/",
|
|
"/tinyfish/",
|
|
# Dynamic provider / toolset passthrough (path templates)
|
|
"/{provider}/",
|
|
"/toolset/",
|
|
# Realtime / streaming
|
|
"/v1/realtime",
|
|
"/realtime",
|
|
"/v1/live/sessions",
|
|
"/live/sessions",
|
|
# Health & ops
|
|
"/health",
|
|
"/metrics",
|
|
"/watsonx",
|
|
)
|
|
|
|
GATEWAY_EXACT_PATHS: frozenset[str] = frozenset(
|
|
{
|
|
"/",
|
|
"/routes",
|
|
"/openapi.json",
|
|
"/docs",
|
|
"/docs/oauth2-redirect",
|
|
"/redoc",
|
|
"/test",
|
|
"/debug/memory/summary",
|
|
"/api/event_logging/batch",
|
|
"/eu_jev/v1/systemone",
|
|
}
|
|
)
|
|
|
|
GATEWAY_MOUNT_PATHS: frozenset[str] = frozenset(
|
|
{
|
|
"/metrics",
|
|
}
|
|
)
|