Without the auto_router feature in the signed enterprise license a proxy may hold
one complexity router with classifier_type heuristic_v2 across config.yaml and the
DB; with it the limit is lifted. The ceiling is derived once from LicenseCheck and
handed to the Router, which refuses the extra router at registration. config.yaml
over the limit refuses to start, and /model/new, /model/update and
PATCH /model/{id}/update refuse the write with a 403 before touching the DB.
Expiry follows the existing max_users/max_teams pattern: judged when the
license is verified, not on every call, and a verify that rejects the license
(expired or unreadable) leaves no signed payload behind. The rollback after a
failed upsert re-admits state that was already serving, so it is exempt from the
ceiling: an edit that fails, including one refused by a ceiling that has since
tightened, leaves the router serving its previous configuration.
A write that leaves a row on heuristic_v2 under a limited license runs in one
transaction that takes a Postgres advisory lock before counting the DB rows plus
this proxy's config.yaml routers, so concurrent writes on any pod cannot both
claim the sole slot and no surplus row is ever persisted.
Only the row insert runs under that lock: the team model bookkeeping, which
needs a second pool connection, runs after the transaction has committed.
PATCH /model/{id}/update follows the same order as create: the row is written
through the slot first and the team's model list is updated only afterwards, so
a refused write leaves the team as it was.
The slot transaction bypasses the repository's publish-on-write, so it
publishes the config change once after commit, as delete_team_models does.
Adds a configurable password-strength policy (default: min 12 chars,
upper/lower/number/special, all individually toggleable, floored at 8
so a misconfigured minimum cannot disable the length check, and
unicode-aware so an accented letter cannot satisfy the special-
character requirement) enforced on every path that sets a local
user's password: /user/update, /user/bulk_update, and the invitation
onboarding claim flow.
Adds general_settings.disable_password_login_when_sso_enabled, which
rejects username/password login on /login, /v2/login and /v3/login
(including the UI_USERNAME/UI_PASSWORD admin fallback) once ANY
configured SSO provider is FULLY ready: every companion secret/
endpoint an OAuth provider needs, checked independently per provider
so a stray leftover client id for an unused provider can't mask a
different, fully configured one; and for SAML, the optional
python3-saml runtime being importable, checked without letting a
fully-missing package's ModuleNotFoundError take down password login
itself. SSO becomes the enforced boundary for interactive UI access
without an incomplete, mixed, or half-installed SSO setup locking
every admin out or breaking login outright. Master-key API access is
untouched, and unsetting the setting plus a restart restores password
login as the documented recovery path.
Resolving github_copilot/chatgpt names through get_llm_provider runs the
provider's OAuth device flow synchronously on the event loop. Adopt the
declared provider in PatternMatchRouter.get_pattern, which the auth
layer's zero-cost budget check walks on every request against wildcard
routers, and in /utils/supported_openai_params.
Direct access on /model/info was read from the user record alone, so an
unrestricted user calling with a key limited to a few models saw every non-team
deployment, including ones the key gets a 403 on. Resolve the key's grant the
same way and intersect the two.
Resolving a grant now also expands access groups, which the key path needs and
the user path was missing.
Claude-Session: https://claude.ai/code/session_01XL7LBFEew4wi8gphVCDq6n
Team-scoped deployments keep the internal model_name_{team_id}_{uuid} routing key and expose the public name in model_info.team_public_model_name. The dashboard links team model chips with the public name, so the exact filter now matches either name via the existing helper.
The registry key was never copied into ModelInfo, so /v1/model/info reported
null for every model, /model_group/info reported false for every group, and
litellm.supports_parallel_function_calling() returned False for provider-prefixed
entries that declare true. Copy it like every other capability flag and pin the
three surfaces with regression tests.
Resolves LIT-6340
A complexity-router setting placed beside complexity_router_config, or inside a
tier entry's litellm_params, is read by nobody: the router loads its settings only
from litellm_params.complexity_router_config. It does not stay inert. The
alias-marker forwarding and the per-tier param spread carry every unrecognized key
onto the outbound request, and all_litellm_params only knows the outer names, so
the key reaches the provider as an unknown body field and every call through that
model group fails with an error naming an internal config key.
Guard the whole set, derived from ComplexityRouterConfig.model_fields so a field
added later is covered, and scoped to complexity-router deployments because the
names only mean this there (embedding_model is a legitimate flat param on an
s3_vectors vector store). Scope is read from the same merged field view the naming
check is judged on, so a router named only by its default model is in scope and a
field added to the required-field table is covered without another edit. The write
endpoints reject with a 400 naming the keys and where they belong, config.yaml
refuses to start for the same reason max_agentic_loops does, and a tier entry is
judged by the config model itself.
An already-stored deployment keeps loading, so an upgrade cannot take a running
gateway down over a row that was written before the gate existed.
* feat(health): opt-in model-group allowlist for background health checks and health-check routing
* fix(health): merge shared health states per writer scope instead of replacing
* refactor(health): drop restating comment and parameterize test scope annotations
* chore: remove stray generated prisma migration file
* fix(health): merge health states against the Redis snapshot, not the pod-local copy
* fix(health): fall back to the pod-local snapshot when the Redis read returns nothing
reload_search_tools_from_db is a read-modify-write of the shared llm_router
global: it reads the whole table, merges the config tools in, and replaces
router.search_tools wholesale. Two of those interleaving lets the older
snapshot's assignment land last and put back a tool the newer one deleted, so a
revoked tool keeps serving on the provider key it carried until the next reload.
Take MODEL_RECONCILE_LOCK, which add_deployment already uses to serialize the
same shape of work on the same global. It has to go on this entry point rather
than in _init_search_tools_in_db, because _init_non_llm_objects_in_db calls that
while already holding the lock and asyncio.Lock is not reentrant.
A separate search-tools-only lock would not close the race: the periodic
reconcile reaches _init_search_tools_in_db under MODEL_RECONCILE_LOCK, so only
that same lock orders an endpoint refresh against a cron tick.
Ordering across workers is unchanged and still reconciles on the next tick.
Creating a search tool through the UI only wrote the row; the router was updated
solely by the add_deployment job, so the tool was unusable for up to
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS (30s by default) even on the worker that
served the write. Tools declared in config.yaml load straight into the router at
startup, which is why they never showed the delay.
The create, update and delete endpoints now refresh the router inline, matching
what the MCP server endpoints already do. The refresh is best-effort: the row is
already committed, so a failure must not surface as a 500 and push the caller
into a retry that creates duplicates.
Two related gaps go with it. _init_search_tools_in_db skipped the router update
whenever the merged list came back empty, so deleting the last search tool left
it live in memory forever. And in store_model_in_db-off deployments the
add_deployment job is never scheduled, so DB-backed search tools never reached
the router at all; that branch now loads them at startup and keeps them fresh on
its own interval, the same way MCP servers already do.
Adds `general_settings.model_list_healthy_only`, which makes `/models`,
`/v1/models/{id}` and `/model/info` hide models whose backing deployments are
all marked unhealthy by background health checks, for every caller, without
each client having to pass `healthy_only=true`. `/model/info` also gains the
per-request `healthy_only` parameter that `/v1/models` already had.
Everything here is opt-in. With the setting absent, the endpoints take the same
code path they do today and no health lookup runs at all.
The listing filter reads the deployment health cache, which until now was only
populated when `enable_health_check_routing` was on, so `healthy_only=true`
silently did nothing in a plain `background_health_checks` setup. The setting
now also keeps that cache filled. That is a pure write: every routing-time
reader is itself gated on `enable_health_check_routing`, and the cooldown and
failure bookkeeping stays behind that flag, so routing is untouched.
Filtering stays presentation-only and fails open. A hidden model is still
callable, and missing, stale or empty health state hides nothing.
The ceiling used to go through `int(... or 3)`, so anything `int()` accepted
worked. Tightening the new shared validator to `isinstance(int)` turned a
config that boots today into a proxy that refuses to start, because
`max_agentic_loops: os.environ/MAX_AGENTIC_LOOPS` is resolved to a string
before it reaches either check, and a YAML-quoted "5" is a string too.
Accept ints, integral floats, and strings that parse to a whole number. Keep
refusing bools, fractional floats, words, and anything below 1.
The ceiling was only checked at the feature level, on
litellm_settings.websearch_interception_params. The per-deployment
litellm_params.max_agentic_loops, which wins over it, went straight into
int(kwargs.get("max_agentic_loops", 3) or 3), so a 0 was swallowed by the
falsy fallback and read as the default 3. Asking for the tightest ceiling
handed you the loosest one. A non-integer booted the proxy and then failed
every request to that model with "invalid literal for int() with base 10".
Both settings now share one validator, which names the field it rejected,
and the per-deployment value is checked while the model list is read at
startup so a bad value stops the proxy rather than surfacing per request.
The check sits in load_config rather than on LiteLLM_Params because the
proxy builds its router with ignore_invalid_deployments=True, where a
validation error drops the deployment silently instead of refusing to
start. This is the same placement the complexity_router_config plugin
check already uses.
Chat completions read the same key through a separate path that turned 0
into 1 and true into a ceiling of 1, so it now shares the validator too
and the key means one thing on both surfaces.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* test(lint): ban blind pytest.raises(Exception) with ruff B017
A bare pytest.raises(Exception) accepts whatever the body throws. The TypeError
a refactor introduces satisfies it exactly as well as the rejection the test was
written for, so the crash reads as a pass and the test never goes red.
All 111 existing sites are narrowed here. A runtime probe recorded the concrete
exception each one actually catches, and each site now names that type. Where
the code under test genuinely raises a bare Exception, the site pins a stable
slice of the message with match= instead.
Two sites tell on themselves. The shared responses-API cancel test raises
"custom_llm_provider is required but passed as None" rather than talking to a
provider at all, because cancel_responses takes a provider, not a model. And
test_bedrock_guardrails_with_streaming was the only test in its file still
passing without AWS credentials, because the NoCredentialsError boto3 raised
long before the guardrail ran satisfied the blind raises.
* fix(test): widen the openai batch-dispatch assertion to OpenAIError
The narrowed NotFoundError only holds where OPENAI_API_KEY is set. Without one
the SDK raises OpenAIError while building the client, long before any 404, so CI
went red. OpenAIError covers both and still rejects a TypeError from a refactor.
tiktoken's BPE merge loop is quadratic in the length of a single regex piece, so a long
run of one repeated character turns a multi-MB payload into minutes of CPU. Encoding in
bounded chunks makes that linear, at a drift of at most ~1 token per chunk boundary.
Chunking alone only makes the stall shorter, so the async paths now count in a worker
thread: tiktoken releases the GIL for its Rust encode, so the loop keeps serving other
requests while a count is in flight. The /utils/token_counter endpoint awaits the new
atoken_counter, and the router's async deployment selection counts off-loop and hands
the result to _pre_call_checks instead of making it count inline.
The chunk size knob is bounded to [1, 4096]: a non-positive value used to raise or
silently report zero tokens, and an arbitrarily large one restored the quadratic cost
this exists to remove. Out-of-range and unparseable values warn and fall back to 1024.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* feat(ui): let admins supply a dark-mode variant of their custom logo
A deployment branded through UI_LOGO_PATH got its light artwork on the
dark sidebar, and there was nothing an admin could set to change that.
Adds UI_LOGO_PATH_DARK, exposed as the logo_url_dark theme setting and a
second field on the UI theme page. /get_image now walks an ordered list
of candidates for the requested theme and serves the first usable one:
the dark logo, then the light logo, then the bundled default.
Falling through rather than failing is the point. An admin who never
sets a dark logo keeps their own light one instead of reverting to
LiteLLM's, and a dark logo that goes missing later degrades to their
light logo rather than dropping their branding entirely.
* fix(ui): recover from a dark logo the browser cannot load
A dark logo given as an http(s) URL is loaded by the browser straight
from the sidebar, so it never passes through the proxy's fallback chain.
A URL that 404s left a broken image where the admin's light logo should
have been, while the same logo given as a local path fell back cleanly.
The sidebar now remembers the dark URL that failed and drops to the light
logo, matching how the proxy resolves an unusable dark logo and how the
provider Logo component already handles a broken image.
The bundled logo is a JPEG, so it carries no alpha and its white
background renders as a bright slab against a dark sidebar. Making it
transparent alone would not be enough either: the wordmark is near-black
and would disappear on dark.
Adds logo_dark.png, derived from the light logo. The sky-blue disc and
train are kept as they are behind a circular alpha mask, and the
wordmark's antialiasing is un-flattened from white into straight alpha
and repainted in the dark theme's own foreground colour. Both files are
1000x257, so swapping between them cannot shift the sidebar header.
/get_image gains a theme query param. The default response is byte for
byte what it was, and a logo configured through UI_LOGO_PATH is served
unchanged in both themes, since custom logos have no dark variant yet.
`ProxyConfig.get_config()` walked the parsed config and replaced every
`os.environ/<KEY>` string with `get_secret(value)` before anything initialized
the secret manager, so a key held only by the manager resolved to `None` and
that `None` was written back into the config. The later fallback in
`load_config` could not recover it, because the key now existed with a `None`
value.
Hoist the initialization into `get_config()`, ahead of the resolution pass, so
every entrypoint gets it: the CLI already did this itself, but the microservice
entrypoints (`gateway/main.py`, `backend/main.py`) uvicorn the app directly and
bypass the CLI. `load_config`'s own call is now redundant and is dropped, so
startup builds the manager once instead of building one and discarding it.
`get_config()` also runs on management-endpoint request paths, so this returns
early once a manager exists rather than rebuilding the client per request.
Also warn when a reference the manager would have been asked for resolves to
`None`. The reporter had no log line at all to work from. `get_secret` only
reaches the manager when reads are enabled and the name is in `hosted_keys`, so
`secret_manager_would_be_consulted` mirrors that gate and keeps the warning off
env-only references, which are expected rather than an error.
Both surfaces wrote zero bytes for the whole time-to-first-token, so an
intermediary with an idle read timeout drops a healthy connection before the
first token. They reached neither keepalive engine, which is what #37322 left
open.
The streaming assistants run spends that wait inside the awaited call that
produces its response, since create_response buffers the first chunk, so it
takes the same open_sse_before_first_byte seam the native routes use. The A2A
route only contacts the upstream agent once its body iterator is first pulled,
so nothing is awaited before the response exists and the gap has to be filled
from inside the stream instead; wrap_sse_stream_with_keepalive_pings already
does that and now takes the filler as a parameter, so A2A gets an SSE comment
its JSON-RPC clients discard rather than Anthropic's ping event.
Off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds.
* feat(complexity_router): custom classifier plugins via classifier_type 'plugin'
Adds a third classification mode where an operator-supplied hook decides the
tier instead of the heuristic scorer or the LLM classifier. The hook implements
an async classify(context) returning a tier name (built-in value, tier_labels
label, or tier_definitions name) or None to decline; failures, timeouts, and
unknown tiers fall back exactly like a failed LLM classifier. The context
carries the request messages and metadata, including caller identity, so a
plugin can route by team, spend, or any business rule.
The plugin resolves from a dotted path at proxy startup with a load-time check
that classify is a coroutine function, and is closed off over HTTP like the
routing plugins list. Routing decisions record the new classifier_plugin cause.
tier_definitions now accepts classifier_type 'plugin' alongside 'llm'.
* fix(proxy): resolve plugin dotted paths in _delete_deployment before hashing ids
The db-sync reconcile re-reads the raw config and hashes litellm_params to
compute which ids the config wants served, but the router's ids were hashed
from the resolved params where plugin dotted paths are live instances. The
mismatched ids made the reconcile evict every plugin-bearing auto-router one
sync after startup, on any proxy with a database connected. This also affected
the existing routing plugins list, not just the new classifier plugin.
Resolving the plugins in _delete_deployment the same way load_config does makes
both sides hash the same canonical form. A plugin module broken on disk at
reconcile time skips cleanup instead of evicting valid deployments, matching
how a get_config failure is handled
* fix(complexity_router): treat non-string plugin verdicts as declines, centralize the empty-mapping sentinel
A hook returning a non-string raised inside resolve_classified_tier outside the
plugin exception boundary, failing the request instead of falling back. Also
moves the read-only empty mapping to constants.py per repo convention and moves
the classifier plugin product docs out of the package README for the docs repo
* refactor(complexity_router): rename the plugin classifier mode to classifier_type 'custom'
The mode value now names the operator's intent while classifier_plugin keeps
naming the mechanism; routing decisions keep the classifier_plugin cause
* refactor(proxy): pin plugin-bearing deployment ids from the raw params instead of resolving in the reconcile
Replaces the previous approach of re-running plugin resolution inside
_delete_deployment, which imported operator modules on every reconcile cycle
and skipped the whole cleanup pass when any one module was broken on disk.
load_config now stamps model_info.id from the raw litellm_params before
resolution swaps dotted paths for live instances, so the reconcile's raw-config
hash matches by construction and needs no resolution at all: a broken module
cannot stall cleanup for unrelated models, and any future param-transforming
resolution is covered by the same pin. _generate_model_id becomes a staticmethod
so the pin can run before the Router exists; its statically dead non-string key
branches are removed. Also documents candidate_models as an informational
snapshot for classifier plugins, unlike the narrowing surface RoutingPlugin
filters
* fix(router): restore _generate_model_id key handling, align classifier context with the routing-plugin pattern
The staticmethod conversion accidentally dropped the non-string-key branches
from _generate_model_id, a silent hash change for any params with non-string
keys; they are restored verbatim. The classifier plugin context now follows
the Router-level routing-plugin recipe exactly: structured messages come from
resolve_structured_messages over the raw messages, and the metadata key comes
from the shared get_metadata_variable_name_from_kwargs helper, which also
replaces the duplicated inline sniff in _pick_model_for_tier. This removes the
raw-or-resolved fallback where a plugin could silently receive resolved
messages when a call site forgot to pass the raw ones
* refactor(router): make generate_model_id public, guard classifier context construction
Two modules legitimately hash deployment ids with the same helper now (Router
and the proxy's config-load pin), so the private name was lying about its
audience and the cross-module call needed a pyright suppression; renaming it
public restores the static safety net. The classifier plugin's RoutingContext
construction moves inside the failure boundary, matching the LLM path where
litellm-side prompt building also falls back rather than failing the request,
and a prompt-only call with no message list is now covered by a test
The Global Control Plane (formerly documented as the HA Control Plane) is
documented as an Enterprise feature, but `worker_registry` carried no premium
check, so any OSS install could run one. Gate it at config load, matching the
`enforced_params` precedent, and fail startup rather than ignoring the registry
silently: a silently dropped registry degrades a control plane into an ordinary
proxy with no signal to the operator.
Also declare `worker_registry` and `general_settings.control_plane_url`, both
load bearing today and neither previously declared, so they appear in the
generated config schema.
Anthropic's Models API declares max_input_tokens and max_tokens as nullable, not
optional, and the live vendor endpoint returns both keys on every entry. The
merged Anthropic-native listing dropped either key whenever LiteLLM could not
resolve a limit, so a client validating against a nullable-but-required schema
saw a malformed entry for any model the cost map does not know.
* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery
* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils
* fix(proxy): make model_list request param optional for direct callers
* style: apply ruff format to changed lines
* style: satisfy ruff strict-rule budget (UP006, I001)
* style: satisfy type-discipline budget (LIT002 mutable-ok, LIT009 pyright ignore)
* style: satisfy LIT001/LIT010 and drop explanatory comment per contributor rules
* fix(proxy): translate team model names in the Anthropic /v1/models response
* ci: trigger buildkite status report
* feat(proxy): carry token limits into the Anthropic-native /v1/models entries
* fix(proxy): cast the injected request so the anthropic-version guard is a real comparison
* fix(proxy): explain the model listing casts so the type-discipline gate passes
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
