Commit graph

12648 commits

Author SHA1 Message Date
mateo-berri
6ca120a674 fix(proxy): coerce and validate the sse keepalive ping interval from config 2026-08-05 17:23:39 -07:00
Mateo Wang
eabcafc1df
perf(pre-commit): fetch basedpyright base counts from CI artifacts (#35970) 2026-08-05 17:21:16 -07:00
mateo-berri
1b6f3cebf1
fix(managed_files): log sanitized validation errors when skipping rows
The skip warning interpolated the full pydantic ValidationError, whose
string embeds input_value with the rejected row's contents. Managed-file
rows carry a caller-supplied filename, so a malformed row copied that
into operational logs.

Log the error locations, types, and messages via errors() with input,
url, and context excluded, keeping the field-level diagnostics without
the values. Non-validation failures fall back to the exception type.
2026-08-06 00:16:05 +00:00
mateo-berri
3d673f9534
fix(managed_files): skip unparseable rows when listing managed files
get_user_created_file_ids validated every row's file_object without a
guard, so a single row failing OpenAIFileObject validation raised
ValidationError and turned the whole GET /v1/files response into a 500.
#35365 covered the null case only, leaving malformed or partial rows
able to take the entire listing down.

Rows now parse through a helper that returns None on failure and logs a
warning, matching how list_user_batches already tolerates rows it cannot
parse, so one bad row costs its own entry instead of the caller's whole
listing. Null rows stay silent since the batch cost poller registers
those legitimately.

Refs #35361
2026-08-05 23:58:20 +00:00
Souravrajvi0
388943ac17
fix(proxy): register managed batch output files on terminal retrieve (#34092)
Terminal batch retrieve could return the raw provider output_file_id, which
skips managed-file ownership checks on /v1/files/{id}/content and lets any key
on the proxy download another user's batch output.

Retrieve now registers the missing managed-file row before responding, and
attributes ownership to the durable batch owner rather than the retrieving
caller, so output and error ids always come back as unified managed ids.

Fixes #33989
2026-08-05 16:38:49 -07:00
Akash Naickar
e6e18d406a
fix(model-prices): correct replicate model key typo (#34800) 2026-08-05 16:37:44 -07:00
tin-berri
55e666a05f
feat(complexity_router): report LLM classifier cost per request via routing_decision and x-litellm-classifier-cost header (#36015) 2026-08-05 16:27:32 -07:00
ryan-crabbe-berri
7e8d0d3130
fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion (#36010)
* fix: rebuild models_by_provider in add_known_models so cost map reloads reach wildcard expansion

* fix: refresh models_by_provider in place so captured references survive reloads
2026-08-05 16:24:43 -07:00
yuneng-jiang
c898d341c0
Merge pull request #36011 from BerriAI/litellm_maint_batch_2026_07
fix(proxy)!: apply request-parameter checks consistently across body, path and form inputs
2026-08-05 16:23:16 -07:00
mateo-berri
131339d8e5 fix(proxy): send keepalive pings on anthropic messages SSE streams during upstream silence 2026-08-05 16:15:04 -07:00
mateo-berri
e87b8a098a fix(managed_files): derive unified output file ids deterministically so concurrent registrations converge 2026-08-05 16:08:56 -07:00
Yuneng Jiang
298fb8ce56
fix(health): drop a stored-credential reference along with the credentials it names
A connection test that redirects the destination already leaves the configured
credentials behind. It kept litellm_credential_name, which names the same stored
secrets and is resolved further down the call, so the reference is now dropped
with them. A request that sets no connection fields of its own is unaffected,
which is how the Admin UI tests a configured model.
2026-08-05 16:07:04 -07:00
Yuneng Jiang
59173c3a20
feat(health): let allow_client_side_credentials re-enable configured-credential reuse
The proxy-wide opt-in that already governs callers supplying their own
connection parameters now also governs whether a connection test may pair a
request-supplied endpoint with the configured deployment's credentials. Off by
default, which keeps configured credentials scoped to the endpoint the
configuration names; on, the previous merge behaviour is available unchanged.
2026-08-05 15:58:49 -07:00
Yuneng Jiang
b468acb31c
fix(health): stop inheriting configured credentials when a connection test sets its own
A request that supplies its own connection fields describes a connection of its
own, so the configured deployment's credentials are no longer merged underneath
it. Anything the request leaves unset still comes from the configuration, so
naming a configured model and testing it as configured is unchanged, and adding
a second deployment for an already-configured name works as before.

Replaces the earlier outright rejection, which also refused requests that
supplied a complete connection of their own.
2026-08-05 15:49:41 -07:00
devin-ai-integration[bot]
aa1180c0c9
fix(core_helpers): map generic 'error' finish_reason to 'stop' (#33972)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-05 22:39:12 +00:00
mateo-berri
2ba4e91766 feat(guardrails): add scan_only_tool_results to scope unified guardrails to tool results 2026-08-05 15:38:08 -07:00
Yuneng Jiang
e5effcb861
fix(health)!: let configured deployment parameters win over request overrides
When a connection test names a model that resolves to a configured deployment,
that deployment's routing and credential parameters are authoritative. A request
supplying a complete connection of its own is unaffected.

BREAKING CHANGE: /health/test_connection no longer lets a request replace the
routing or credential parameters of a configured model it names. Supply the full
connection parameters instead of naming a configured model.
2026-08-05 15:17:43 -07:00
Yuneng Jiang
5b2c92d749
fix(proxy)!: parse bracket-notation form metadata the same way its JSON form is parsed
Multipart callers express nested metadata as flat bracket-notation keys, which
reach the request-body check as literal keys rather than as a metadata dict.
The check now rebuilds them with the same helper the endpoints use, so both
encodings are handled identically and cannot drift apart.

BREAKING CHANGE: a multipart field such as `litellm_metadata[api_base]` is now
subject to the same request-body parameter rules as its JSON equivalent. Set
`general_settings.allow_client_side_credentials`, or the deployment's
`configurable_clientside_auth_params`, to keep passing these.
2026-08-05 15:17:43 -07:00
Yuneng Jiang
fc4be70a37
fix(proxy)!: share one destination check between body and path-supplied model
The URL-destination check previously ran over request-body fields only. The
per-field logic moves into reject_url_valued_destination(field, value) so a
deployment name resolved from the request path runs the same check against the
same admin allowlist.

BREAKING CHANGE: a deployment name supplied in the request path that parses as
an http/https destination is now refused. Add the host to
`provider_url_destination_allowed_hosts` in litellm_settings to keep it working.
2026-08-05 15:17:43 -07:00
Mateo Wang
f047124b5a
feat(pre-commit): save full lint output to a per-worktree log file (#36004)
* feat(pre-commit): save full lint output to a per-worktree log file

* docs(claude): point agents at the pre-commit log instead of rerunning

* fix(pre-commit): warn when the log cannot be created or fully written
2026-08-05 15:16:52 -07:00
mateo-berri
f16f3e23cd fix(tool_permission): fail closed on unverifiable SSE streams and end the turn when every tool call is denied
An SSE stream that cannot be positively identified as Anthropic (no
parseable message_start event) now blocks instead of passing through
unscanned, closing the bypass where any raw-SSE backend skipped tool
permission checks entirely. Buffered chunks are joined back into one
stream before parsing, so events split across network chunk boundaries
assemble correctly instead of being silently dropped. Rewrite mode now
resets finish_reason to stop when no tool call survives, so the
re-encoded Anthropic stream reports stop_reason end_turn and clients do
not wait for a tool result that never comes
2026-08-05 15:06:51 -07:00
devin-ai-integration[bot]
f54cd287a2
fix(bedrock): grant bedrock:CountTokens in OIDC session policy (#33145)
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-05 14:55:46 -07:00
Yassin Kortam
9d58923065
fix(langfuse): stop a collected httpx handler from closing a shared client (#35981)
A cached HTTPHandler hands its raw httpx.Client out to consumers that keep it
for the process lifetime. When the shared client cache expires the entry on its
TTL or evicts it under the 200-entry cap, nothing references the handler, so it
is collected and its finalizer closed the client those consumers still hold.
Langfuse ingestion then failed silently on the SDK's background flush thread
until the process restarted.

A finalizer running proves only that nothing references the handler; it proves
nothing about the client. Both handlers now close the client during finalization
only when they built it and are still its sole referrer, so an unshared client is
still released promptly and a handed-out one is left alone. That keeps the
pooled-socket reclamation the finalizer was providing, which measures identical
to base over 2000 handler create-and-drop cycles.

Explicit close() stays, now gated on _owns_client so the wrapper never closes a
caller-injected client, and __aexit__ routes through it.

LangFuseLogger also keeps a reference to the handler whose client it hands the
SDK. Previously that handler was a local that went out of scope immediately,
leaving the client reachable only from the SDK. It still shares the cached
client, so no extra clients are created per logger.
2026-08-05 14:54:31 -07:00
Yassin Kortam
87dbb632b2
test(utils): pin the register_model replay test to the recorded half (#35994)
test_reapply_runtime_registrations_replays_register_model_overrides asserts that
a fetched catalog value survives the replay for a key an operator override does
not mention. Any Router still alive in the process re-asserts its own deployments
first, so a router serving openai/gpt-4o writes its model_info over that catalog
value and the assertion reads the router's number instead. Routers built by
earlier tests stay in the weak set until they are collected, which made the test
depend on collection timing and fail intermittently in shards that run the router
tests alongside it.

The live-router rebuild is covered in test_router_model_cost_isolation.py, so
this test now runs with the replay callback unset and exercises the recorded
registrations it is about.
2026-08-05 14:50:53 -07:00
Yassin Kortam
7984f4fa64
fix(logging): extend secret redaction to records litellm does not emit directly (#35977)
The redaction filter was attached to the handler shared by litellm's own
loggers, so it only covered records litellm emits. A litellm value can also
reach a log record through a dependency logging on its own logger, and those
records never pass through a litellm handler.

Attach the filter to each dependency logger that can carry one. The filter goes
on the emitting logger rather than on the root logger or a root handler, since
Logger.handle applies the emitting logger's filters before any handler runs, so
every downstream handler is covered regardless of who owns it.
2026-08-05 14:50:42 -07:00
Yassin Kortam
b8ef8508b5
fix(ci): make the env-key doc gate see bare get_secret and get_secret_str reads (#35996)
The gate required a litellm. prefix on get_secret and get_secret_str, so any
module importing either function directly bypassed it: 335 environment variables
read under litellm/ were invisible to it. The three patterns collapse into one
with the prefix optional, a negative lookbehind so attribute calls on unrelated
objects cannot match, and litellm.utils. accepted since four call sites reach
get_secret that way.

Widening the patterns alone would demand about 320 new rows in the central
reference table, most of them provider credentials that are already documented
on their own provider pages. So the gate now looks across every page of the docs
site rather than only that one table, which leaves 143 keys genuinely
undocumented instead of 322.
2026-08-05 14:49:55 -07:00
Abhimanyu Kapur
c76882b51b
fix(auto-router): stop the embedding model's context window from failing long requests (#35956)
* fix(auto-router): stop the embedding model's context window from failing long requests

The auto-router embeds the last user message to pick a model and sent it to the
embedding model unbounded. Embedding models carry 512 to 8k token windows while the
chat models they route to carry 200k+, so any prompt over the encoder's window failed
at the routing step with a 400 the destination model would never have raised.

Cut every doc to a character cap inside LiteLLMRouterEncoder, which is the one choke
point the auto-router, complexity-router, semantic guard and MCP tool filter all share.
Default 2000 chars, roughly 500 tokens, which fits even a 512-token self-hosted encoder,
overridable per deployment with auto_router_max_input_chars and globally with
DEFAULT_MAX_EMBEDDING_INPUT_CHARS.

Truncation alone cannot cover provider-side batch and byte limits, so any failure of
the route call now falls back to the auto-router's default model instead of propagating.
That path also fixes two latent bugs: a no-match left the auto-router alias in place as
the model name, which fails downstream with "Unmapped LLM provider" rather than reaching
default_model, and an empty route list raised IndexError.

Fixes #17869
Fixes #20277

* fix(auto-router): make the embedding input cap opt-in so guards still see whole prompts

Defaulting the cap inside the shared encoder truncated every consumer, not just the
auto-router. The semantic guard builds the same encoder, so its pre-call check would
have classified only the first 2000 characters while the full message still reached the
model, which a benign opener in front of an injection payload walks straight past. The
MCP tool filter and complexity router were silently narrowed the same way.

The encoder now defaults to sending docs whole and cuts only when a caller passes
max_input_chars. The auto-router is the only caller that does, so guard, MCP filter and
complexity-router behaviour is unchanged from before this branch.

DEFAULT_MAX_EMBEDDING_INPUT_CHARS becomes DEFAULT_AUTO_ROUTER_MAX_INPUT_CHARS, since it
is now specific to the auto-router, and drops its env override: the per-deployment
auto_router_max_input_chars already covers it, and every env var in constants.py has to
be documented, which is what broke the documentation and code-quality checks.

Also drops the added comments and the redundant type: ignore that review flagged.

* test(auto-router): cover the max_input_chars wiring from litellm_params

Nothing asserted that auto_router_max_input_chars on the deployment reaches the
AutoRouter that embeds prompts. Dropping the wiring left every test green while the cap
silently reverted to the default, so an operator with a 512-token embedding model could
not lower it and every long prompt would fall back to the default model instead of
being routed.

* test(auto-router): cover the populated route-choice list branch

The route layer can hand back a list, and picking its first element is where the
IndexError lived: the empty case was covered but the populated one was not, so the
branch that reads route_choice[0].name could be deleted with every test still green.
2026-08-05 14:47:40 -07:00
Devin AI
53ee9c8293 fix(anthropic): fall back when only some compaction iterations report thinking tokens
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-05 21:34:36 +00:00
Yassin Kortam
c3a8962c00
fix(proxy): only treat a recoverable database outage as grounds to serve without one (#35864)
`is_database_connection_error` answered True for any `PrismaError` it did not
recognize, on the reasoning that an unclassified failure might be an outage and
the safer default was to keep serving. That default is inverted for faults that
never resolve. A query engine that is missing or version-skewed, a malformed
generated query, or a misused transaction all satisfied the predicate, so with
`allow_requests_on_db_unavailable` enabled the proxy would absorb one, boot
clean, and keep issuing fallback identities for as long as the process ran.

The predicate is now an allowlist: the httpx transport errors, prisma's
`EngineConnectionError`, and a `no_db_connection` ProxyException. That is what a
real outage produces, since the query engine is a local HTTP server and an
unreachable database surfaces as a transport failure against it, so the
high-availability path is unchanged. Anything unrecognized is now treated as
permanent and surfaces instead of being absorbed.

Deciding whether to serve without a database and deciding what to tell the
caller are different questions, so they no longer share a predicate.
`is_database_infrastructure_error` keeps the previous broad behavior and now
backs the reporting and recovery paths: service-unavailable classification, the
access-group endpoint's status mapping, and the health watchdog's reconnect
trigger. Their behavior is unchanged. Without that split, a permanently faulted
engine would have started reporting as an authentication failure, sending an
operator after a credential problem that does not exist.
2026-08-05 14:15:13 -07:00
Yassin Kortam
309e96c27b
fix(jina_ai): resolve the documented JINA_API_KEY as a fallback (#35992)
The Jina key fallback chain read JINA_AI_API_KEY three times in a row
before falling through to JINA_AI_TOKEN, so two of the four slots were
dead. Jina's own documentation publishes JINA_API_KEY, and litellm's
rerank validate_environment already tells users to set that name, but
nothing ever read it: a user who set only JINA_API_KEY got no key
resolved and Jina answered AUTH_MISSING_API_KEY.

Replace one of the repeats with JINA_API_KEY and drop the other.
JINA_AI_API_KEY stays first so no install that resolves a key today
changes which key it picks.
2026-08-05 14:13:42 -07:00
Devin AI
aadfa89ff6 Merge branch 'litellm_internal_staging' into litellm_fix_bedrock_adaptive_thinking_token_accounting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-05 21:13:36 +00:00
Yassin Kortam
4562b539d8
fix(docker): fail the image build when the generated prisma engine paths drift off /opt/prisma (#35979)
The generated client bakes absolute query engine paths at build time, and
prisma-python scans them before it reads PRISMA_QUERY_ENGINE_BINARY. That
scan propagates EACCES rather than skipping a candidate, so a path baked
under the build user's HOME crashes client startup with a bare
PermissionError for every other uid, and the documented override cannot
recover from it.

Assert in the runtime stage that every baked query engine path sits under
the fixed, world-readable /opt/prisma bake, so a regression in the
generate step breaks the build instead of shipping an image that only
starts under the uid that built it. Adds an image-level test that runs
the same resolution as an arbitrary non-root uid.
2026-08-05 14:12:28 -07:00
mateo-berri
bee787b4b5 fix(guardrails): scan /v1/messages tool traffic
Guardrails silently skipped three surfaces on the Anthropic Messages
path, so an agent loop driven by /v1/messages ran unguarded:

- The Anthropic input translation never walked tool_result blocks, so
  content returned by a local tool (a curl, a file read, an MCP call)
  reached the model unscanned in both the string and list content
  shapes, images inside a tool_result included.
- tool_permission only understood ModelResponse, so an Anthropic
  non-streaming response or a raw SSE stream carrying tool_use blocks
  passed through with no rule ever evaluated.
- ContentFilterGuardrail scanned inputs["texts"] but never
  inputs["tool_calls"], so the arguments a model proposes for a tool
  call went unchecked.

Tool call arguments are parsed as JSON before filtering so a MASK
action rewrites the value and leaves the payload valid JSON; non-JSON
arguments fall back to scanning the raw string. Denied tool_use blocks
are dropped from the Anthropic content array and replaced with a text
block, and stop_reason resets to end_turn when nothing tool-shaped
survives.
2026-08-05 14:11:27 -07:00
Miles Adkins
431f61b4f7 fix(fireworks_ai): prefer native values silently on extras conflicts
Align with the API gateway translation: instead of raising BadRequestError
on alias or competing-constraint conflicts, the explicit Fireworks-native
param wins and the NIM/vLLM extra is dropped with a debug log. Covers
truncate_prompt_tokens vs prompt_truncate_len, chat_template_kwargs
enable_thinking vs reasoning_effort/thinking, guided_* vs response_format
(including response_format nested in an explicit extra_body, which the
previous conflict check missed), and multiple guided_* params (priority
order json, grammar, choice). Malformed non-object chat_template_kwargs
is also dropped with a log instead of raising.
2026-08-05 16:02:47 -05:00
Miles Adkins
6d80d05099 fix(fireworks_ai): align extras translation with the API gateway matrix
min_tokens is accepted natively by the Fireworks API (verified live), so
stop stripping it and let it pass through extra_body. Add the NIM-specific
include_reasoning and nvext keys to the strip set. enable_thinking=true
now omits reasoning_effort (model default) instead of forcing medium,
matching the gateway translation and preserving default-off models'
behavior; enable_thinking=false still maps to none.
2026-08-05 15:56:33 -05:00
Yassin Kortam
8ec562f279
fix(ai21): resolve the documented AI21_API_KEY instead of a misspelled name (#35985)
get_api_key resolved the ai21 key from AI211_API_KEY, with a doubled 1. Every other
ai21 code path reads AI21_API_KEY, including the validate_environment branches that
report it as the missing one, so the name a user is told to set was ignored here.

No user path reaches this branch today, since every provider-resolution site rewrites
custom_llm_provider to ai21_chat and sets the key from a correctly spelled read first,
so this is a correctness fix rather than a bug fix. It is worth making because the
env-var documentation gate reads this call site: leaving the misspelling in place would
require a row for AI211_API_KEY in the environment variables reference table, which
would turn a typo into public API
2026-08-05 13:53:13 -07:00
Devin AI
af2246c5b8 fix(anthropic,bedrock): report provider thinking tokens instead of classifying them as text
Resolves LIT-5244

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-05 20:47:05 +00:00
Mateo Wang
df831b807f
chore(typing): replace Any seams with real types across responses, proxy, and provider adapters (#35809)
* chore(typing): replace Any seams with real types across responses, proxy, and provider adapters

Replace Any-typed payload dicts, record shapes, and provider request/response
seams with TypedDicts, Protocols, and precise annotations in the ten litellm/
files carrying the highest combined basedpyright reportAny + reportExplicitAny
counts. No behavior changes.

Adds a regression test covering the managed-id list path so a prisma client
missing the managed tables keeps returning a fail-closed empty page.

* fix(passthrough): walk scalar request bodies through managed-id rewrite again

The top-level dispatch in rewrite_body_ids only handled dict and list
bodies, so a truthy scalar JSON body (bare string, number, bool) hit
dict.items() and raised AttributeError where the merge base passed it
through, and a bare managed-ID string body lost resolution. Restore the
base behavior by dispatching through _walk, widen the implementation to
object with a catch-all overload, and pin both paths with regression
tests
2026-08-05 13:45:21 -07:00
Yassin Kortam
267adf709a
fix(bedrock): sign Bedrock managed-file S3 requests with S3SigV4Auth (#35983)
S3 rebuilds the canonical request from the wire path with single
percent-encoding, which botocore models as S3SigV4Auth. Generic SigV4Auth
quotes the already encoded path a second time, so an object key holding any
character that percent-encodes was signed over %2520 while the request
carried %20, and S3 answered 403 SignatureDoesNotMatch.

The S3 logger was corrected in #35726; the Bedrock managed-files upload and
retrieval paths copied that same pre-fix pattern and were left behind, so a
configured bucket prefix with a space 403s every file upload and every file
content read.
2026-08-05 13:43:30 -07:00
Yassin Kortam
de00655363
fix(migrations): keep the toolchain heal from raising on an unreadable nodeenv cache (#35986)
heal_incomplete_nodeenv_cache() stats a $HOME-derived path with a bare
Path.is_dir(). pathlib only swallows ENOENT-shaped errnos, so a cache
directory the process cannot search raises PermissionError instead of
answering False. Images bake that cache under the build user's home, whose
mode is 0700, so a container started under any other uid dies there before
the Prisma CLI is ever invoked, and the migration never runs.

Tolerate OSError while inspecting the cache, matching the guard
nodeenv_cache_dir() already carries, so an unreachable cache means there is
nothing to heal rather than a crash. This restores the never-raises
contract ensure_prisma_toolchain() documents.
2026-08-05 13:38:53 -07:00
mateo-berri
3728f3ea62 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_daily_any_cleanup_08_04_2026 2026-08-05 13:14:37 -07:00
Miles Adkins
599283584f feat(fireworks_ai): drop reasoning_effort=auto to the model default
Fireworks rejects reasoning_effort="auto" (accepted set: low, medium,
high, xhigh, max, none, adaptive), so OpenAI-compatible clients sending
it 400. Omitting the param means model default on Fireworks, which is
exactly what auto means on OpenAI's side, so skip it in
map_openai_params instead of forwarding.
2026-08-05 15:09:16 -05:00
tin-berri
32deaff015
feat(spend): rebuild the auto-router benchmarks backend as a per-session rollup (#35910)
Folds every successful auto-routed request into LiteLLM_AutoRouterSession with one
conditional upsert at spend-write time, classifying each turn (same model, first
visit, return to tier, out of order) against the row's own columns so nothing is
read before the write. The upsert's placeholders and argument tuple both derive
from the transaction dataclass's own field order, so the SQL and the call site
cannot drift apart. GET /auto_router/benchmarks aggregates the rollup, grouped
by the full (router, type) identity, and never scans LiteLLM_SpendLogs. A turn's
cache interaction is derived once from its usage record (savings.py owns the
extraction; compute_savings_spend derives cache reads from usage_object itself),
hits are counted order-independently so the overall hit rate matches its covered
denominator, caller-chosen session ids are bounded before entering the primary
key, and a poisoned statement drops only its own session's remaining turns.
Return misses inside the recorded TTL are named for what the telemetry shows
(within_ttl) rather than a presumed cause, since a provider can evict early.
Savings ride each router's derived baseline by default, so the response carries
no deployment-wide baseline label. Rollup retention has its own
maximum_autorouter_session_retention_period setting, pattern-identical to the
spend-logs knob and running in the same cleanup job on its own cutoff. Every
drain trigger sizes the queues through one owner and the enqueue honors
disable_spend_logs beside the tool-usage queue it mirrors.
2026-08-05 20:06:32 +00:00
Abhimanyu Kapur
bea65b6dcc
fix(autorouter): match CJK keyword_tier_rules that regex word boundaries miss (#35984)
* fix(autorouter): match CJK keyword_tier_rules that regex word boundaries miss

Single-word keywords were matched with a \b...\b regex. Every CJK character is a
regex word character and CJK is written without spaces, so \b never fires between
two of them and a rule like 发票 silently missed 我需要开发票, falling through to
complexity scoring instead of the configured tier.

Keywords containing CJK now match as plain substrings, the same way multi-word
phrases already did. The gate reads the keyword rather than the prompt, so a
keyword with no CJK in it keeps word boundary matching regardless of the script
the prompt is written in.

* fix(autorouter): cover Han extensions in planes 2 and 3, not just up to U+2FA1F

The supplementary range stopped at U+2FA1F, so Extension G and H ideographs kept
the word boundary path and stayed unmatchable. Both planes are dedicated to CJK
ideographs, so covering them whole also handles later extensions without chasing
each new block.
2026-08-05 20:02:54 +00:00
Mateo Wang
c6dbf48944
Merge pull request #35916 from BerriAI/litellm_passthrough_live_credentials
fix(proxy): resolve pass-through credentials live from router deployments
2026-08-05 12:57:46 -07:00
Yassin Kortam
347798b80e
fix(router): keep custom model_info across a price data reload (#35491)
A price data reload replaced litellm.model_cost wholesale, discarding every
runtime registration: the deployment model_info the Router registers from
model_list, and pricing overrides passed to litellm.register_model. Custom
model groups lost max_input_tokens / max_output_tokens in /model_group/info,
and a deployment whose backend model is in the catalog silently reverted to
upstream values. Runtime registrations are now recorded and replayed on top of
the freshly fetched catalog.

Router._pre_call_checks resolved the per-deployment model name only after the
model-info lookup, so an unregistered model left it unset and the supported
params check ran against the bare model group name, raising "LLM Provider NOT
provided" out of deployment selection. The name is now resolved first, and an
unresolvable provider skips that check rather than failing the request.

Resolves LIT-4675
2026-08-05 19:56:50 +00:00
Yassin Kortam
7ac1085931
fix(auth): return 403 from the OAuth2 enterprise gate (#35838)
The enterprise gate on the OAuth2 auth path raised a bare `ValueError`,
which the terminal handler in auth_exception_handler.py converts to a 401.
Every sibling enterprise gate answers 403, including `_premium_user_check`
and the SSO gate. A 401 tells the client its credential was wrong and to
retry with a better one, and no credential can satisfy that while the
install is unlicensed, so it invites a retry loop that can never succeed.

It now raises a 403 `ProxyException` shaped like the SSO gate. Two response
fields move with it: the `Authentication Error, ` prefix goes away, since
the catch-all built that around `str(e)` and a `ProxyException` is re-raised
unmodified, and `param` becomes `premium_user`, naming the condition an
operator has to clear.

The gate's own text also gains the sentence break it was missing. The
message concatenated straight onto `CommonProxyErrors.not_premium_user`,
rendering as "premium usersYou must be a LiteLLM Enterprise user".
2026-08-05 12:53:56 -07:00
Abhimanyu Kapur
b8df48cd7f
feat(auto-router): let operators replace the LLM classifier's system prompt (#35855)
* feat(auto-router): let operators replace the LLM classifier's system prompt

The complexity router's LLM classifier has always sent one built-in rubric, so the
router could only ever grade difficulty. Operators can now supply their own system
prompt, which replaces the rubric outright and repurposes the same tier machinery for
whatever taxonomy the prompt defines, data sensitivity being the obvious case.

Replacement is total: neither the rubric nor its closing line is appended, since both
describe grading difficulty over a "current message" and a prompt grading something
else is entitled to contradict them. That closing paragraph is also the classifier's
prompt-injection defense, so the config field and the dashboard editor both warn that
a replacement omitting it lets a caller ask for a tier and get it.

The heuristic fallback still scores complexity, which is meaningless for a repurposed
taxonomy, so classifier_fallback now chooses between the heuristic scorer and routing
straight to default_model. The default_model path bypasses tier pools, the adaptive
bandit, and escalation, because no tier was decided and the point of that fallback is
a known destination. It reports itself as default_model_fallback in the spend logs.

The dashboard's prompt editor prefills from a new
/auto_router/classifier/default_prompt endpoint rather than a copy of the rubric in
the frontend, and stores no override when the draft matches the default, so later
rubric improvements still reach every router that never customized it.

Tier names stay SIMPLE/MEDIUM/COMPLEX/REASONING; a custom prompt redefines what they
mean, not what they are called.

* fix(complexity-router): don't let the default_model classifier fallback bypass routing plugins

* fix(complexity-router): don't pin a session to the default model after a classifier failure

* fix(complexity-router): omit the tier from a default-model-fallback routing decision

The classifier never answered, so no tier was decided. The record reported the
tier whose pool happens to hold default_model, which reads in the spend log and
the UI as if the request was classified. Matches how default_fallback already
records a route that no tier produced.

* fix(proxy): allowlist /auto_router/ on the UI backend component

The new GET /auto_router/classifier/default_prompt is a UI-consumed management
route, so it belongs on the control plane. Without the prefix it was exposed by
neither component and test_gateway_plus_backend_covers_full_app failed.

* docs(ui): reword the classifier prompt disclaimer

Frames the closing paragraph as a strong recommendation rather than a
description of what gets dropped, names prompt injection explicitly, and
notes the tier names stay fixed regardless of their display names.

* fix(complexity-router): stop logging a fabricated tier on the plugin fallback path

The classifier-failed fallback resolves a tier so the routing-plugin pipeline has a
pool to filter, but nothing about the request produced that tier. The non-plugin
short-circuit already dropped it from the logged decision; the plugin path still
reported it, so a spend log claimed a classification the request never received.
Record the pool as a plugin-filtered-pool signal instead.

Also name the real problem when the resolved tier has no models at all: that raised
"No candidate models left after routing-plugin filtering" and sent operators hunting
for a policy plugin that never narrowed anything.
2026-08-05 19:48:11 +00:00
mateo-berri
2bd6af1b64 Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_daily_any_cleanup_08_04_2026 2026-08-05 12:43:11 -07:00
Yassin Kortam
09dd167b5a
feat(sgr): make the gateway middleware the source of truth for successful requests (#35717)
SGR has had two independent definitions. The admin UI derived it from
SpendLogs, so it counted what litellm's logging callbacks observed and could
attribute and price. BillableRequestMetricsMiddleware counted what the proxy
actually answered at the ASGI edge, but only exported to OTLP for enterprise
metering. The two disagree by design in places, and the SpendLogs figure goes
quiet whenever spend logging is disabled or the callbacks are bypassed.

This adds LiteLLM_DailyGatewayRequests, written by the middleware, and points
the dashboard's Successful Requests tile at it.

Requests fold into an in-memory map at record time rather than going through a
queue like the spend path. A count is a pure aggregate, and every dimension of
the key is chosen by the proxy from a closed set: the date, the category, and a
route that the classifier maps to one of a fixed list of strings rather than
passing the raw path through. Nothing a caller sends can add a key, so the fold
and the table are bounded by (days x categories x routes) however much traffic
arrives; the spend queue blocks once full, which is not acceptable in the
response path. A scheduler job drains it on the existing batch interval, and a
failed flush merges its counts back so a database blip undercounts nothing.

The middleware previously returned early when no billing recorder was
injected, which is the unlicensed case. The new sink is not license-gated, so
that early return now requires both sinks to be absent. The billing recorder
keeps its 2xx-only gate; the sink takes every status so failed_requests is
real. The sink is not told which deployment served the request, unlike the
billing recorder. That id is a sha256 over litellm_params, credentials
included, so a caller who puts a credential in the request body mints a fresh
one per distinct value. No configuration is needed for that: api_base and
base_url are on _BANNED_REQUEST_BODY_PARAMS and need allow_client_side_
credentials, but api_key is not on that list, and both reach the same
_handle_clientside_credential branch. The read endpoint aggregates the
dimension away regardless, so the key is better off without it.

The new table carries no key, user or team dimension, so /gateway/daily/activity
is restricted to proxy admin roles and the per-key and per-model breakdowns
keep reading the daily spend tables. The old path is left running and marked
with TODOs.

A fetched result carries the range key it was fetched for, and the render
selects it only when that key matches the range on screen. Both the gateway
counts and the spend aggregate go through that rule: the request tiles read the
first and fall through to the second, so stamping only one of them would leave
the tile showing a superseded range by the other route.

The paginated pages behind that aggregate are reached through a failure flag,
so the flag is stamped too. A flag left over from the previous range would let
those pages through while a new range is in flight, which is the same defect
one fallback further down.
2026-08-05 12:40:47 -07:00