scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-09-11 22:51:28 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 154
46618 commits 2327 branches 1406 tags 2.4 GiB
Commit graph

4 commits

Author SHA1 Message Date
yuneng-jiang
6a0d03914c
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802) 
* test: drop the cwd-relative sys.path.insert calls from the test suite

TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.

Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.

Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.

* test: drop the duplicate imports the sys.path sweep exposed to F811

* test(pre-call-utils): restore the os import the new bedrock tests need
 2026-08-22 09:25:58 -07:00
tin-berri
add095b494
Fix: ComplexityRouter should not score system prompt text for code/technical complexity (#36721) 
System prompts (harnesses, tools, framework boilerplate) are session-wide
constants identical across all requests. Scoring them saturates keyword-match
signals and produces false-positive high-complexity classifications on
trivial utterances like 'hi', routing them to expensive models (sonnet/opus)
instead of tier-1 haiku. A real ~1.6KB CLI-agent harness alone supplied
5 codePresence + 2 technicalTerms matches, overshadowing user signal.

Rescope four scoring dimensions (codePresence, technicalTerms, simpleIndicators,
multiStepPatterns) from full_text (system + user) to user_text (user only).
reasoningMarkers was already scoped this way. This returns 0.63 of the weight
budget to text that actually varies per-request.

Now that every dimension scores user_text only, _score_keyword_match's
disclosable_text param is redundant -- it existed solely to let the signal
name terms matched in the caller's own message while withholding terms
matched only in the (invisible-to-the-caller) system prompt. With no more
system-prompt text in scope, text and disclosable_text were identical at
every call site, so the param is dropped and the function collapses to a
single text argument.

Add mutation-proven regression test: trivial 'hi' message with realistic
Claude Code agent system prompt now routes to haiku tier-1 (not sonnet).

- Unfixed: haiku -> sonnet (bug)
- Fixed: haiku -> haiku (correct)

Invert three pre-existing assertions in TestSignalsNeverQuoteTheSystemPrompt
to capture the corrected behavior: system-prompt-only terms produce no signal.

Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-13 11:15:14 -07:00
tin-berri
71dfab7177
feat(router): record why the auto-router picked a tier and show it in the logs (#35016) 
Auto-routed requests were indistinguishable from ordinary ones once logged:
the spend log recorded the requested model group and the resolved deployment,
but nothing about which tier was chosen or what chose it. That information
existed only inside verbose_router_logger f-strings, so answering "why did my
prompt land on the cheap model" required log access and a running proxy.

The complexity, quality, and adaptive pre-routing strategies now return a typed
StandardLoggingRoutingDecision on their PreRoutingHookResponse, and
Router.async_pre_routing_hook records it once for every attempt. Those three
previously side-channelled their own state through three different metadata
keys; the decision now travels on the hook contract itself, so the bucket is
resolved in one place, through get_or_create_metadata_bucket, which already
owns the question of which dict holds proxy-internal metadata and replaces a
non-dict value instead of skipping the write. Recording happens on every
attempt rather than only on a successful route: a fallback from an auto-router
group to a plain group re-enters the hook with the same request kwargs, and a
decision left behind there would attribute the first router's tier to the
deployment that actually served the retry. The log details drawer renders the
result as a Routing card between Request Details and Metrics; the card is
absent on rows that carry no decision, so ordinary and pre-upgrade rows are
unchanged.

Three defects surfaced while making the recorded cause truthful, each of which
would have persisted a wrong answer. The complexity router hardcoded
cause=complexity_scorer even when the LLM classifier decided, and its silent
fallback to the heuristic on classifier failure meant a row could claim an LLM
verdict the LLM never gave; the cause now reports the path that actually ran.
The keyword that triggered a tier rule was discarded before logging, as was
the escalation keyword. The 2-reasoning-marker override returned REASONING with
a score far below the REASONING boundary and no marker saying so, which reads
as a scoring bug to anyone comparing the two; it now emits a reasoning-override
signal, and the card labels those rows as an override instead of claiming the
score met a boundary. The LLM path no longer reports a synthetic score of 1.0,
and heuristic decisions carry a snapshot of the tier boundaries that mapped the
score, so a historical row stays interpretable after the boundaries change.

Signals name a matched term only when the caller's own message contains it.
Scoring still reads the system prompt, but a term matched solely there is
reported as a count, since signals reach a spend row the caller can read and
naming one would disclose a term from a prompt it cannot see.

routing_decision is stripped from caller-supplied metadata at ingress, so a
client cannot forge its own provenance.
 2026-07-30 11:55:10 -07:00
Krrish Dholakia
e7bc316db0
Litellm krrish staging 04 20 2026 (#26138) 
* feat(router): add auto_router/quality_router for quality-tier routing (#25987)

* feat(router): add auto_router/quality_router for quality-tier routing

Adds a new auto-router type that routes a request to a model at a target
quality tier. The quality tier is inferred by re-using the existing
ComplexityRouter's classification, then mapped through an admin-configured
complexity_to_quality table. Each candidate model declares its own
quality_tier in model_info.litellm_routing_preferences.

Resolution strategy: exact tier match, else round up to the next higher
tier, else fall back to default_model.

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* feat(quality_router): add capability-based filtering

Each deployment can declare a `capabilities: List[str]` field in
`model_info.litellm_routing_preferences` (e.g. ["vision",
"function_calling"]). Requests can pass `litellm_capabilities` in
`request_kwargs` to require specific capabilities — the router will only
route to deployments whose declared capabilities are a superset.

Resolution still walks tier (exact → round up), but at each tier filters
by capability before picking. Falls back to default_model only when it
also satisfies the required capabilities; otherwise raises rather than
silently routing to a model that lacks a required capability.

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* feat(quality_router): expose routing decision in response headers

For transparency, expose the QualityRouter's routing decision in the
proxy response headers:

  x-litellm-quality-router-model       → picked model_name (e.g. "haiku-vision")
  x-litellm-quality-router-tier        → resolved quality tier (e.g. "1")
  x-litellm-quality-router-complexity  → ComplexityTier name (e.g. "SIMPLE")

Mechanism: the pre-routing hook stashes the decision in
request_kwargs["metadata"]["quality_router_decision"]. After the call
returns, Router.set_response_headers lifts the decision into
response._hidden_params["additional_headers"] alongside the existing
x-litellm-model-group / x-litellm-model-id headers. Existing metadata
keys (trace_id, user_id, etc.) are preserved.

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* feat(quality_router): replace capabilities with keyword override

Drops the capability-based filtering in favor of a keyword-based override
for v0:

- RoutingPreferences.keywords: List[str] (replaces capabilities) — each
  deployment can declare substring keywords.
- If any declared keyword (case-insensitive) appears in the user message,
  the router short-circuits the complexity-classification flow and routes
  to the matching deployment.
- Tiebreaker for overlapping keyword matches: quality_tier DESC, then
  cheapest model_info.input_cost_per_token ASC. Unpriced models lose ties
  to priced ones.

Decision metadata + headers now expose the override:
  x-litellm-quality-router-via       → "keyword" | "quality_tier"
  x-litellm-quality-router-keyword   → matched keyword (only on keyword route)
  x-litellm-quality-router-complexity → complexity tier (only on tier route)

Removes:
- request_kwargs["litellm_capabilities"] reading
- _model_capabilities, _model_supports_capabilities,
  _first_capable_model_at_tier, capability filter in
  _resolve_model_for_quality_tier

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* feat(quality_router): add explicit `order` to RoutingPreferences

Adds an explicit priority field to RoutingPreferences for resolving
collisions deterministically:

  RoutingPreferences.order: Optional[int]   # lower wins; unset = +inf

Used as the PRIMARY tiebreaker in two places:

1. Keyword overlap: when multiple deployments declare the same matching
   keyword, sort by (order ASC, quality_tier DESC, input_cost_per_token
   ASC, model_name ASC). Explicit always beats implicit.

2. Tier resolution: when multiple deployments share a quality tier,
   `_resolve_model_for_quality_tier` picks the one with the lowest
   order. The tier list is now sorted at index-build time.

This lets admins make routing decisions explicit when the natural
quality-and-price ordering would pick the wrong model.

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* feat(quality_router): reorder tiebreak to (quality, order, price)

Changes the tiebreak ordering so quality_tier always wins first, then
explicit `order` is used to break ties within the same tier, then price
breaks the rest:

  1. quality_tier DESC      ← best model wins first
  2. order ASC              ← explicit priority within a tier
  3. input_cost_per_token ASC
  4. model_name ASC

Previously `order` was the primary key — that meant a tier-2 model with
`order=1` would beat a tier-3 model with no `order`, which is the wrong
default. Now `order` only resolves collisions among same-tier candidates.

Tier resolution (within a single tier) keeps the same key minus quality:
(order ASC, cost ASC, name).

Test renames + flips:
  - test_explicit_order_overrides_quality_tier → test_quality_wins_over_explicit_order
  - new: test_order_breaks_tie_within_same_quality_tier

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* fix(quality_router): resolve Greptile review feedback

Addresses four P1 findings from PR review plus test coverage:

1. set_model_list missing quality_routers reset
   - Hot-reloading the Router would leave stale QualityRouter instances
     pointing at the old model_list. `set_model_list` now clears
     `self.quality_routers` alongside the other indices.

2. Round-down fallback before default_model
   - `_resolve_model_for_quality_tier` now rounds DOWN to the closest
     lower tier after round-up fails, before falling back to
     `default_model`. Degrades gracefully rather than jumping straight
     off-tier.

3. RoutingPreferences validation bypass
   - `_build_tier_index` now instantiates `RoutingPreferences(**prefs)`
     so invalid shapes (e.g. non-int quality_tier) raise a clear
     ValueError instead of silently succeeding.

4. Config-ordering dependency
   - `_tier_to_models` is now built lazily on first access. Previously,
     eager construction in `__init__` meant a QualityRouter deployment
     had to appear AFTER all its referenced models in config.yaml,
     because `Router._create_deployment` populates `model_list`
     incrementally. Any `available_models` defined after the router
     entry would silently be reported as missing.

Also adds 6 new tests covering each fix:
- test_invalid_quality_tier_type_raises_clear_error
- test_router_can_be_instantiated_before_its_targets_exist
- test_set_model_list_clears_quality_routers_registry
- test_rounds_down_when_no_higher_tier_exists
- test_rounds_down_prefers_closest_lower_tier
- test_prefers_round_up_over_round_down

Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>

* style: apply black 24.10.0 formatting to pre-existing offenders

Unblocks the LiteLLM Linting check for this PR — these 12 files are already
failing `black --check` on main (the lint workflow only runs on PRs, so main
drifts). No behavior changes; formatting-only.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Update litellm/router.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Support /v1/responses in complexity router (#26137)

* feat(proxy): add --reload flag for uvicorn hot reload (dev only)

Opt-in CLI flag, off by default, no env var. Only affects the uvicorn
run path; gunicorn/hypercorn paths and prod (which doesn't pass the
flag) are unaffected.

* Feature/add audio support for scaleway (#26110)

* feat(scaleway): add SCALEWAY to LlmProviders enum

* feat(scaleway): add audio transcription config and dispatch wiring

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(scaleway): add behavior tests for audio transcription config

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(scaleway): advertise audio_transcriptions in endpoint-support JSON

* docs(scaleway): document audio transcription support

* fix(scaleway): address PR review — plain-text response_format + missing-key fail-fast

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(scaleway): cover new response paths, drop gettysburg.wav coupling

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* Prompt Compression - add it to the proxy (#25729)

* refactor: new agentic loop event hook

simplifies how to create logic for tool based multi llm calls

* fix: compress - make it work on anthropic input as well

* fix(compress.py): working prompt compression for claude code

ensures claude code messages can run through proxy easily

* docs: add agentic loop hook guide

* docs: add agentic_loop_hook to sidebar

* fix: fix multiple arguments error

* fix: fix tool call loop for compression on streaming /v1/messages

* fix: fix linting errors

* fix: fix ci/cd errors

* feat(litellm_pre_call_utils.py): use claude code session for litellm session id

allows claude code logs to be stitched together, making it easy to know they were all part of the same conversation

* fix: suppress incorrect mypy warning rE: module

* revert: drop PR's changes to litellm/proxy/_experimental/out/

Restores the 34 HTML files under _experimental/out/ to their pre-PR
paths (X/index.html -> X.html). All renames are R100 (content
unchanged); no other files are touched.

* fix: address greptile review comments on PR #25729

- Skip ``kwargs["tools"] = []`` injection when compression is a no-op —
  Anthropic Messages rejects empty tool arrays on requests that did not
  originally declare tools.
- Move agentic-loop safety guards (fingerprint cycle / max depth) out of
  the per-callback try/except so they propagate instead of being swallowed
  by the generic exception handler. Extracted _check_agentic_loop_safety.
- Gate generic ``x-<vendor>-session-id`` capture behind the
  LITELLM_CAPTURE_VENDOR_SESSION_HEADERS env var (off by default) to
  preserve backwards compatibility; explicit x-litellm-* headers are
  unaffected.
- Fix monkeypatch target in pre-call-hook test to patch the actual
  module-level binding
  (litellm.integrations.compression_interception.handler.compress).
- Add regression tests for empty-tools skip and opt-in session capture.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* revert: drop LITELLM_CAPTURE_VENDOR_SESSION_HEADERS flag

Generic x-<vendor>-session-id header capture is a new feature and only
runs *after* the explicit x-litellm-trace-id / x-litellm-session-id
checks, so it does not change behavior for any existing caller that was
already using the LiteLLM headers — no backwards-incompatibility to gate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(compress): replace input_type with CallTypes call_type

Drop the bespoke ``CompressionInputType`` literal and use the existing
``litellm.types.utils.CallTypes`` enum instead.  ``litellm.compress()``
now takes ``call_type: Union[CallTypes, str]`` (default
``CallTypes.completion``) — no new concept to learn, and the enum is
already the way the rest of the codebase talks about request shapes.

Supported values: ``completion`` / ``acompletion`` (OpenAI chat-completions
shape) and ``anthropic_messages`` (Anthropic structured content blocks).

Updated: compress(), the compression_interception handler, tests, docs,
and the two eval scripts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Support /v1/responses in complexity router

Adds cross-format support to the complexity router via the guardrail
translation handler dispatch. Adds get_structured_messages to base
translation plus OpenAI chat, Responses, and Anthropic handlers.
Auto-router helper _extract_text_from_messages handles tool-call and
multimodal messages. Widens async_pre_routing_hook messages type to
Dict[str, Any].

Fixes https://github.com/BerriAI/litellm/issues/25134

* chore: apply black formatting

* fix: fallback to trying each handler when route inference fails

---------

Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

* test: cover _is_quality_router_deployment and init_quality_router_deployment

* fix: reset auto_routers on set_model_list to prevent hot-reload ValueError

* style: apply black formatting to websearch_interception and agentic_streaming_iterator

---------

Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: nhyy244 <106547304+nhyy244@users.noreply.github.com>
 2026-04-20 16:22:12 -07:00