mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-10 22:41:41 +00:00
20 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
6a0d03914c
|
test: drop the cwd-relative sys.path.insert calls from the test suite (#37802)
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
|
||
|
yucheng-berri
|
5285ae86d5
|
fix(ptu): warn when config.yaml declares PTU while attribution is off (#37898) | ||
|
ryan-crabbe-berri
|
91599aef69 | test: say whether a match= pattern is a regex or a literal (ruff RUF043) | ||
|
yucheng-berri
|
8122cfc1ec
|
fix(ptu): require an operator-declared id on a config.yaml reservation (#37794)
A config deployment is otherwise keyed by a hash of its resolved litellm_params, so rotating a credential or editing an endpoint mints a second identity and the catch-up bills the reservation again under it. Flat cost is keyed by that id and a written charge is never retracted, so the duplicate is permanent. The id is read before set_model_list mints one, or the rule would inspect the value it is meant to reject. Duplicates are counted once per config entry across the whole file, so the check is order-independent and an organization fan-out cannot collide with itself. The refusal names the id the deployment already uses, since inventing a fresh one starts exactly the second identity this prevents |
||
|
ryan-crabbe-berri
|
b76def0e5d
|
test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A TypeError from a refactor, a botched fixture, an import that moved: all of them read as the rejection the test claims to police, so the test goes green for the wrong reason and stays green after the behaviour it guards is gone. PT011 closes that gap for the 317 sites B017 could not reach, because B017 only fires on a single-statement body with no `as e` binding. Each pattern here is the message the code actually raised, recorded by running the sites under a plugin that logged the concrete type and text per call site, so the assertions describe observed behaviour rather than a guess. Where a site raises more than one message across its parametrize cases, the pattern is an alternation of what was seen; where the exception carries an empty `str()` and puts the text on `.message`, the site keeps a narrow `noqa` with the reason. PT014 removes four parametrize cases that were listed twice. The duplicate re-runs an assertion that already passed, and it usually marks a case someone meant to vary and forgot to edit. |
||
|
devin-ai-integration[bot]
|
18242aec9a
|
fix(router): isolate deployment model info (#37687)
Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
abdde94ad5
|
fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do (#37703)
* fix(ptu): refuse an incomplete config.yaml reservation the way the endpoints do POST /model/new answers 400 when PTU fields are set without a team_id, a ptu_effective_from, or the count and rate together. config.yaml ran none of those checks, so the same deployment loaded and served, billing per token while accruing no flat cost, with nothing logged. The rule moves into litellm_core_utils.ptu_pricing so both paths state it once. Registration refuses such a deployment and names it, and the proxy's ignore_invalid_deployments keeps that to the one entry. Only enforced while PTU cost attribution is enabled, so a proxy that never opted in is unchanged. * refactor(ptu): build the refusal message in the module that owns the rule router.py raised a message it composed itself, which put proxy-facing wording on the shared SDK surface. ptu_config_error now takes the deployment name and returns the whole sentence; the endpoints still ask without a name and their 400 bodies are unchanged. |
||
|
yucheng-berri
|
c2b3c4b1e4
|
feat(ptu): accrue flat cost for PTU deployments declared in config.yaml (#37556)
* feat(ptu): accrue flat cost for PTU deployments declared in config.yaml The flat-cost rollup reads deployments from LiteLLM_ProxyModelTable, and config.yaml models never reach that table by design, so a PTU deployment declared there accrued no flat cost at all while still billing its traffic per token. The provider bills the reservation whichever file declared it. The rollup now also reads the deployments the router holds that no database row owns, identified by db_model, skipping the per-request credential clones that carry original_model_id and reuse their source's PTU config under a fresh id. Registering such a deployment zeroes its pricing, since reserved capacity already pays for the traffic it serves, and leaving a rate unset falls back to the public cost map, which makes the double charge the default rather than an opt-in. The rules both halves apply now live in one module. The rollup's test for what it will charge and the router's test for what to zero have to agree, or a deployment one accepts and the other declines serves its traffic for free. That module also owns the fields the write endpoints already zero, so the two paths cannot drift: tiered_pricing is emptied rather than zeroed because its tiers outrank the rates beside them, the search context table is written zeroed because an absent one means the provider default, and any further rate the deployment itself declares is zeroed alongside the standing set. The prune is bounded to the deployments a run scanned, but only for a run that priced a config-declared deployment. Deciding a row is garbage on staleness alone stays correct while every run derives its charges from the same table, so a database-only run sweeps exactly as it did before; once one host's charges come from a file the others cannot read, a row it never considered is not evidence of anything. Behaviour change worth calling out: a zeroed deployment sorts ahead of an unpriced sibling in QualityRouter's cost tiebreak, where an unset rate previously sorted last. Reserved capacity really is the cheaper choice, but the ordering moves. * refactor(ptu): drop a Final rebind and two redundant isinstance guards The basedpyright budget rejected reassigning a Final in the datetime coercion and two isinstance calls the router entry's own type already guarantees. Filtering the built records rather than the raw entries removes both guards and leaves _router_deployment as the single validator. |
||
|
mateo-berri
|
18752c860c | fix(cost): honor explicit zero tier rates and skip synthesized backend output rates | ||
|
tin-berri
|
1911269ddf
|
fix(router): never price a strategy-router alias (#36691)
* fix: never price a strategy-router alias A strategy-router alias (auto_router/complexity_router/<name>) is never the deployment that gets called or billed, but custom pricing configured on it was being treated as real pricing in two places: - registered in litellm.model_cost under the alias deployment id, so an explicit zero made _is_cost_explicitly_configured() report the group as a genuinely free model and every budget check was skipped, while the request routed to a paid deployment and accrued real spend - copied onto request_kwargs by the alias-params merge, so the routed deployment got re-registered at the alias price and the request billed 0.0 Both are fixed at the writer, so config, /model/new and price-map reload all take the same path Co-Authored-By: Claude <noreply@anthropic.com> * chore: annotate filtered cost-map copy for the mutable-collection gate Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Yassin Kortam
|
347798b80e
|
fix(router): keep custom model_info across a price data reload (#35491)
A price data reload replaced litellm.model_cost wholesale, discarding every runtime registration: the deployment model_info the Router registers from model_list, and pricing overrides passed to litellm.register_model. Custom model groups lost max_input_tokens / max_output_tokens in /model_group/info, and a deployment whose backend model is in the catalog silently reverted to upstream values. Runtime registrations are now recorded and replayed on top of the freshly fetched catalog. Router._pre_call_checks resolved the per-deployment model name only after the model-info lookup, so an unregistered model left it unset and the supported params check ran against the bare model group name, raising "LLM Provider NOT provided" out of deployment selection. The name is now resolved first, and an unresolvable provider skips that check rather than failing the request. Resolves LIT-4675 |
||
|
mateo-berri
|
1e6a34850d | fix(router): propagate capability flags to shared backend cost map key | ||
|
mateo-berri
|
3f712e3fde | fix(router): stop custom model_info leaking onto shared backend cost map key | ||
|
Mateo Wang
|
d0c82c308d
|
fix(main): stop per-request custom pricing from clobbering shared model_cost pricing (#32163)
* fix(main): stop per-request custom pricing from clobbering shared model_cost pricing
A request routed through a wildcard deployment with explicit zero pricing
(e.g. openai/* with input_cost_per_token: 0) registered that pricing on the
shared {provider}/{model} key in litellm.model_cost, so sibling deployments
relying on built-in pricing logged $0 until process restart (LIT-3991).
Request-time registration in completion()/embedding() now mirrors the
router-startup isolation: router-originated requests register full pricing
under the deployment's unique model id only, while the shared backend key
receives the entry with custom pricing fields stripped. Direct SDK calls
without a router deployment id keep the legacy shared-key registration.
The stripping logic is shared via
CustomPricingLiteLLMParams.strip_custom_pricing_fields and reused by
Router._create_deployment and Router.add_deployment.
* test: update legacy tests that asserted per-request pricing leaking into shared model_cost
test_router_fallbacks_with_custom_model_costs asserted the shared
claude-sonnet-4-5-20250929 entry ends up with the deployment's 30/60
pricing, which is exactly the cross-deployment leak this PR removes; it
now asserts the shared key keeps the built-in pricing, matching the
test's stated goal.
test_cost_calc.py::test_run computed streaming cost via
completion_cost(response), which only matched the non-stream cost while
the shared gpt-3.5-turbo entry was poisoned with the per-request
2/token pricing; it now passes the request's custom pricing explicitly
via custom_cost_per_token.
|
||
|
mubashir1osmani
|
21cd1d1a4f
|
fix(router): isolate all per-deployment pricing overrides from sibling deployments (#31021)
* fix(router): isolate all per-deployment pricing overrides from sibling deployments CustomPricingLiteLLMParams is the authoritative set of per-deployment pricing fields, used to strip overrides from the shared backend-alias key so one deployment cannot pollute a sibling that shares the same backend model. It had drifted from ModelInfoBase: tiered and per-unit cost fields such as input_cost_per_token_above_272k_tokens, cache_read_input_token_cost_above_*, output_vector_size, ocr_cost_per_*, and the regional uplift multipliers were absent, so a deployment overriding any of them leaked the override into litellm.model_cost under the shared key and every sibling read the wrong rate via /model/info (LIT-3897). Add the missing fields so the denylist covers every ModelInfoBase pricing field, and guard against future drift with a test asserting the two stay in sync, plus a regression test that a tiered override stays isolated to its own deployment model_id key. * chore(ui): regenerate schema.d.ts for custom pricing fields |
||
|
Yassin Kortam
|
410b892f77
|
fix(register_model): preserve built-in cache pricing when registering custom overrides under unmapped keys (#30044)
* fix(spend-tracking): fall back to direct spend-counter increment when reservation reconcile fails When the reservation-reconcile path in `_reconcile_budget_reservation_for_counter_update` hits a Redis error, it now correctly returns an empty set so that `increment_spend_counters` re-runs the direct increment for the affected counters. Previously, the function logged the failure, invalidated the reserved counters, and still returned the reserved counter keys, which caused the caller to skip the direct increment. With the increment skipped and the counter deleted, the next request reseeded the counter from `LiteLLM_VerificationToken.spend`, a column the batched flusher only updates every few seconds, so the enforced cross-pod spend value collapsed to a stale snapshot and budget gating stopped firing for affected keys. Adds a regression test that exercises the failure path with a flaky redis backend and asserts the actual response cost lands in the shared counter. * fix(register_model): preserve built-in cache pricing when registering custom overrides under unmapped keys When a custom-priced model is registered under a key shape that get_model_info cannot resolve (e.g. litellm_params.model set to bedrock/bedrock/us.anthropic.claude-sonnet-4-6 or another non-canonical alias), register_model previously fell back to an empty existing_model. The merged entry then carried only the fields the user set explicitly (input/output cost, provider) and dropped cache pricing. Downstream the cost calculator defaulted cache_creation_input_token_cost and cache_read_input_token_cost to 0, silently dropping the bulk of the bill for cache-heavy Anthropic traffic. register_model now attempts to resolve a canonical built-in entry by stripping provider prefixes, region prefixes, and provider-specific suffixes before giving up. When a variant resolves, its defaults (notably cache pricing) are inherited while the user's explicit overrides still win. When nothing resolves and the user supplied no cache pricing, it logs a warning instead of silently under-billing. * fix(router): inherit built-in cache pricing on deployments with partial custom pricing A deployment configured with only input_cost_per_token and output_cost_per_token under model_info was being registered under its model_info.id with no cache cost fields. The cost calculator then defaulted cache_creation_input_token_cost and cache_read_input_token_cost to 0, silently billing cache_read and cache_creation tokens at zero. For cache-heavy Anthropic traffic this drops the bulk of the bill. When the deployment's litellm_params.model resolves to a built-in cost-map entry, pull the cache pricing fields from there before registering. User-specified cache fields still win on merge; only missing fields are inherited. Pairs with the register_model fallback added earlier in this branch: that handles unmapped key shapes like bedrock/bedrock/x, this handles deploy-id keys whose backend model is mapped. * fix(register_model): inherit only cache pricing on unmapped-key fallback, not provider The unmapped-key fallback in register_model copied the entire resolved built-in entry, so registering openai/command-r-plus inherited the cohere built-in's litellm_provider and get_model_info(custom_llm_provider=openai) could no longer resolve it. Restrict the fallback to the cache-pricing fields, matching the router-side _inherit_builtin_cache_pricing, so the cache-cost dropout stays fixed without clobbering the registered provider. Add a direct unit test for Router._inherit_builtin_cache_pricing so the router coverage check sees it, and pin the fixed spend-counter contract: when reservation reconcile fails the counter must hold the directly incremented cost rather than being left at None. |
||
|
Sameer Kankute
|
b7e978a5c3
|
Litellm oss staging 04 21 2026 2 (#26569)
* fix(bedrock): use model info lookup for output_config support instead of hardcoded check Replace hardcoded _is_claude_4_6_model() string matching with supports_output_config flag in model_prices_and_context_window.json, accessed via _supports_factory(). This follows the project's established pattern for model capability checks (per AGENTS.md rule #8). Bedrock Invoke now conditionally preserves output_config for models that declare supports_output_config=true (currently Claude 4.6 models), while stripping it for older models to avoid request rejection. Ref: https://github.com/BerriAI/litellm/issues/22797 * fix(vertex_ai): single-flight credential refresh to prevent thundering herd (#26024) * fix(vertex_ai): single-flight credential refresh to prevent thundering herd When GCP credentials expire under high concurrency, all requests simultaneously call credentials.refresh() via asyncify, saturating the 40-thread anyio pool and blocking the proxy for 20+ seconds. This adds: - Per-credential asyncio.Lock in get_access_token_async for single-flight refresh (1 coroutine refreshes, others wait on the lock) - Background refresh when token_state is STALE (usable but near expiry), returning the current token immediately with zero added latency - threading.Lock on the sync get_access_token path - Uses google-auth's TokenState enum (FRESH/STALE/INVALID) instead of reimplementing expiry logic Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: address PR review comments - Use asyncio.create_task() instead of deprecated get_event_loop().create_task() - Track in-flight background refresh tasks to prevent duplicate refreshes when multiple STALE-path callers pass through the lock before the first background task completes - Add token validation in the STALE branch (consistent with FRESH/INVALID) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: lazy-import TokenState to avoid breaking when google-auth is not installed Also extract helper methods to bring get_access_token_async under the PLR0915 statement limit (50). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: apply Black formatting to test file and update uv.lock Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove user-provided project_id from log messages (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: avoid leaking token value in error message, log type instead Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: restore uv.lock to match litellm_oss_branch Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove project_id from remaining log message (CodeQL log injection) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove remaining project_id from log and error messages Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: reuse cached credentials in VertexAIPartnerModels (#26065) * fix: reuse cached credentials in VertexAIPartnerModels instead of creating new VertexLLM per request VertexAIPartnerModels.completion() was creating a throwaway VertexLLM() instance on every call to get an access token, bypassing the credential cache inherited from VertexBase. This caused a fresh token fetch for every single request, adding significant latency overhead. Fix: call super().__init__() to initialize VertexBase's credential cache, and use self._ensure_access_token() instead of a new VertexLLM instance. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: apply same credential caching fix to VertexAIGemmaModels and VertexAIModelGardenModels Same bug as VertexAIPartnerModels: both classes had `pass` in __init__ instead of `super().__init__()`, and created throwaway VertexLLM() instances per request instead of using self._ensure_access_token(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(fireworks): add glm-5p1 metadata and parallel_tool_calls (#26069) * fix(chatgpt): preserve responses routing and recover empty output (#25403) (#26219) - preserve existing shared backend `mode` when router deployment registration reuses a provider/model key already in `litellm.model_cost` (prevents alias with `mode: chat` from downgrading shared `chatgpt/gpt-5.4` from `responses` to `chat` and triggering 403s on /v1/chat/completions) - teach the ChatGPT Responses parser to recover `response.output_item.done` entries when `response.completed.output` is empty - add defensive /responses -> /chat/completions bridge fallback that reconstructs output items from raw SSE when `raw_response.output` is empty - regression coverage for shared alias routing, empty completed.output parsing, and SSE bridge recovery Closes #25403 Co-authored-by: afoninsky <andrey.afoninsky@gmail.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(deps): relax core runtime dependency pins from exact == to ranges When litellm migrated from Poetry to uv (PR #24905, v1.83.1), the core dependency specifications in pyproject.toml changed from Poetry bare-version strings (e.g. openai = "2.30.0") to PEP 621 exact pins (openai==2.24.0). Poetry bare-version strings are actually caret ranges (^X.Y.Z == >=X.Y.Z,<X+1), but PEP 621 == is exact. This means every downstream package that installs litellm as a library dependency is now forced to downgrade aiohttp, pydantic, openai, click, and 8 other common packages to exact old versions. Fix: restore range specifiers for the 12 core runtime dependencies. The optional extras (proxy, proxy-runtime, etc.) are consumed primarily by Docker images where exact pins are appropriate and are left unchanged. The uv.lock file continues to provide exact reproducibility for Docker builds and CI. Fixes: #26154 * Add Rubrik as officially-supported guardrail plugin (#25305) * Add Rubrik as officially-supported guardrail plugin Adds tool blocking and batch logging integration with an external Rubrik webhook service. The plugin validates LLM tool calls against a policy service (fail-open on errors) and batch-logs all requests/responses. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Update Rubrik docs: config.yaml as primary, env vars as fallback Restructures the Quick Start to present config.yaml as the recommended approach with tabbed UI, and environment variables as an alternative fallback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add Rubrik env vars to config_settings reference Fixes documentation validation by adding RUBRIK_API_KEY, RUBRIK_BATCH_SIZE, RUBRIK_SAMPLING_RATE, and RUBRIK_WEBHOOK_URL to the environment settings reference table. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Add fallback message when blocking service returns empty explanation Prevents whitespace-only violation message when the tool blocking service blocks tools but returns an empty content field. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat(ocr): add Reducto parse OCR support (#26068) * feat(ocr): add Reducto parse OCR support * fix(reducto): address OCR review feedback * chore: refresh uv lockfile * Revert "chore: refresh uv lockfile" This reverts commit |
||
|
Kris Xia
|
d1d2400862
|
fix(router): register model info under responses/-stripped variant (#27531)
Squash-merged by litellm-agent from krisxia0506's PR. |
||
|
Ishaan Jaffer
|
e8461b5b97
|
style: run black formatter on files from main merge | ||
|
Shivam Rawat
|
1ee43b11de
|
[Fix] prevent shared backend model key from being polluted by per-deployment custom pricing (#20679)
* bug: custom price override for models * added associated test |