mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-07 08:26:10 +00:00
54 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
f079e4061b
|
fix(proxy): deliver budget alerts on webhook-only alerting and accept ALERTING_WEBHOOK_URL (#38441)
* fix(proxy): deliver budget alerts on webhook-only alerting and accept ALERTING_WEBHOOK_URL ProxyLogging.budget_alerts forwarded to the alerting pipeline only when 'slack' was in general_settings.alerting, so alerting: ['webhook'] plus WEBHOOK_URL silently never delivered a budget alert (the config /health/services?service=webhook exists to test). Forward when 'webhook' is present too; SlackAlerting.send_alert already fans out per channel. Also accept a provider-neutral ALERTING_WEBHOOK_URL env fallback for the Slack-format channel (any Slack-compatible receiver works), mark it as a sensitive var, and de-brand the admin UI alerting copy. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): format settings.tsx with prettier Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate schema.d.ts for updated alerting description Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: retrigger checks after ALERTING_WEBHOOK_URL docs merged Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
3e2999f29f
|
fix(proxy): run SMTP send_email off the event loop with a connection timeout (#38473)
* fix(proxy): run SMTP send_email off the event loop with a connection timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): format utils.py and update _create_smtp_connection tests for timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep malformed SMTP_TIMEOUT inside the email error boundary Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: retrigger ci Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: exclude misaligned circleci coverage flag from merged codecov report Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: retrigger ci for codecov and benchmarks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: disable carryforward for the circleci codecov flag Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: exclude carried-forward coverage from the codecov patch status Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * ci: stop carrying forward the dead circleci codecov flag Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Mateo Wang
|
5e60ec5c31
|
Merge pull request #38743 from BerriAI/litellm_techdebt_20260829
refactor: clean up tech debt that landed on 2026-08-29 |
||
|
devin-ai-integration[bot]
|
0de1825450
|
fix(health): honor allow_requests_on_db_unavailable in readiness probe (#37640)
* fix(health): honor allow_requests_on_db_unavailable in readiness probe Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): bound readiness DB check and pass reconnect timeout Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): bound whole readiness DB check with one deadline Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(health): keep readiness deadline fallback within lint budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(health): suppress TQ008 for proxy-global readiness patches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(db): release reconnect lock when a waiting reconnect is cancelled Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: milan <milan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> |
||
|
Devin AI
|
9bfb332904 |
refactor: replace fresh getattr/setattr and test type-ignores with typed access
Same-day debt cleanup on code that landed in the last 24 hours. No behavior change. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
1a5a856e3e | fix(guardrails): defer native /v1/messages stream logging until post_call scans finish | ||
|
Deepanshu Lulla
|
72f1b3e969
|
feat(guardrails): add Lakera v2 skip-message honoring and advisory (inject_system_message) mode (#34940)
* feat(guardrails): honor Lakera v2 skip-message flags and add advisory (inject_system_message) mode Squashed rebase of bugfix/lakera-v2-skip-system-tool-messages onto latest litellm_internal_staging (900+ commits ahead; a commit-by-commit rebase hit repeated conflicts against the same files across earlier review-round commits, so the branch's cumulative diff was reapplied in one pass instead). Adds skip_system_message_in_guardrail/skip_tool_message_in_guardrail support to Lakera v2, a third on_flagged: "inject_system_message" advisory mode, and the associated masking-safety-guard hardening (multimodal content, non- maskable message fields, combined messages+input, and structured Responses- API input in advisory delivery) found across this PR's review rounds. * fix(guardrails): don't let one invalid guardrail config crash proxy boot init_guardrails_v2 had no try/except around initialize_guardrail, so a guardrail whose litellm_params fail validation at construction time (for example Lakera's on_flagged=inject_system_message combined with mode=during_call, or a malformed advisory_system_message template) raised uncaught and crashed the entire proxy at startup, taking down every other, correctly-configured guardrail in the list. Catch ValueError/TypeError per guardrail, log a warning, and skip it, matching the same pattern already used for the DB-driven guardrail-creation path in guardrail_endpoints.py. * fix(guardrails): preserve message fields and mask PII before advising in Lakera v2 Mask-in-place degraded to a hard block for any message carrying a field beyond role/content (tool_call_id, tool_calls, name, cache_control), for a message excluded by skip_system_message_in_guardrail/skip_tool_message_in_guardrail, or for a message with no inspectable text, since it rewrote data["messages"] wholesale from a synthetic role/content-only list built for the Lakera API call. That made masking effectively unusable for any real tool-calling conversation and made the skip flags flip every PII-only violation to a hard block instead of masking just the in-scope text. Replace the wholesale rewrite with a scope-index merge, reusing the same merge_guardrailed_scoped_messages helper the OpenAI/Anthropic guardrail translation handlers already use for this: patch content in place on a copy of each original message actually sent to Lakera, and leave every skipped/no-text/out-of-scope message untouched at its original position. This also fixes on_flagged="inject_system_message" (advisory mode) shipping raw unmasked PII to the model: a PII-only violation is now masked the same way regardless of on_flagged, and the advisory note is reserved for flags masking can't resolve on its own. Addresses maintainer-reported regressions on BerriAI/litellm#34940. * fix(guardrails): satisfy new lint gates for the masking/advisory fix Parameterize the write-back helper's dict param and suppress the two new lint rules that landed on the base while this branch was in flight: TQ008 (patching an internal collaborator) for two pre-existing tests unrelated to this change, and LIT001 for a param that genuinely needs to mutate the caller's request dict in place. * fix(guardrails): normalize role casing in Lakera v2 masking scope, log skipped guardrails louder Greptile finding: the masking scope helper compared roles case-sensitively while filter_messages_by_skip_flags (used to build what's actually sent to Lakera) normalizes casing, so an uppercase-cased "System"/"TOOL" role survived the scope filter but was excluded from the inspected list. The resulting length mismatch raised inside the strict positional zip, turning a maskable PII-only violation into an unhandled request failure. Lowercase the role comparison to match. Also, per veria-ai's finding that a skipped invalid guardrail now fails open: log it at error level with an explicit note that the proxy is starting without that guardrail, so it's not mistaken for routine info. * fix(guardrails): mask maskable PII in mixed violations before advising in Lakera v2 on_flagged="inject_system_message" only masked when a violation was PII-only; a mixed violation (PII plus a non-PII flag like prompt injection) fell straight through to the advisory branch with the raw PII still in place, in both async_pre_call_hook and async_moderation_hook. Mask whatever Lakera returned location data for before appending or logging the advisory, so a mixed violation never ships raw PII just because something else was also flagged. Also degrade to blocking, same as block mode already does, when nothing can be safely masked at all (multimodal content, or messages combined with a Responses API input field) instead of showing an advisory note next to raw, unredacted content. Widened call_v2_guard/_mask_pii_in_messages/the write-back helper's message parameters from list to Sequence to match what's actually passed through from _filter_skipped_messages, instead of duplicating list(...) casts at every call site. * fix(guardrails): don't hard-block advisory mode for non-PII flags on non-maskable input Bugbot finding: gating the entire inject_system_message branch on is_multimodal_input hard-blocked every flagged request on Responses instructions, combined messages+input, or multimodal content, including a prompt-injection-only violation with no PII at all. Masking safety only matters when there's actual PII to mask; a violation with no PII needs no masking, so the advisory should still be delivered normally. Only degrade to blocking when the breakdown actually contains a PII detection and masking isn't safely possible. Otherwise, mask whatever's maskable (if any) and deliver the advisory as before. * fix(guardrails): require payload and breakdown for Lakera v2 advisory mode Advisory mode's mixed-violation masking safety net can only redact detected PII when Lakera's response carries both the breakdown (to detect a PII hit at all) and payload (the location data to mask by). payload=False or breakdown=False alongside on_flagged='inject_system_message' silently forwarded raw PII next to the advisory note. Reject that combination at construction and hot-reload time instead. * fix(guardrails): skip_system_message_in_guardrail must not force-block Lakera masking _has_responses_instructions treated any non-empty data["instructions"] as unsafe to mask regardless of skip_system_message_in_guardrail, even though that flag excludes the instructions-derived synthetic system message from what Lakera ever inspects. PII detected purely in the maskable non-system content was force-blocked instead of masked. Also fixes pre-existing LIT010 (missing Final) violations in _has_responses_instructions, _breakdown_has_pii_violation, and async_post_call_success_hook that the rebase's lowered budget ceiling now flags. * chore: retrigger CI (GitHub Actions runner-acquisition failure on prior push) * fix(guardrails): address maintainer review findings on Lakera v2 advisory mode - Gate advisory_system_message template validation on on_flagged= 'inject_system_message', since block/monitor mode never reads it. - Allow on_flagged='inject_system_message' with mode='during_call' at construction/hot-reload instead of rejecting it; async_moderation_hook already degrades gracefully (masks if possible, else logs a warning). - reinitialize_guardrail now restores the previous live instance when the new config fails to initialize, instead of leaving the guardrail deleted entirely with nothing enforcing it. - PATCH /guardrails/{id} rolls back the DB write and returns 422 when the in-memory sync rejects the new config, instead of persisting a config that never actually took effect and returning 200. - Qualifire now rejects on_flagged values it doesn't implement (only Lakera should accept 'inject_system_message'; LitellmParams flattens the field across every guardrail config mixin). * fix(tests): satisfy lint gates and update collateral test for advisory-mode fixes - Add match= to a too-broad pytest.raises(ValueError), and suppress the new TQ008 mocker.patch findings (same pattern already used by sibling scenarios in this test). - test_init_guardrails_v2_skips_invalid_guardrail_instead_of_crashing_boot used mode='during_call' + on_flagged='inject_system_message' as its invalid-config example; that combination is now accepted, so swap in the payload/breakdown-missing case and add a test confirming during_call advisory mode constructs successfully. * docs(CLAUDE.md): auto-capture review learnings without being asked This session found three real bugs a human maintainer caught after eight rounds of bot review and live-proxy verification all missed them. Add a standing instruction to write learnings.md entries the moment a root cause is understood, in both the repo-wide file and any relevant skill's own file, instead of relying on being asked. * feat(guardrails): add scan_raw_request flag so YAML order can't change enforcement Maintainer finding on BerriAI/litellm#34940: guardrails for the same hook run sequentially over one shared, progressively-mutated request dict, so declaring a masking guardrail before a blocking one hides the violation from it (200 vs 400 depending purely on YAML order). scan_raw_request opts a guardrail into always evaluating a snapshot taken before any guardrail in the hook ran, regardless of its declared position. Same contract as run_in_parallel: block-only, its own mutations discarded. Verified live: real proxy, real Gemini call, two custom guardrails (a redactor then a blocker). Same request, same declared order -- without the flag the blocker never sees the raw secret (200); with it, the blocker correctly rejects before any provider call (400). * fix(guardrails): harden scan_raw_request against review findings - Use safe_deep_copy instead of a bare deepcopy for the raw-request snapshot; request payloads commonly carry unpicklable objects (e.g. an otel span in metadata), which previously raised on every guarded request when tracing was enabled (Bugbot, High). - Only compute the snapshot when a guardrail actually opted in, and take it before _maybe_execute_pipelines runs, so a pipeline-mutated payload can't hide a violation from a scan_raw_request guardrail outside the pipeline (veria-ai). - Log a warning when a scan_raw_request guardrail returns a modified payload, since that mutation is discarded and the combination is otherwise silently exploitable for a masking-capable integration misconfigured this way (veria-ai). * chore(openapi): regenerate lazy snapshot and dashboard schema types The lazy OpenAPI snapshot (litellm/proxy/_lazy_openapi_snapshot.json) and the derived dashboard schema.d.ts had drifted stale relative to the guardrail config model changes across this PR's rounds (advisory mode, scan_raw_request, and upstream additions picked up by rebasing). Regenerated via the CI's own documented fix: uv run python -m litellm.proxy._lazy_openapi_snapshot npm run gen:api (via make check) * chore(openapi): pick up cache_hit_filter field after rebase * fix(guardrails): stop scan_raw_request warning from firing on every call _process_guardrail_callback always returns a dict once a guardrail runs (mark_pre_call_hook_ran unconditionally stamps bookkeeping metadata), so comparing the result to non-None warned on every request even when the guardrail never touched the payload. Compare against a bookkeeping-only baseline instead, so only an actual content mutation triggers the warning. * fix(guardrails): make scan_raw_request snapshots independent of safe_memory_mode safe_deep_copy can return the original object under litellm.safe_memory_mode, or alias a per-key reference on copy failure. Under that mode, the scan_raw_request comparison baseline aliased raw_request_snapshot (and therefore the live request), letting mark_pre_call_hook_ran write a premature execution marker that a deployment-level guardrail sharing the same name would read as "already ran" and skip. Also affected the feature's core isolation guarantee: input_data itself could alias the live request under the same mode. Replace every scan_raw_request snapshot with _independent_snapshot, which never returns an alias, only a genuine copy or None. * fix(guardrails): gate during_call mixed-violation masking behind an actual PII check The during_call branch for a mixed violation under on_flagged=inject_system_message unconditionally masked and reassigned data["messages"], even for a pure prompt-injection violation with zero PII, unlike async_pre_call_hook which already gates the same call behind _breakdown_has_pii_violation. The unconditional reassignment touched shared request state during a hook documented as racing with the concurrent LLM dispatch, for no reason when there was nothing to mask. * fix(guardrails): stop scan_raw_request from silently no-op'ing on real requests _independent_snapshot did one whole-dict copy.deepcopy and returned None on any failure. Every real proxy request carries data["litellm_logging_obj"] (a Logging instance nesting a live OTel span with a real lock) by the time pre_call_hook runs, which can never be deep-copied, so the snapshot failed on every real request and silently fell back to the live, unisolated data with no warning -- defeating the entire feature in production while every existing test (none of which set litellm_logging_obj) kept passing. Rework the helper to deep-copy each top-level key independently, falling back to the original reference only for the specific key that fails, same crash tolerance as safe_deep_copy's own per-key fallback. It never returns None now; only the keys scan_raw_request actually depends on (messages/ input, metadata/litellm_metadata) need to be genuinely independent. * fix(guardrails): block during_call when PII can't be safely masked Greptile finding (P1, security): async_moderation_hook's inject_system_message branch had no equivalent to async_pre_call_hook's degrade-to-blocking case for a PII violation on input that can't be safely masked (e.g. combined messages+input). It fell through to the advisory no-op branch and let raw, unredacted PII reach the model with no protection at all. Raising still blocks the response from reaching the caller even though during_call races with the LLM dispatch, the same mechanism on_flagged="block" already relies on for this hook, so add the same block-instead-of-advisory branch pre_call already has. * chore(lint): fix LIT002 ceiling after rebase merge conflict resolution * fix(lint): suppress genuine LIT002 hits instead of padding the ceiling My earlier rebase conflict resolution for type-discipline-budget.json's LIT002 limit was too low, then overcorrected by padding it well above the actual measured count. Root-caused instead: _independent_snapshot and the PATCH-endpoint rollback path legitimately construct plain, mutable request-payload/config dicts (matching this file's existing precedent for the same shape), so suppress those four sites with `# mutable-ok:` rather than reshaping code that must stay a plain dict by contract. Set the limit to the exact current measured total; the small remaining gap vs upstream's own committed ceiling is pre-existing drift in litellm_internal_staging itself (its own tree already measures over its committed limit), not attributable to this PR. * fix(guardrails): stamp live request when a scan_raw_request guardrail runs _run_sequential_guardrail_callback and _run_parallel_pre_call_guardrails only called mark_pre_call_hook_ran on throwaway snapshot copies for a scan_raw_request guardrail, never on the live request returned to the caller. A later async_pre_call_deployment_hook (router-level guardrail re-check) reads that marker on live kwargs to decide whether to skip re-running the same guardrail; since it was never stamped there, the guardrail ran a second time on live data, doubling the external call and re-applying whatever scan_raw_request's contract says should be discarded. * fix(guardrails): revalidate Qualifire's on_flagged on live config reload on_flagged was validated only in __init__. The base CustomGuardrail.update_in_memory_litellm_params is a generic setattr loop with no revalidation, so a live config update (PUT /guardrails/{id}, no restart) could setattr on_flagged="inject_system_message" onto a running instance, bypassing the constructor's rejection -- silently blocking every flagged request under an "advisory" label. Mirrors LakeraAIGuardrail's own update_in_memory_litellm_params override added earlier in this PR. * fix(guardrails): honor scan_raw_request for pipeline-managed guardrails A scan_raw_request=True guardrail that is itself a pipeline step never saw raw_request_snapshot: PipelineExecutor.execute_steps had no way to receive it, and pipeline-managed guardrails are fully excluded from the normal sequential/parallel loops that implement the flag. Such a guardrail silently evaluated whatever an earlier pass_data step in the same pipeline had already rewritten, defeating the flag for pipeline-managed guardrails. Moves the snapshot helper (renamed independent_snapshot) from proxy/utils.py to litellm_core_utils/core_helpers.py so pipeline_executor.py can use the same independent-copy logic without a circular import, threads raw_request_snapshot through _maybe_execute_pipelines and PipelineExecutor.execute_steps/_run_step, and discards a scan_raw_request step's returned data the same way the sequential/parallel loops already do. * chore(openapi): pick up upstream drift after rebase onto litellm_internal_staging * fix(guardrails): stop attempting PII masking during during_call in Lakera v2 Greptile finding (P1, security): during_call runs concurrently with the LLM dispatch. In the common path, the provider call already binds its messages kwarg before this guardrail's coroutine gets a chance to run, let alone before its own network round trip to Lakera completes -- masking here can never reliably reach the outgoing request, and _apply_redacted_messages_back_ preserving_fields reassigns to a new list object rather than mutating in place, so even winning the race wouldn't help. This affected both the PII-only and mixed-violation masking branches, all added in this same PR. Remove masking from async_moderation_hook entirely and let PII violations fall through to the normal on_flagged branching: block under "block" or "inject_system_message" (extending the existing multimodal-only block to cover every PII case, since masking is proven non-functional regardless of input shape), log-and-allow under "monitor" -- consistent with how every other violation type in this hook is already handled. --------- Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com> |
||
|
yuneng-jiang
|
0eb7c3ad05
|
feat(proxy): add paginated GET /public/v1/model_hub (#38636)
* refactor(proxy): move the shared list framework to a surface-neutral package
The list framework and its RFC 9457 problem machinery sat under
management_endpoints/management_v1/, which was the right home while
/management/v1 was its only consumer. The public surface is about to build
on the same framework, and a control-plane package is the wrong thing for a
public route to import.
Moves list_framework.py in full, plus everything in common.py except
MANAGEMENT_V1_PREFIX, to litellm/proxy/list_api/. Every importer is updated
directly instead of leaving re-export shims, so each symbol keeps exactly
one import path. ManagementProblem keeps its name: renaming it would touch
the app-wide exception handler and every call site for no behavioural gain.
The framework's own tests move alongside the code they cover. The fastapi
removed-name guard in test_common.py now globs both packages, so budgets.py
and spend_logs.py stay covered after leaving the framework's directory.
Pure move, no behaviour change: the 179 tests across both packages pass
unchanged.
* feat(proxy): add paginated GET /public/v1/model_hub
The public Model Hub page loads every public model group in one call.
Measured on a live proxy with 300 published groups, /public/model_hub
answers with 328 KB in a single response and the page renders all 300 rows
into the DOM. At a few thousand models that is multiple megabytes and a
page that stops responding, which is what a customer reported.
Adds GET /public/v1/model_hub, the first resource on the unauthenticated
/public/v1 surface. It is built on the shared list framework, so it gets
the {data, meta, links} envelope, RFC 9457 problems, strict unknown and
duplicate query parameter rejection, and sort validation without
reimplementing any of it. Sorting covers model_group, mode, the token
limits and the per-token costs, `q` searches model_group, and the filters
are the ones the page actually offers: mode and providers. Default sort is
alphabetical, which is what a browse list wants and what these rows can
support: they carry no creation timestamp.
/public/model_hub is untouched. The shipped UI still calls it and its
migration is a separate change, so this is purely additive alongside it.
Model hub rows are computed off the running router rather than read from a
table, so this adds InMemoryListExecutor: the same QueryPlan applied in
Python instead of rendered to SQL. It matches the SQL executors where it
counts, NULLS LAST in both sort directions and NULL satisfying no
comparison, so a filter means the same thing on either. The other three
public hubs have the same shape and can reuse it as is.
The fix itself is ordering. The endpoint being superseded reads every
latest health check and joins it against the whole model list, so paging
the response alone would have changed nothing. Here the health lookup is
an injected dependency the executor calls on the page slice, after the
filter and the sort, so it resolves health for the rows being served and
no others. PrismaClient gains a bounded read for that, next to the
unbounded one it mirrors. The regression test pins the ordering by
asserting which model groups the lookup is asked about, and fails against
an enrich-then-slice implementation.
* fix(proxy): address self-review findings on the public model hub list
Five adversarial review passes over the branch. What they found:
`is_null` was the one predicate in the in-memory executor that read a
repeated field's container instead of its elements, so a field holding only
nulls was indistinguishable from a populated one. It now lifts over elements
like every other predicate does. Not reachable through this endpoint, whose
only repeated field grants `contains` alone, but the executor is written to
be reused by the other three hubs and the inconsistency was a trap for them.
The fastapi removed-name guard globbed the framework packages but not
`public_endpoints/public_v1`, which `proxy_server` also imports unguarded at
module level, so the new package had none of the protection the test claims
to give. It now covers all three.
Regenerates the dashboard's API types, which the OpenAPI sync check requires
whenever the proxy's route surface moves. The diff is the 65 generated lines
for the new operation and nothing else; no dashboard code changes here.
Also trims comments and docstrings that argued for a decision or restated a
signature rather than explaining code, and wraps a docstring line that ran
past 120 characters.
* ci: run the relocated list framework tests in the proxy-endpoints shard
The framework's tests moved from tests/test_litellm/proxy/management_endpoints,
which the proxy-endpoints shard claims, into a new tests/test_litellm/proxy/list_api
that no shard named. Both coverage guards caught it: the semantic shards have no
catch-all bucket, so the directory would have run nowhere.
Claims it alongside management_endpoints, where the same tests ran before.
* docs(proxy): stop restating the list spec in the model hub route docstring
The docstring listed every sortable field, the page-size cap and the filter
set, all of which already live in MODEL_HUB_LIST_SPEC and all of which the
endpoint hands back in the allowed array of a rejected request. Two copies of
one spec is a prose update owed on every change to the real one.
Keeps what a caller cannot derive from the endpoint itself: what the resource
is, that it needs no authentication, and a working example. Regenerates the
dashboard types, which carry the docstring as the operation description.
* fix(proxy): reject a repeated sort field instead of sorting by it twice
sort took any number of comma-separated keys, and the in-memory executor runs
one full sorted() pass per key before slicing. Naming one allowed field N times
therefore bought N passes over every published model group, synchronously on the
event loop, from a route that needs no credentials. Measured on 300 groups:
0.001s for one key, 0.034s for a thousand, 0.166s for five thousand, and it
grows with the catalogue this endpoint exists to make large.
A repeated field cannot change the ordering, so rejecting repeats costs a caller
nothing and bounds the passes at len(sortable), a number the spec author picks
rather than the caller. That beats an arbitrary cap: no magic number, and the
bound holds for every resource built on the framework.
The tiebreaker is appended after parsing, so sorting explicitly by it stays legal.
Budgets renders one ORDER BY in SQL and never had the amplification, but the
check belongs with the rest of the sort validation rather than in one executor.
* fix(proxy): make the search disjunction one level deep by type
Two CI gates, one cause. AnyOf declared its clauses as Predicate, so both
consumers had to recurse to evaluate one: the SQL renderer through
_render/_render_all, and the in-memory executor through _holds. The recursion
detector flags the latter, and its reason is the same one this PR already ran
into once, a caller-controlled cost that shows up as CPU.
Nothing actually builds a nested AnyOf. _search_predicate is its only producer
anywhere in the repo and it emits Compare leaves, in every call site and every
test. Declaring clauses as tuple[Compare, ...] makes that a fact the type
checker keeps rather than a comment, and _holds then evaluates a disjunction of
leaves with no recursion at all.
Also marks the new health read's broad except, which the strict gate counts,
and covers the ordering comparison operators. The endpoint exposes only
eq/in/contains, so gt/gte/lt/lte were live code no test evaluated.
* fix(proxy): keep the new health read inside the type-discipline ceiling
The bounded health query added ten LIT002 violations, which pushed the
codebase total past its budget. The gate counts across the tree and compares
to the merge base, so a file already carrying debt does not absorb new
violations.
Returns an empty tuple rather than an empty list on the two no-result paths:
the signature already promises a Sequence, so that is a free two-violation
reduction and a better type. Builds prisma's order argument from a tuple of
pairs, which turns four literals into one. The three that remain are prisma's
own API shape and each carries its reason.
Both budget gates now pass against the merge base.
* fix(proxy): clear the two basedpyright errors the new route added
The type-check budget is over its ceiling on the base already, so the gate
blames any increase: reportArgumentType 2574/2564 and reportPrivateUsage
1815/1808, one each, both from this file.
fastapi types a route's tags as list[str | Enum], so the tuple was an argument
error; budgets.py has the same one and it is part of what put the rule over.
Passing a list is what the signature asks for, marked because an inline list
is a construction the discipline gate counts.
_get_model_group_info is private by name but is the shared reader the endpoint
this supersedes imports the same way, so the import carries a rule-scoped
ignore with that reason rather than a copy of the function.
basedpyright now reports zero errors across both new modules, and all three
budget gates pass against the merge base.
|
||
|
mateo-berri
|
f824ca7433 | fix(responses): run prompt hook before provider credential resolution in sync responses() | ||
|
mateo-berri
|
dbc819dc77 |
fix(prompts): apply prompt templates before routing on /v1/responses and honor ignore_prompt_manager_model
On /v1/responses the prompt template ran inside litellm.aresponses, after the router had already resolved a deployment and injected its api_key/api_base, so a prompt whose metadata.model pointed at another provider sent the old deployment's credentials cross-provider (401). The proxy now runs the prompt template for aresponses in the pre-call hook, before routing, so the router picks the deployment that matches the swapped model. As a backstop, the SDK refuses a cross-provider swap when explicit credentials are already present instead of forwarding them. ignore_prompt_manager_model and ignore_prompt_manager_optional_params saved on a prompt were only read by the generic manager, so dotprompt prompts ignored them on every endpoint. PromptManagementBase now merges the prompt spec's flags with the per-request ones for every manager, and the generic manager no longer drops caller flags when no spec is present. |
||
|
mateo-berri
|
4582496c8a |
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_decrease_anys_opus5_round2
# Conflicts: # basedpyright-code-budget.json # litellm/proxy/auth/user_api_key_auth.py # litellm/proxy/management_endpoints/team_endpoints.py # litellm/proxy/management_helpers/utils.py # ruff-strict-budget.json # tests/test_litellm/proxy/management_endpoints/test_team_endpoints.py # type-discipline-budget.json |
||
|
mateo-berri
|
f89a3693ba |
fix(responses): resolve previous_response_id for a just-written turn
The session lookup reads spend logs straight out of the database, so a follow-up sent seconds after the turn it chains off found nothing while the row was still queued in the worker that served it, and the conversation was dropped without an error. Responses calls now ask the spend-log writer to flush on its next pass instead of waiting out its poll interval, and the lookup gives a just-finished turn a short second chance. Replaying a session also accepted `input` only as a string or a single dict, so the standard list shape dropped every user turn and left the model with assistant messages alone. |
||
|
devin-ai-integration[bot]
|
f48d219c50
|
fix(guardrails): run policy pipelines when the caller sends its own metadata (/v1/messages, Claude Code) (#36889)
* fix(guardrails): resolve guardrail pipelines from the canonical metadata bucket Policy-resolved pipelines are stored in litellm_metadata on routes like /v1/messages, but the pre_call reader fell back to the caller-supplied metadata field first, so a request that sends its own top-level metadata (Claude Code sends metadata.user_id) skipped every pipeline-managed guardrail. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): drive the pipeline regression through a registered guardrail Exercise the real executor with a guardrail in litellm.callbacks instead of patching PipelineExecutor.execute_steps at class scope. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): read pipeline state from the bucket the policy engine wrote Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): type the policy pipeline state accessors Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): annotate policy pipeline state casts Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
d4a32771fd
|
fix(proxy): scan batch records with the content hooks that are not guardrails (#37786)
* fix(proxy): scan batch records with the content hooks that are not guardrails Guardrails were made to run on batch uploads by scanning each record through the pre-call hook with the walk limited to guardrails. That limit exists because the same branch carries the rate limiters and budget accounting, which must count an upload once rather than once per line. It also excluded every enforcement hook written as a plain CustomLogger, so prompt-injection detection, Azure content safety, banned keywords and the blocked-user check never saw a batch record at all. Content that is a hard 400 online reached the provider verbatim through batch. A CustomLogger now declares whether its pre-call hook judges the payload or merely counts the request. The four that judge it opt in, the walk admits them, and both short-circuits learn about them, including the one that decides whether the file is streamed off disk in the first place: a proxy configured only with one of these hooks was skipping the scan entirely. Nothing that counts a request is marked, so an upload still costs one slot and one budget check. * refactor(proxy): drop the per-hook comment the attribute contract already states * test(proxy): make the classification a ledger, and pin the wiring with a real hook The classification test listed the two non-enterprise hooks by hand, so unmarking either enterprise one changed nothing and the mutation matrix passed with both surviving. It now walks the hook registries and fails on any pre-call CustomLogger that is on neither side, which also gives the flag the forcing function it lacked: an enforcement hook added later would otherwise default to off and silently skip batch records, which is the bug being fixed here. Nothing exercised the path the bug actually lived on either, since every test raised its own exception rather than a real hook's. One test now drives the shipped prompt-injection hook through the scan, which pins the part no synthetic exception reaches: a chained exception reads as a failure to judge, so refactoring any of these hooks to `raise ... from` would turn every per-record drop into an aborted upload. Also records why a hook that rewrites the payload for routing stays unmarked, and that only the leaf class is consulted. * test(proxy): set the callback list through monkeypatch rather than writing the global |
||
|
Yassin Kortam
|
40b8300ac2
|
fix(spend): bound each spend-log write statement by row count as well as bytes (#37758)
The Prisma query engine is a separate process whose resident memory grows with what it is asked to hold and glibc never returns it, so a pod's memory floor ratchets up to its worst statement and stays there for the life of the worker. #34956 bounded a spend-log flush by payload bytes, which caps that floor when prompts are stored and does nothing when they are not: rows carrying only attribution metadata run about 1.2 KB, so a 1000-row statement is roughly 1.2 MB, the 2 MB byte budget never binds, and every statement stays at 1000 rows forever. The engine charges per row as well as per byte. Measured on a container running the same engine build (5.4.2) against real Postgres, with rows shaped like a store_prompts_in_spend_logs=false deployment, writing the same 200,000 rows: rows/statement engine RSS still resident after the flush 1000 179 MB 500 91 MB 250 41 MB 100 19 MB None of those statements came near the byte budget, so the whole difference is row count. The floor is a plateau rather than a leak: 1,000,000 rows written at 1000 per statement settles around 229 MB and stops climbing. Adds SPEND_LOG_WRITE_BATCH_MAX_ROWS, default 100, applied alongside the existing byte budget so whichever binds first splits the statement. Both are needed, since bytes are what track a prompt-carrying row and rows are what track the engine's per-row bookkeeping. One consequence worth naming: a flush now issues more statements, and a statement that fails under a poison flood costs one insert before any isolation runs, so the irreducible floor rises by the statement count. The isolation budget still caps the amplification on top of that, and the tests assert the bound derived from the configured row cap rather than a constant. |
||
|
ryan-crabbe-berri
|
b76def0e5d
|
test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A TypeError from a refactor, a botched fixture, an import that moved: all of them read as the rejection the test claims to police, so the test goes green for the wrong reason and stays green after the behaviour it guards is gone. PT011 closes that gap for the 317 sites B017 could not reach, because B017 only fires on a single-statement body with no `as e` binding. Each pattern here is the message the code actually raised, recorded by running the sites under a plugin that logged the concrete type and text per call site, so the assertions describe observed behaviour rather than a guess. Where a site raises more than one message across its parametrize cases, the pattern is an alternation of what was seen; where the exception carries an empty `str()` and puts the text on `.message`, the site keeps a narrow `noqa` with the reason. PT014 removes four parametrize cases that were listed twice. The duplicate re-runs an assertion that already passed, and it usually marks a case someone meant to vary and forgot to edit. |
||
|
mateo-berri
|
affe2b4529 | Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_azure_postgres_entra_auth | ||
|
yucheng-berri
|
3a31331435
|
fix(proxy): run pre-call guardrails on batch input file uploads (#37519)
* fix(proxy): run pre-call guardrails on batch input file uploads POST /v1/files with purpose=batch was the only route in files_endpoints that never reached pre_call_hook, so guardrails did not see batch content at all and records reached the provider unscanned. Stream the uploaded JSONL a record at a time and run each record's body through the existing pre_call_hook dispatch under the call type its url maps to, so guardrail resolution, key and team config, and the per-endpoint translations are reused rather than reimplemented. The hook gains a guardrails_only mode for this, since the same callback loop also drives rate limiters, budget hooks, prompt templates and hanging-request alerting, none of which should fire once per record. A guardrail that blocks raises its own exception, which propagates untouched so its status code survives. A record a guardrail would rewrite, a record that cannot be parsed, and a record whose url cannot be scanned all reject the upload, since silently skipping any of them is the bypass this is meant to close. Per-record redaction lands separately. The scan only runs when a guardrail that actually runs pre_call, or a guardrail pipeline, is configured, so deployments without one are byte for byte unchanged. * fix(proxy): compare the dict a batch guardrail returns, not the one it was given async_pre_call_hook may return a replacement dict instead of mutating its input, and process_pre_call_hook_response then makes that replacement the request. The scan only inspected the dict it passed in, so a guardrail that redacts by returning a copy was treated as a no-op and its record uploaded unchanged. * fix(proxy): treat a missing batch body key as different from a null one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(proxy): document the guardrails_only mode on pre_call_hook * fix(proxy): resolve a batch record's scan type from its body when the url is unfamiliar The scanner only accepted five exact urls, but callers write that field by hand and the provider transformers are far more permissive: bedrock treats any non-empty url as chat and vertex strips query strings and trailing slashes. Uploads that work today would have started failing the moment a pre-call guardrail was configured. Normalize the url before lookup and fall back to the body shape when it is unfamiliar, so a record we can still read is a record we still scan. Only a body with no messages, prompt or input is now refused, and the error says so instead of listing urls that were never the whole set. Also pins the default side of the guardrails_only gate: the hanging-request alert and prompt templating are asserted to still fire when the flag is absent. * refactor(proxy): drop batch guardrail checks the upload validation already makes check_batch_file_upload now runs first and rejects a line that does not parse, a line that is not an object, and a line missing custom_id, method, url or body, so the guardrail scan can rely on all four. Its own parse handling was unreachable through the endpoint and is gone, along with the tests for it. What is left is the case that validation does not cover, a body whose value is not an object, since it only checks that the key is present. * fix(proxy): resolve a batch record's call type from the url path, not the whole url A record naming its route in full, which is how callers actually write batch files, matched no known route, so it fell through to the body shape. A Responses record carries `input`, and that reads as an embedding, so the record was scanned as the wrong call type and any guardrail scoped to chat or Responses skipped it while the upload was accepted. Chat records survived only because their body shape happens to map back to the same call type. The url is now reduced to its path before matching. Guardrails that pick their policy from a request header, such as noma choosing an application id, saw no headers at all during the scan and fell back to a default, so a batch record could be evaluated under a different policy than the same content sent online. The sanitized headers the proxy already stores in request metadata now travel with the scan. Also drops the bare `dict` annotation, the unreachable non-dict branch on the guardrail chain's own return, and the type alias that was missing its `TypeAlias`, which together were failing the lint gate. * fix(proxy): give each batch record its own copy of the scan metadata The narrowed metadata was handed to every record as a shallow copy, so `headers` and `tags` stayed shared with the upload request and with the other records in the same window. A guardrail that writes into one of those in place, which several do to record their own bookkeeping, would have its write show up in every record scanned after it and in the request itself. The narrowing already removed the values that cannot be copied, so each record now gets a deep copy. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
5c213127e8 |
feat(proxy): authenticate to Azure Postgres with Microsoft Entra ID tokens
Azure Database for PostgreSQL Flexible Server takes a Microsoft Entra ID access token as the connection password, and those tokens last about an hour, so a proxy pointed at one dies shortly after boot unless something keeps minting fresh ones Set AZURE_POSTGRESQL_AUTH=True (or pass --azure_postgresql_auth) alongside DATABASE_HOST, DATABASE_USER, and DATABASE_NAME, and the proxy mints a token at startup, assembles the connection URL around it, and refreshes it in the background for as long as the process runs. That is the same shape IAM_TOKEN_DB_AUTH already had for AWS RDS, so the two now share one code path: a tagged union picks the minting strategy once, and the wrapper, the read replica, and the refresh loop all read the choice off it instead of each guessing from the environment. Setting both toggles is a startup error, in the chart as well as in Python The helm chart gets database.writer.useAzureEntraAuth and the matching reader knob next to the existing useIAMAuth Fixes #29661 Co-authored-by: David Balatoni <balcsida@gmail.com> |
||
|
mateo-berri
|
708ff0b910 | fix(proxy): retry end-user spend updates on Postgres deadlock instead of dropping them | ||
|
Mateo Wang
|
b69068c290
|
Merge pull request #26900 from BerriAI/litellm_model-deprecation-alerts-55bc
feat(proxy): proactive model deprecation alerts and `/model/deprecations` endpoint |
||
|
mateo
|
1e63134adb |
fix(slack_alerting): hold a pod lock so a fleet sends one deprecation alert per day
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo
|
816fa50394 |
refactor(proxy): make the deprecation loop entrypoint public and drop a dead None check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
6e7984e537
|
fix(proxy): requeue spend logs when the DB write fails with a transport error (#36716)
* fix(proxy): requeue spend logs when the DB write fails with a transport error Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): hardcode the spend log queue cap and drop the stale re-export Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): keep the spend log requeue within the type discipline budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): apply the spend log queue cap to producer appends too Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): lower the spend log queue cap to 1k and make it env configurable Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bound the spend log queue by bytes instead of row count A row cap cannot bound memory: a row carries the whole prompt under store_prompts_in_spend_logs, so a cap that rides out an outage of counter-only rows is an OOM once prompts are stored. Every enqueue and dequeue now goes through one pair that tracks what the queue costs and drops the oldest rows past a 64 MB budget. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): make the spend log queue byte budget env configurable Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): use a string default for the spend log queue byte budget env read Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): make the spend log queue byte total a public attribute The queue it accounts for is already public, and a private name only bought reportPrivateUsage errors at every call site. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: shivam <shivam@berri.ai> |
||
|
devin-ai-integration[bot]
|
865ed96765
|
fix(proxy): force prisma recreate on postgres cached-plan error (#36428)
`_query_first_with_cached_plan_fallback` recovers from Postgres's "cached plan must not change result type" by recreating the Prisma client, which drops both the server-side plans and the engine's client-side statement-name cache. Since #30183 the shared reconnect path probes the writer with `SELECT 1` first and skips the recreate when it answers, which is right for the IAM token refresh it was added for and wrong here: the connection is healthy, it is the session's prepared statements that are stale, so the probe always passes and always vetoes the recreate. Callers now pass `force_recreate` to skip that probe, and only the cached-plan fallback does. Getting past the probe is not enough on its own. Both cooldown checks would still skip the recreate for 15 seconds after any earlier reconnect, which outlives the 10 second auth retry window, so a migration landing in that window kept 503ing. `force=True` would fix that but would also let every concurrent caller of the same burst kill the engine the first one just built. The caller instead names the engine it observed before the query, and the cooldown is waived only while that engine is still the live one, so the first caller repairs the pool and the rest fall back to the normal cooldown. That engine has to be the one the query actually ran on. `query_first` is a top-level read, so with a read replica configured it is dispatched to the reader and it is the reader's prepared statements that go stale, while `writer_db` names a different engine with its own counter. The observation and the cooldown comparison both go through `read_db`, added alongside `writer_db` and backed by a `read_target` property on the routing wrapper that `__getattr__` now dispatches through so the two cannot drift. The observation carries the wrapper, not just its generation. `read_db` resolves to the reader while it is available and to the writer once it is not, and those counters are independent and both start at zero, so comparing a bare number across that switch pits one engine's counter against another's. Equal by coincidence waives the cooldown for an engine already replaced; unequal gates a caller that needs the recreate. Identity settles it, and is sound because the engine object is never re-pointed without the generation also moving. Three smaller holes on the way out. The waiver is withdrawn once a repair of that same engine has been tried and failed, so a burst collapses onto one attempt instead of each caller running its own recreate serially; the record is keyed per engine rather than counted globally, so an unrelated reconnect failure cannot suppress a stale reader's recovery and a writer failure cannot evict the reader's record. And a forced recreate that the optimistic-lock guard declines is no longer reported as a success on either the direct or the heavy path, since the routing wrapper leaves the reader untouched in that case; a decline is deliberately not counted as a failure, so the caller's own backoff still gets its waiver on the next attempt. A decline on the heavy path clears the dead-engine flag before raising. The clear after the cycle is skipped by any raise, which is right for a failure and wrong here, and the non-forced path already clears it on a decline, so this restores that policy rather than inventing one. Stranding the flag would route the next cycle back down the probe-free heavy branch, where the refreshed generation matches and the recreate kills the healthy engine a refresh just spawned, which is #29176. Clearing that flag is necessary and not sufficient. The escalation check re-arms it whenever the consecutive-failure count sits at the threshold, so a decline that left the count alone sent the very next attempt back down the same path. A decline is raised only at the generation guard, and the generation moves only after a replacement connects, so a decline is proof that a replacement succeeded and the count is reset on it. Fixes #36418 Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
59eeae374c
|
fix(mcp): expose client HTTP headers to logging callbacks and hooks (#36724)
* fix(mcp): expose client HTTP headers to logging callbacks and hooks MCP protocol tool calls built a synthetic Request with only content-type, so metadata.headers reaching logging callbacks and guardrails was empty while /mcp-rest/tools/call exposed the full set. Rebuild the synthetic request from the connection's raw headers (shared with the sampling path), and pass sanitized headers to the pre-call hook, the MCP to LLM guardrail bridge and the Responses API MCP bridge. Credential headers stay masked and proxy key headers stripped. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): strip custom proxy key and upstream MCP credential headers from logging copies Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(mcp): make client side auth header name accessor public Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): strip custom proxy key and client redaction opt-out from mcp headers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): drop custom proxy key header in the synthetic request builder Strips general_settings.litellm_key_header_name in build_synthetic_mcp_request so every caller, including sampling, is covered, and reverts passing general_settings into add_litellm_data_to_request on the tool call path since that also switches on enforced_params. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: shivam <shivam@berri.ai> |
||
|
mateo
|
a0a536216f | Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_model-deprecation-alerts-55bc | ||
|
devin-ai-integration[bot]
|
3864e12415
|
fix(spend): stop losing spend log rows when a flush is cancelled (#34826) | ||
|
mateo
|
4e7e2f53b9 |
fix(proxy): schedule the deprecation loop when a config reload enables alerting
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
be71a8fdbf
|
fix(alerting): dedupe scheduled Slack spend reports across pods (#36489)
* fix(alerting): dedupe scheduled Slack spend reports across pods Every pod ran its own weekly/monthly spend report jobs, prometheus fallback stats cron, and daily report loop, so deployments with multiple replicas or uvicorn workers received one copy per pod. Gate each scheduled send behind the shared PodLockManager redis lock. The lock is never released: its TTL (the full reporting window for the weekly interval job, whose per-pod anchors drift by boot time and jitter) doubles as a sent-this-window marker. acquire_lock returning None (no redis wired) proceeds, preserving single-pod behavior. Also generalize the pod lock could-not-acquire log line, which claimed to be about spend tracking for every consumer. Fixes #14809 * fix(alerting): harden spend report locks after adversarial review Weekly lock TTL gets an hour haircut: with ttl equal to the interval, the winner re-fires just before its own key expires, reacquires without a TTL refresh, and the key then lapses in time for a trailing pod to re-send. Job/lock ids move to litellm/constants.py per convention, and spend_report_frequency now rejects non-positive day counts, which previously coerced to an every-second schedule and would now compute a negative lock TTL that silently never sends. Adds the missing test coverage the review flagged: startup_event's pod_lock_manager wiring (identity-asserted), the prometheus closure's positive path, and the ungated immediate prometheus send pinned to exactly one await. * test(alerting): consolidate spend_report_frequency validator coverage Drops a duplicate non-positive-days test and parametrizes the survivor over the suffix half of the validator too * fix(alerting): route the startup prometheus fallback send through the pod lock Greptile caught that the boot-time send still ran once per pod when PROMETHEUS_URL is set, the same duplication class this PR removes * fix(alerting): make report lock acquisition non-reentrant Greptile caught that a pod booting within an hour of the fallback stats cron sent twice: the startup send takes the lock, then the cron fire hits acquire_lock's reacquire branch, which returns True for the holder. Window-marker gates now pass allow_reentrant=False so a live lock blocks everyone including its holder; leader-election consumers keep the reentrant default * test(proxy): give spec'd ProxyLogging mocks a db_spend_update_writer _initialize_slack_alerting_jobs now reads it for the pod lock manager, and spec=ProxyLogging blocks instance-only attributes |
||
|
mateo
|
2fe152a1d2 |
fix(proxy): only schedule the deprecation loop when alerting is configured
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
855c49d0ef | fix(proxy): skip prisma-dependent hooks when no database is attached | ||
|
devin-ai-integration[bot]
|
1a45bf9afe
|
fix(proxy): resolve entity access groups in the model listing endpoints (#36230)
* fix(proxy): resolve entity access groups in the model listing endpoints Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(proxy): reuse the fetched team object when listing models Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover key-level access group resolution in model listing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: shivam <shivam@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
4178a857ae
|
fix(spend): bound each spend-log write statement by payload bytes (#34956)
The Prisma query engine is a separate Rust process whose resident memory is a high-water mark: it grows with the payload of the largest single statement it executes and glibc never returns that memory to the OS, so a pod's memory floor ratchets up to its worst-ever write and stays there for the life of the worker. Memory-based autoscaling then reads a number that reflects the largest write the pod has ever done rather than what it is doing now. The spend-log flush handed Prisma a fixed 1000 rows per create_many. With store_prompts_in_spend_logs enabled a single row carries the full prompt and response, so one statement can be tens of megabytes and permanently costs hundreds of megabytes of RSS. Row counts cannot express that budget: the same 1000 rows range from well under a megabyte to tens of megabytes. Split each flush into statements bounded by encoded payload size (SPEND_LOG_WRITE_BATCH_MAX_BYTES, default 2MB) on top of the existing 1000-row cap. What is measured is the encoded statement, so the budget counts what actually goes on the wire: the JSON escaping of quotes and newlines, multibyte characters at their encoded width, the field names and separators a 25-column row carries, and the brackets and row separators the rows carry as one collection. Deployments that do not store prompts keep one statement per 1000 rows and are unaffected; prompt-carrying flushes get several small statements instead of one huge one. A row larger than the budget is still written on its own rather than dropped, and a row the serializer refuses counts as zero rather than raising out of the flush and dropping every row queued behind it. Splitting a flush must not multiply what a poison-row flood costs, so the poison-isolation allowance is threaded through every statement of a 1000-row group instead of being handed out fresh per statement. That is only safe because the allowance now counts failed inserts rather than every insert: the one insert a statement needs when nothing is poisoned is not charged, so a healthy flush never runs the allowance down however many statements it splits into, and a statement reached after the allowance is spent is still attempted so clean rows behind a flood still persist. Failed inserts for a group are bounded by the allowance plus one baseline insert per statement, which restores the constant-per-group ceiling the single-statement path had. Resolves LIT-4765 |
||
|
Tin Chi Lo
|
2e12614a5b |
fix(proxy): stop retrying post-send ambiguous DB errors in every spend writer
Resolves LIT-4823. An adversarial review reproduced against real Postgres that a batched increment upsert stalling past the prisma engine timeout leaves its transaction open on the pooled connection; the retry draws the same connection, its statements stack into the still-open transaction, and one commit applies both increment sets while the writer reports success. httpx.ReadTimeout is exactly that post-send case and every spend writer retried it. DB_RETRY_SAFE_ERROR_TYPES (ConnectError only, the failure that proves the statements never reached the database) is now the single owner of what a non-idempotent writer may retry. All seven entity and daily spend writer retry arms and the tool usage flush consume it. DB_CONNECTION_ERROR_TYPES is unchanged for the idempotent spend-log writer, whose create_many with skip_duplicates may safely retry the full tuple. The corruption was reproduced on update_daily_user_spend (seeded 10|100|1, expected 11|110|2, observed 12|120|3); the new policy tests pin that a ReadTimeout drops the batch loudly on the first attempt and a ConnectError still retries. |
||
|
Tin Chi Lo
|
c8b0530c30 |
fix(proxy): roll up tool spend daily instead of scanning SpendLogs
GET /v1/tool/spend served the Cost Optimization card with two raw queries
over LiteLLM_SpendLogToolIndex x LiteLLM_SpendLogs on every dashboard load;
the totals query's driving scan was all of SpendLogs in the window. Both
per-request tables reach 1M+ rows at customer scale, so the card cost
O(traffic) per view and had to be capped at 30 days.
The index writer also mined proxy_server_request.tools, i.e. tools DECLARED
in the request body, attributing each request's full spend to tools that
never ran; and all non-MCP mining ran against payload fields that are '{}'
unless store_prompts_in_spend_logs is enabled, so non-MCP coverage silently
depended on a privacy setting.
Now the spend writer builds a ToolUsageTransaction at request time from
invoked tools only, resolved by the shared get_tool_calls_from_response
normalizer so every response surface (chat completions, Responses API,
Anthropic Messages) is covered; the tool registry's response arm delegates
to the same owner. Transactions queue beside the spend-log queue and the
flush job writes index rows plus a new LiteLLM_DailyToolSpend rollup
(date, tool_name PK) in one transaction, retrying connection errors with
backoff (a failed batch commits nothing, so the retry cannot double-count)
and dropping the batch with an error log on anything else.
The endpoint aggregates in SQL: by_tool is the top TOOL_SPEND_TOP_TOOLS
tools by spend via group_by and daily covers only those tools, so the
response is bounded by days x TOOL_SPEND_TOP_TOOLS regardless of range or
tool-name cardinality; the 30-day clamp is gone. total_spend is dropped
from the response; it was never rendered and its deduplicated semantics
are not computable from a rollup. Spend-log retention deliberately does
not touch the rollup, so tool spend history outlives per-request rows.
|
||
|
hcl
|
2a55d23731
|
fix(proxy): merge model-level guardrails before pre_call_hook (#29654)
* fix(proxy): merge model-level guardrails before pre_call_hook DB/UI-assigned guardrails (litellm_params.guardrails) only fire on post_call paths today: _check_and_merge_model_level_guardrails is called in utils.py:2234 + utils.py:2498 + common_request_processing.py:1665, but never before pre_call_hook in common_request_processing.py:963. PR #23774 fixed the non-streaming post_call case; pre_call was left broken. At the pre_call site, add_litellm_data_to_request strips client-supplied metadata.model_info (pricing spoofing guard) and route_request hasn't run yet, so model_info.id is unavailable. Extend the helper to fall back to llm_router.get_deployment_by_model_group_name(model_alias) when model_id is missing — that uses the O(1) model-name index already maintained by the router. Closes #29652 * fix(mcp): surface mcp_server_name in synthetic _convert_mcp_to_llm_format payload Addresses veria-ai Medium finding + proxy-infra CI failure on this PR. ParallelRequestLimiterV3 reads data["mcp_server_name"] for call_mcp_tool hook payloads when applying key/team mcp_rpm_limit. _convert_mcp_to_llm_format was omitting the field, so a key with mcp_rpm_limit could exceed it via the MCP path. Reads from kwargs.get("mcp_rate_limit_server_name") to match how pre_call_tool_check resolves the alias-then-server-name fallback before invoking hooks. * fix(proxy): union guardrails across group deployments on alias fallback Addresses second veria-ai Medium on #29654: the alias fallback called get_deployment_by_model_group_name(), which returns ONE deployment. A guardrail set on a non-first deployment would silently not run on pre_call when the model_id is missing. Switch to get_model_list(model_name=...) and take the UNION of litellm_params.guardrails across all matching deployments (with dedup). Trade-off documented in the comment: pre_call cannot know which deployment route_request will select, so the conservative choice is to apply any guardrail set on any eligible deployment. Updated test stubs to use get_model_list. Added 3 new tests covering union, dedup, and the all-empty case. * test(model_level_guardrails): align integration test with get_model_list union API * fix(proxy): ignore client-supplied model_info.id on pre_call merge + lint Addresses 3rd veria-ai Medium on #29654: add_litellm_data_to_request preserves client-supplied metadata.model_info when the caller's key/team has allow_client_pricing_override. The pre_call merge previously trusted that id, so a caller could spoof an unguarded model_info.id while requesting a guarded alias and bypass guardrails. New `trust_client_model_info: bool` param on the helper. The pre_call call site passes False; post_call paths (existing) keep True. Also fixes the ruff failure on the union loop: pulled the .get() into a local + isinstance(list) check before iterating, so mypy stops complaining about `object` not being iterable. 2 new regression tests covering spoof-and-bypass + default-trust behavior. * fix(proxy): pass team_id to alias-lookup + restore scalar-string guardrail acceptance Addresses two more reviewer findings on #29654: veria-ai Medium: route_request resolves team-scoped public model names with metadata.user_api_key_team_id. The pre_call alias fallback called get_model_list(model_name=...) without the team_id, so team-scoped deployments were invisible and their pre_call guardrails silently skipped. Now reads team_id from metadata or litellm_metadata and passes it to get_model_list. greptile P1: the isinstance(deployment_guardrails, list) guard added for mypy narrowing silently dropped bare-string guardrail values that the existing post_call path used to truthy-accept. Restored by wrapping a scalar string into a one-element list on both paths. 4 new tests: team_id passthrough (metadata + litellm_metadata), scalar on post_call, scalar on alias-union. 36/36 tests pass. * style: black formatting on _check_and_merge_model_level_guardrails team_id assignment * chore: ruff format * fix(lint): remove unused noqa PLR0915 directive RUF100 flags the # noqa: PLR0915 on common_processing_pre_call_logic because PLR0915 is not in this repo's enabled ruff rule set (lint.extend-select in ruff.toml), so the directive suppresses nothing and fails the lint job. * refactor(proxy): hoist guardrail-merge import to module top The pre_call guardrail-merge helper was imported inside common_processing_pre_call_logic with a # noqa: PLC0415, which the type-discipline gate counts as an unexplained suppression (LIT003). The inline import's cyclic-import justification does not hold: this module already imports from litellm.proxy.utils at top level, and utils.py does not import common_request_processing at module load. Fold the helper into the existing top-level import and drop the inline import, clearing the suppression instead of budgeting for it. --------- Co-authored-by: Yassin Kortam <yassin.kortam@gmail.com> |
||
|
Noah Nistler
|
8177230a29
|
feat(guardrails): add run_in_parallel opt-in for concurrent pre_call and post_call guardrails (#33770)
* feat(guardrails): add run_in_parallel opt-in for concurrent pre_call guardrails Pre-call guardrails run sequentially because each may mutate the request payload and later guardrails depend on earlier mutations. Deployments with several slow block-only pre_call guardrails (external moderation, Bedrock, LLM-judge) therefore pay the sum of their latencies. during_call guardrails run concurrently but alongside the LLM call, so a violating payload has already been sent, which is unacceptable when the request must never reach the model. This adds a per-guardrail run_in_parallel flag (default off). Guardrails that opt in are pulled out of the sequential loop and run concurrently via asyncio.gather after every sequential (payload-mutating) guardrail has run, so they observe the mutated payload and still form a hard barrier before the LLM call; the first to raise blocks the request. Their returned data is discarded since they are declared block-only. The flag is wired from LitellmParams onto the guardrail instance at the same generic choke point in initialize_guardrail that already sets skip_system_message_in_guardrail, so no per-provider initializer needs to change. * feat(guardrails): extend run_in_parallel opt-in to post_call guardrails post_call_success_hook ran guardrails sequentially for the same reason pre_call did: response-modifying guardrails thread the response forward. But block-only output scanners (which read the response and reject on violation without changing it) serialize for no benefit and add latency. This reuses the existing run_in_parallel flag for the post_call hook. Opted-in post_call guardrails are pulled out of the sequential loop and run concurrently via asyncio.gather after the sequential (response-modifying) guardrails and before the non-guardrail CustomLogger callbacks, so they inspect the final response and still block it from reaching the client if any raises. Their returned response is discarded since they are block-only. The apply_guardrail path sets data["guardrail_to_apply"] immediately before awaiting, and unified_guardrail pops it before its first suspension point, so concurrent guardrails never race on that key under asyncio's cooperative scheduling. * fix(guardrails): await all parallel guardrails and prioritize blocks over reroutes Addresses review feedback on the run_in_parallel opt-in. asyncio.gather propagated the first exception without cancelling or awaiting the siblings, so a block at t=0 left the other guardrails running as unobserved background tasks (wasted external calls plus event-loop warnings), and a fast SensitiveDataRouteException/ModifyResponseException could return a reroute or passthrough before a slower block finished, letting crafted input bypass the block. Both the pre_call and post_call parallel batches now gather with return_exceptions=True so every guardrail runs to completion, then raise any blocking exception ahead of a flow-changing one. The registry choke point wrote bool(None)==False onto every instance when the config omitted run_in_parallel, silently disabling a constructor-set default; it now only writes when the config provides an explicit value. * fix(guardrails): record lifecycle logs for every concurrently-run guardrail The log_guardrail_information decorator skipped its auto-record when it saw that the count of standard_logging_guardrail_information entries in the shared request_data had grown during the wrapped call, taking that as proof the wrapped function had recorded its own richer entry. That heuristic breaks the moment guardrails run concurrently (parallel pre_call/post_call, during_call): a sibling guardrail's append inflates the shared count, so a guardrail that did not self-record wrongly concludes it already did and drops its own entry. The result is that enabling run_in_parallel silently loses per-guardrail lifecycle logs, so the Admin UI Request Lifecycle timeline and downstream loggers (Datadog, Langfuse, OTEL, spend logs) show only one of the concurrent guardrails. Replace the shared-count heuristic with a ContextVar flag set when a guardrail records its own entry. asyncio copies the context into each gathered task, so the flag is isolated per concurrent guardrail while still catching the self-record-then-skip-auto-record case within a single invocation. * test(guardrails): declare run_in_parallel on post_call guardrail mocks The post_call partition reads run_in_parallel on every CustomGuardrail callback. A MagicMock(spec=CustomGuardrail) has no run_in_parallel (it is set in __init__, not on the class) so the attribute access raised, and even a class-level default would return a truthy child mock that wrongly routes the double into the parallel batch. Declare the flag False on the shared mock factories so these pre-existing hook tests exercise the sequential path they assert on. * fix(guardrails): harden run_in_parallel reads and address review feedback Read run_in_parallel via getattr(..., False) in the pre_call and post_call partitions so a third-party CustomGuardrail subclass that overrides __init__ without chaining super().__init__() no longer raises AttributeError on a path that previously worked. Drop the redundant in-function GuardrailEventHooks import in _run_parallel_post_call_guardrails (already imported module-level). Remove the flaky wall-clock upper-bound assertions from the two concurrency tests; the all-start-before-any-end overlap assertion is the timing-independent signal that actually proves concurrency. |
||
|
devin-ai-integration[bot]
|
8536e3b80e
|
fix(proxy): source /v1/models token limits from the cost map instead of Router.get_model_group_info (#33721)
* fix(proxy): source /v1/models token limits from cost map instead of Router.get_model_group_info Resolves the per-model get_model_group_info fan-out on GET /v1/models (and /models) that pegged the event loop on wildcard listings (#33636). create_model_info_response now reads max_input_tokens/max_output_tokens from litellm.get_model_info (the static cost map) rather than the router, which aggregated and deepcopied every deployment in a group per listed model. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): inject model-info lookup into create_model_info_response for deterministic coverage Inject the cost-map lookup (defaulting to litellm.get_model_info) so the except and max_output_tokens branches are exercised deterministically and the token-limit tests no longer hardcode mutable cost-map values. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(proxy): surface custom deployment token limits on /v1/models via cheap index lookup Add Router.get_configured_token_limits, an O(1) model-name index lookup that reads a concrete deployment's configured max_input_tokens/max_output_tokens without triggering pattern matching or deep copies. create_model_info_response layers this over the cost map so custom deployments absent from the cost map still surface their limits, and admin-configured limits override cost-map defaults, while wildcard-expanded names stay on the fast path. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
devin-ai-integration[bot]
|
f7fc679f27
|
fix(logging): preserve callback order in get_combined_callback_list (#33005)
Replace list(set(...)) dedupe with dict.fromkeys so callback insertion order is preserved deterministically instead of being randomized by set iteration order (influenced by PYTHONHASHSEED). Applies to both the Logging and ProxyLogging implementations. Fixes #33003 Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
yucheng-berri
|
f31dacbcd4
|
fix(proxy): never log raw virtual keys in key insertion debug output (#33268)
* fix(proxy): never log raw virtual keys in key insertion debug output * fix(proxy): tolerate None token in insert_data debug log redaction |
||
|
Yassin Kortam
|
8417b962a2 |
fix(proxy): recover Prisma DB reconnect loop when client is disconnected
Once the active Prisma client is in the disconnected state, every DB call raises ClientNotConnectedError. The reconnect machinery was supposed to recover from this, but _get_engine_pid() inspected the broken client via prisma's _engine property, which re-raises that same error, so recreate_prisma_client failed before it could build a replacement client and the proxy looped on failed reconnects forever (issue #28322 showed 1486+ consecutive failures over 30 days with zero recoveries) Guard both _get_engine_pid implementations with is_connected() so a disconnected client reads as "no engine" (pid 0) and the recreate path proceeds to construct and connect a fresh client |
||
|
Yassin Kortam
|
52dc15adfe
|
fix(proxy): isolate poison spend-log rows so one bad record can't drop the whole batch (#31705)
update_spend_logs flushes the queue with a single create_many per batch, so one row carrying bytes Postgres refuses (a residual NUL byte is the canonical case) fails the entire insert and drops every good spend log alongside it. PR #29515 strips NUL bytes from the JSON columns, but the scalar string columns (end_user, model, session_id, ...) still flow through unsanitized, so a poisoned row can still reach the write and take a batch of up to 1000 good rows down with it. On a genuine data-layer rejection the batch is now bisected so the good rows still persist and only the offending row is dropped and logged with its request_id. The classification lives in PrismaDBExceptionHandler.is_prisma_data_error (matched by exact type so systemic subclasses like a missing table are not mistaken for a single poison row), which keeps prisma an in-function import and litellm.proxy.utils importable without the proxy extra. Transport failures, including the "can't reach database server" outage that prisma mislabels as a DataError, are re-raised unchanged so the existing connection-retry path still runs and a transient outage never turns into silent per-row data loss. The bisection carries a per-batch isolation budget so an authenticated caller flooding poisoned rows cannot amplify one failed bulk insert into ~2N failed inserts and N log lines; once the budget is spent the still-failing remainder is dropped wholesale under a single log line. Resolves LIT-4103 |
||
|
Yassin Kortam
|
c14329128b
|
fix(guardrails): match policy-pipeline block response to direct guardrail attachment (#31421)
When a guardrail blocked a request through a flow-builder policy pipeline, the proxy discarded the guardrail's own exception and synthesized a generic guardrail_pipeline_error response, so the same guardrail produced a different HTTP response and trace span depending on whether it was attached directly or via a policy. The pipeline now carries the guardrail's original exception and re-raises it verbatim on block, enriching it with the blocking guardrail's name and mode exactly as the direct path does, so the two attachment methods are indistinguishable to clients and tracing. The generic pipeline error remains only as a fallback for blocks with no underlying exception (e.g. a guardrail that could not be found). Resolves LIT-4041 |
||
|
Yassin Kortam
|
f2fa23b0ec
|
fix(guardrails): instrument during-call and post-call guardrail latency (#31414)
litellm_guardrail_latency_seconds was only emitted for pre-call guardrails. during_call_hook and post_call_success_hook ran guardrails without recording any latency, so during-call and post-call guardrail time was invisible in the metric and leaked into litellm_overhead_latency_metric, making the documented "subtract guardrail latency from overhead" workaround under-report total guardrail time. Extract the find-the-PrometheusLogger-and-record step into _emit_guardrail_metrics and add _run_guardrail_with_metrics, a single wrapper that times a guardrail coroutine, classifies its outcome (success / intervened / error), enriches any raised HTTPException, and records the latency under the given hook_type. Route the pre-call emit, during_call_hook, and post_call_success_hook through it so every guardrail phase contributes to the metric the same way. Resolves LIT-3999 |
||
|
yuneng-jiang
|
6f6aec2930
|
fix(proxy): serialize team budget_limits to JSON in jsonify_team_object (#31045)
POST /team/new with any budget_limits returned 500 because jsonify_team_object serialized members_with_roles but left budget_limits as a raw Python list, which Prisma's Json column rejects. /team/update and /key/generate worked only because each json.dumps the windows itself. Serialize budget_limits in the shared helper, guarded by isinstance(list) so the pre-serialized /team/update path is unaffected. |
||
|
Sameer Kankute
|
4c25b7a13d
|
chore: litellm oss staging (#30745)
* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708) OpenAI GPT-5 models require max_completion_tokens >= 16. Health checks were using 5 (proxy/health_check.py) and 10 (health_check_helpers.py), causing failures on GPT-5 models. Fixes #23836 * fix: increase health check max_tokens from 5 to 16 (#23836) (#26610) GPT-5 models enforce a minimum of 16 for max_output_tokens. The current default of 5 still causes health checks to fail for these models. Bump the non-wildcard default to 16 — the smallest value that satisfies all known provider minimums while keeping health checks lightweight. Also tightens the wildcard test assertion from a weak disjunctive check to strict key-absence. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696) * fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema. * fix: remove async keyword from test. * fix: make Bedrock Mantle Responses routing data-driven per model (#30700) * Make Bedrock Mantle Responses routing data-driven per model Route Bedrock Mantle models to the native Responses API based on each model's price-map capability signal instead of a hardcoded model-name heuristic, and derive the OpenAI-compatible base path segment per model. Responses dispatch now selects the native config when the model advertises responses support (/v1/responses in supported_endpoints, or mode=responses), both overridable via register_model and proxy model_info. This enables native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing chat-completions emulation. Capability is per-model, so gpt-oss-120b routes natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss substring. The wire path is a separate concern, driven by the existing use_openai_responses_path flag rather than a model-name match: gpt-5.x and gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat config now derives its base from the same flag, fixing gemma-4 chat-completions requests that previously went to /v1 instead of /openai/v1. Cost maps: add supported_endpoints to the gpt-oss entries (responses for the non-safeguard variants, chat-only for safeguard) and supported_endpoints + use_openai_responses_path to all three gemma-4 entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address review: move capability helper into bedrock_mantle package Move the Responses capability check out of utils.py into litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses, alongside its companion wire-path helper mantle_base_segment. Both are now pure functions of (model, model_cost): the price-map mode/supported_endpoints read replaces the get_model_info call, so the rules are unit-testable without patching global state and the Bedrock Mantle package is self-contained. Use str | None instead of Optional[str] on the new signatures to satisfy the ruff UP045 strict-rule gate. Add direct unit tests for both helpers. Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b now legitimately supports Responses, so it can no longer be the "None after restore" vehicle; use the chat-only safeguard variant, which isolates the register/restore effect from the model's own capability. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366) * fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect. Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure. Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme. Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string. Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection. * fix: resolve CI failures and proxy DB URL typing issue * fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653) The tiered cost calculator resolved a tier's per-token cost with `tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or` short-circuits on any falsy value, a tier that legitimately prices a component at 0.0 (e.g. a free-cache-read tier with cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated as missing and silently billed at the full fallback rate (input_cost_per_token / output_cost_per_token). The flat-pricing path in the same module already handles this correctly with an `is None` guard. Resolve tier costs through a small helper that mirrors it, so 0.0 is honored at both the in-range and overflow sites. No shipped model currently has a 0.0 tier cost, so this is a latent defect; the fix makes the tiered path consistent with the flat path and prevents over-charging the first time such a tier appears. Adds unit tests covering the in-range and overflow paths, and drops an unused import flagged by ruff in the touched test file. * feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507) * fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618) In the messages->chat/completions bridge, translate_anthropic_tools_to_openai merged every non-mapped tool key into the function parameters dict. The Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object' -> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type). Exclude 'type' from the passthrough. Fixes #30557. * fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183) An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the running query-engine and spawns a new one. That planned kill was indistinguishable from a crash, and three reconnect paths used two uncoordinated locks, so a single refresh triggered a cascade of engine kill/respawn cycles: 1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old engine, spawn new one. 2. The engine-death watcher sees that kill, assumes a crash, and calls `attempt_db_reconnect(force=True)` (a different lock, `_db_reconnect_lock`) -> recreate again -> kills the fresh engine. 3. In-flight queries failing during the swap are classified as transport errors and trigger their own `attempt_db_reconnect` -> recreate again. Fix coordinates planned restarts across the wrapper and the watcher: - PrismaWrapper records the old engine PID in `_expected_engine_deaths` before killing it; all four watcher death-detectors (waitpid thread, pidfd, already-dead probe, os.kill poll) consume that PID and skip the reconnect instead of treating it as a crash. - `recreate_prisma_client` now serializes through `_reconnection_lock` and bumps a monotonic `_engine_generation`. Callers pass `expected_generation` as an optimistic-lock token, so racing/cascading recreates collapse into a single restart (losers no-op). This closes the two-lock gap. - The direct reconnect path probes the writer with SELECT 1 before recreating; a healthy connection (e.g. engine already replaced by a refresh) skips the recreate entirely. - `_safe_refresh_token` coalesces: it skips when the current token still has more than the refresh buffer of runway, so stacked triggers (proactive loop + __getattr__ fallback) don't each restart the engine. An `on_engine_replaced` hook re-arms the watcher on the new PID. RoutingPrismaWrapper forwards `expected_generation` and skips recreating the reader when the writer recreate was skipped. * feat(bedrock): support file content retrieval for batch output files (#30595) Implements transform_file_content_request and transform_file_content_response in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch files. The request transform resolves the file id (direct s3:// URI or base64 unified id) to its S3 object, validates bucket and key prefix against the server-configured bucket, and SigV4-signs an S3 GetObject using the same credential and region resolution as the existing upload path. The credential and region params are validated into a typed model at the boundary, so the only untyped values left are the botocore signing primitives. Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries s3_bucket_name (previously dropped when building deployment credentials) and the managed-files hook passes the deployment credential snapshot when routing afile_content, so unified-id content retrieval works with per-model bucket config instead of only the AWS_S3_BUCKET_NAME env var. Preserves managed-file access control: the proxy file-content endpoint now rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the owner/team check that only runs for unified ids and let a caller read another tenant's batch output by its object key. Managed outputs are reachable only through their unified file id. The afile_content "not found" error now reports the caller's unified id rather than the resolved internal S3 URI. Fixes #16186, #15563 * fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646) * fix(oci): map Cohere tool array/object params to lowercase builtins OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare "List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema arrays. MLflow {{trace}} judges trip this: their tools (get_root_span, get_span) take an attributes_to_fetch array. The lowercase builtins list/dict are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but both are lowercased for consistency). Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest). Adds a unit regression on the transformed parameterDefinitions plus a gated integration test exercising an array-param tool end to end. * fix(oci): make Cohere agentic tool-calling continuation work Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges drive once a tool has been executed and its result is fed back. Request side: litellm pulled the last user message into the top-level `message` and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that ("cannot specify message if the last entry in chat history contains tool results"), and an empty message alone is rejected too ("message must be at least 1 token long or tool results must be specified"). OCI carries the current turn's results in a dedicated top-level `toolResults` field. The Cohere transform now sends an empty message, keeps the user turn in chatHistory, and puts the results in `toolResults`, matching the langchain-oracle reference. Tool results are no longer represented as chatHistory entries. Response side: tool-grounded answers come back with citations carrying `documentIds` (camelCase) and no `document_ids`, which made the required `CohereCitation.document_ids` field fail validation and sink the whole response parse. Those citations are never surfaced, so the field (and CohereSearchQuery's generation_id) is now optional. Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest), single and multi-round tool loops. Adds unit regressions on the transformed request shape and on citation parsing, plus gated integration tests for the continuation. * feat: integrate Repelloai Argus guardrail (#30673) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * refactor(guardrails/repello): clean up typing and remove lint-any workarounds - Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout - Use dict[str, object] instead of bare dict in all signatures - Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly - Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel - Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks - Use TypeAdapter.validate_json() instead of response.json() + manual dict construction - Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any - Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check - Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType - Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel] * fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning - Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate - Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks * refactor: modifications for lint check * feat: add Pinstripes as an OpenAI-compatible provider (#30567) * feat: add Pinstripes as an OpenAI-compatible provider Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.) with per-token pricing and no subscriptions. Changes: - `litellm/llms/openai_like/providers.json`: register pinstripes with base_url, api_key_env, and max_completion_tokens→max_tokens mapping - `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders - `litellm/constants.py`: add to openai_compatible_providers and openai_compatible_endpoints lists - `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect provider when api_base is "https://pinstripes.io/v1" - `provider_endpoints_support.json`: document supported endpoints - `tests/`: 7 unit tests covering provider registration, resolution, URL auto-detection, api_base override, and Router config Usage: import litellm response = litellm.completion( model="pinstripes/ps/glm-4.5-air", messages=[{"role": "user", "content": "Hello"}], api_key=os.environ["PINSTRIPES_API_KEY"], ) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): resolve Greptile P1 review comments - Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works - Set responses: false in provider_endpoints_support.json — not actually wired up - Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): add api_base_env and correct responses capability - Add api_base_env: PINSTRIPES_API_BASE to providers.json - Set responses: false in provider_endpoints_support.json Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): wire up Responses API — add supported_endpoints Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so JSONProviderRegistry.supports_responses_api returns true correctly, matching what provider_endpoints_support.json advertises. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(pinstripes): enable embeddings endpoint Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings. Add /v1/embeddings to supported_endpoints and set embeddings: true. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json Matches the file's existing convention. Flagged by Greptile review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pinstripes): set a2a: false — A2A protocol not implemented All comparable JSON-configured providers (tensormesh, parasail, empiriolabs, libertai, neosantara) have a2a: false. Pinstripes does not implement the Google A2A protocol, so this should be false to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: inference_provider <max@redactedlab.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(rag): attach existing OpenAI file ids (#30628) * fix(rag): attach existing OpenAI file ids * chore: use modern typing in rag ingest fix * chore: retrigger ci * fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341) cache_control_injection_points was only consumed by the chat/completions prompt-management hook; on the native Anthropic /v1/messages path it was forwarded unused, so deployment-level cache injection was silently dropped (cache_creation_input_tokens stayed 0 for Anthropic-native clients). Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject cache_control at block level for system / tools / message locations (the only forms /v1/messages accepts), wire it into the native anthropic_messages handler, and pop the param so it does not leak upstream as an unknown field. A {location: message, role: system} config is redirected to the top-level system prompt so the same YAML works on both endpoints. Injection respects Anthropic's 4-block cache_control limit shared across system, tools, and messages: client-supplied markers count toward the cap and are never overwritten, a slot is reserved per Bedrock tool_config point, and injection stops once the budget is exhausted. Locations this path cannot represent (tool_config) are forwarded downstream instead of being silently consumed, mirroring get_chat_completion_prompt's remaining_points pass-through. Built on litellm_internal_staging. Refs BerriAI/litellm#30293 * fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522) * fix(proxy): release budget reservation on cancel when no chunk was delivered The pre-call budget reservation increments the cross-pod spend counter by a request's worst-case cost, then reconciles it on success (cost callback) or error (failure hook). A client disconnect or timeout cancels the request and surfaces as CancelledError / GeneratorExit, which neither path catches, so the reservation leaks. Under a retry storm the leaked holds accumulate, pin the counter above real spend, and return spurious 429 "Budget has been exceeded" to keys whose spend is far below budget; the counter only recovers when its TTL lapses, so the failure is intermittent and self-healing. Release the reservation in async_streaming_data_generator (which the Anthropic and Google SSE generators delegate to) on the (CancelledError, GeneratorExit) path, alongside the existing max_parallel_requests release. release_budget_ reservation_on_cancel runs under asyncio.shield so it completes despite the in-progress cancellation, is guarded by the reservation's finalized flag, and swallows a failing release so it cannot replace the in-flight cancellation. The refund is gated on whether a chunk reached the client. The flag is set immediately before the yield, after the slow-path hook await: an async generator suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk sees it True (keep the hold), while a cancellation during the slow-path await leaves it False (refund, nothing sent). A non-streaming cancellation delivers nothing and a completed non-streaming response is reconciled by the success callback, so neither needs a release here. Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): reconcile a cancelled reservation to input cost, not zero A streaming request cancelled before the first chunk previously reconciled its reservation to zero and finalized it. But by the time the generator is consuming the response the provider call was already dispatched, so the input tokens were billed even though no chunk reached the client, and the success/failure cost callbacks are skipped on cancellation. Refunding to zero let a caller send an expensive request and abort pre-token to dodge the input charge. Compute the request's input-token cost at reservation time and reconcile the cancelled reservation to it instead of zero. The worst-case output portion of the reservation is still released (so a legitimate mid-flight cancellation no longer pins the counter and 429s the key), while the input the provider already processed is charged. --------- Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(caching): encode object name in GCS cache GET path (#30378) GCS cache reads always missed when gcs_path was set. The GET methods interpolated the object name directly into the URL path, while the GCS JSON API requires it to be URL-encoded (a "/" must be sent as %2F). With gcs_path configured the object name is "<prefix>/<sha256>", so the raw slash produced a malformed object path and GCS returned 404. httpx does not raise on 4xx, so the status_code == 200 check fell through and get/async_get returned None, silently missing on every read. Without gcs_path the key has no slash, which is why this went unnoticed. Wrap the object name with urllib.parse.quote(..., safe="") in get_cache and async_get_cache. Apply the same encoding to the name= query parameter in set_cache and async_set_cache so the key written matches the key read back. Adds regression tests asserting the GET path and SET query are encoded (%2F) when gcs_path is set, for both sync and async paths; these fail on the unpatched code. Fixes #30377 * chore: add soniox stt-async-v5 model (#30672) * fix(proxy): include model group aliases in v1 model info (#30626) * Include model group aliases in v1 model info * Fix model info alias implementation * removed extra blank line * chore: rerun CI * fix(lint): remove redundant noqa directive in proxy_cli.py * fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme * Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme" This reverts commit |
||
|
Sameer Kankute
|
e33e2917c6
|
chore: litellm oss 170626 (#30637)
* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (#30089) * fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets, /realtime/calls and their /v1 + /openai/v1 variants) to LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as LLM API routes. Without this, non-admin virtual keys received 401 'Only proxy admin can be used to generate, delete, update info for new keys/users/teams' when calling these endpoints. Fixes #29923 * fix(proxy): validate session.model for realtime routes in model-access check The GA Realtime WebRTC HTTP routes resolve the effective model from the nested session.model (falling back to the top-level model), but the auth layer's get_model_from_request() only extracted the top-level model. A model-restricted virtual key could therefore place a disallowed model in session.model, leave the top-level model unset, and skip can_key_call_model() entirely - obtaining an ephemeral token for a model it is not allowed to use. Extract session.model for the realtime client_secrets/calls routes so the model-access check runs against the model the request will actually use. Legitimate callers are unaffected; their permitted model still validates. Relates to https://github.com/BerriAI/litellm/issues/29923 * fix(proxy): classify realtime transcription_sessions routes as LLM API routes Add the GA Realtime WebRTC transcription_sessions HTTP routes to openai_routes so is_llm_api_route() returns True for them, matching the client_secrets and calls routes already fixed. These endpoints are registered with user_api_key_auth in realtime_endpoints/endpoints.py, so without this a non-admin virtual key calling POST /v1/realtime/transcription_sessions would hit the admin-only 401 branch. Extends the regression test parametrization accordingly. --------- Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com> * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (#30272) * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models * fix(proxy): degrade /v1/models gracefully when model-group lookup fails --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: sort tiered token-cost thresholds numerically (#30375) * fix: sort tiered token-cost thresholds numerically _get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a lexicographic sort, so for tiers whose thresholds have different digit lengths (e.g. 90k vs 128k) a request crossing both was billed at the lower tier that sorted first. Sort by the parsed numeric threshold instead, so the highest tier the request actually crosses is applied. * refactor: reuse _parse_above_token_threshold for inline threshold parse --------- Co-authored-by: Eric (GabiDevFamily) <271972409+santino18727-debug@users.noreply.github.com> * fix(openai): preserve cache_control for openai-compatible custom endpoints (#30387) * fix(openai): preserve cache_control for openai-compatible custom endpoints * fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation * fix(proxy): drain all daily-spend batches per flush cycle (#30281) (#30505) * fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (#30545) * fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers * test(types): add regression test for internal rate-limit fields in all_litellm_params * fix(init): add bool type annotation to suppress_debug_info (#30531) Module-level `suppress_debug_info = False` had no annotation, so strict type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to `True` (as done in proxy_server.py and router.py) then fails with an invalid-assignment error. Annotate it as `bool` to match every other flag in this module. * fix: coalesce null aggregates in update_metrics for no-spend keys (#29945) * feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (#30006) * feat(team_endpoints): Add query parameter key_limit to /team/info * feat(team_endpoints): update schema.d.ts to include the new query parameter * feat(team_endpoints): add tests for limitting key count in /team/info response * feat(team_endpoints): Apply suggestions from greptile * Set greater-than constraint on key-limit * Fix type * fix(router): release aiohttp connection when stream iteration ends abnormally (#30271) * fix(router): release aiohttp connection when stream iteration ends abnormally A streaming response that terminates with a mid-stream read timeout, a task cancellation (client disconnect), or GeneratorExit never closed the underlying aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body EOF, so each abnormally terminated stream permanently leaked one slot from the shared TCPConnector pool. During a backend traffic spike the pool drains; once exhausted every subsequent request to that host waits for a slot, times out and surfaces as a 408, indefinitely, even after the backend recovers. Only a proxy restart cleared the in-memory sessions, which matched the reported symptom of a router stuck returning 408 for a healthy vLLM backend. Close the response in a finally clause when iteration ends. On a fully read response the connection was already released at EOF and close() is a no-op, so keep-alive reuse for normal requests is unchanged. Fixes #30192 * test(aiohttp): cover GeneratorExit path with a mock instead of a live socket The previous slot-release test started a real aiohttp TCP server, which can flake in offline CI and does not exercise this fix's code path directly. Replace it with a dependency-injected mock that closes the stream generator (GeneratorExit) and asserts the response is closed, covering the third abnormal-exit path the finally block handles * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273) * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery * refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils * fix(proxy): make model_list request param optional for direct callers * feat(dashscope): add Responses API support (#30286) * feat(dashscope): add Responses API support DashScope's OpenAI-compatible endpoint serves /responses, so register a DashScopeResponsesAPIConfig that routes dashscope/* responses calls to {api_base}/responses without rewriting the upstream model id, instead of falling back to the chat-completions -> responses emulation pipeline. Closes #29780 * feat(dashscope): mark responses API as not supporting native websocket Matches the hosted_vllm/perplexity/openrouter responses configs, which all override supports_native_websocket() to False since the OpenAI-compatible endpoint has no native wss:// responses transport. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): preserve error_message on ProxyException failures (#30381) * fix(spend-logs): preserve error_message on ProxyException failures `StandardLoggingPayloadSetup.get_error_information` used `str(original_exception)` to populate the human-readable error message stored in `spend_logs.metadata.error_information.error_message`. `ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in its constructor but does NOT call `super().__init__(message)` and does NOT define `__str__`. As a result, `str(ProxyException(...))` returns the empty string, and every auth/budget/quota rejection was landing in spend_logs with `error_message=""` despite a fully populated traceback. Operator impact: dashboard "LLM Failure" rows became untriageable — the only way to tell a 401 from a 429 was to manually unpack the traceback JSON via psql. Burst failure patterns (e.g. a UI session polling with a stale token) produced 20-30 indistinguishable `error_code=401` rows per second. Fix: prefer the `.message` attribute (set by ProxyException and every litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback is retained for non-litellm exception types, preserving prior behavior. Test plan: - 2 new unit tests in tests/test_litellm/litellm_core_utils/ test_litellm_logging.py: * test_get_error_information_prefers_message_attribute_over_str * test_get_error_information_falls_back_to_str_when_no_message_attr - Existing test_get_error_information_error_code_priority still passes - End-to-end verified: bad-key 401 now stores full "Authentication Error, Invalid proxy server token passed..." message in spend_logs.metadata.error_information.error_message * fix(spend-logs): preserve explicit empty .message + drop dead reference Greptile P2 on #30381. The truthiness check `if message_attr:` silently skipped an explicit empty-string `.message` and fell through to `str(original_exception)`. For ProxyException-shaped objects both produce empty, so the bug was latent; for other exception types it would inject a different string into error_information.error_message and corrupt the signal. Use `is not None` so an empty string survives verbatim. Also drop the stale `See e2e/cases/11.` comment reference — that path does not exist anywhere in the repo and confuses future readers. Regression test added: an exception with `.message=""` and a non-empty `super().__init__()` arg must yield error_message == "". * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (#30382) * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response The non-streaming /v1/messages response carries a LiteLLM-injected usage.total_tokens = input_tokens + output_tokens that is not part of the Anthropic API spec. This caused three problems: 1. Shape divergence with streaming on the same endpoint. message_delta.usage in the SSE path never carries total_tokens. Clients parsing both paths get two different schemas from one endpoint. 2. Shape divergence with upstream. Direct calls to https://api.anthropic.com/v1/messages return no total_tokens field, so clients using the official Anthropic SDK couldn't rely on it, and clients that did rely on the LiteLLM-injected one broke when bypassing the proxy. 3. Numerical misuse. total = input + output undercounts when cache_read_input_tokens and cache_creation_input_tokens are non-zero, because cache tokens are reported in their own fields. A 100k-token cached prompt with 1 non-cache input token + 200 output tokens reports total_tokens = 201, off by ~99.8% from any reasonable definition of "total." Fix: add _strip_total_tokens_from_anthropic_response in litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the success path of anthropic_response right before returning. Only mutates dict-shaped responses; streaming (which already lacks the field) is left untouched. spend_logs / Prometheus continue to compute total_tokens internally for billing — this fix only strips the field from the wire response. Scope: only the Anthropic passthrough endpoint /v1/messages. The OpenAI-shape /v1/chat/completions is unaffected. * fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage Two P1 greptile threads on #30382: P1 — **Backwards-incompatible removal without a feature flag** Stripping `usage.total_tokens` unconditionally breaks any client currently reading the LiteLLM-shaped non-streaming /v1/messages response. Per the codebase's policy (mirrors #30418), gate behind a new flag. - `litellm.strip_anthropic_total_tokens: bool = False` (default — backward-compat: clients keep seeing total_tokens). - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`. - Docstring: planned to flip to True in a future major release; opt in early. P1 — **Silent no-op if `result` is a Pydantic model** `base_process_llm_request` may return a Pydantic-style object whose `.usage` is a plain dict (the most common shape — e.g. objects wrapping raw upstream JSON). The original `isinstance(response, dict)` guard skipped strip on those, so `total_tokens` would still hit the wire. Helper now also reads `getattr(response, "usage", None)` and strips when that's a dict. Strongly-typed Pydantic `Usage` sub-models with required `total_tokens` fields are still skipped — those impose type constraints the helper doesn't try to subvert. Tests: - `test_strips_total_tokens_on_pydantic_model_with_dict_usage` - `test_flag_defaults_off` 8/8 pass locally. * fix(anthropic): drop env var for strip flag (docs CI) Mirrors #30418's pattern (`expose_router_debug_in_errors: bool = True`, no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var introduced in the prior commit was flagged by `tests/documentation_tests/test_env_keys.py` because the documentation file `docs/my-website/docs/proxy/config_settings.md` lives in `BerriAI/litellm-docs` (separate repo) and registering a new env key requires a parallel docs PR — a friction we avoid here by exposing the flag only as a Python attribute + `litellm_settings` config key, both of which load through the existing proxy config plumbing without needing the env-var registry to be updated. No semantic change: default still False, behavior identical when set via `litellm.strip_anthropic_total_tokens = True` or `litellm_settings.strip_anthropic_total_tokens: true` in config.yaml. Verified locally: env scan no longer surfaces the key; 8/8 tests pass. * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (#30413) * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 * test: resolve model prices JSON relative to test file for pip installs * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (#30417) * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit signal from upstream inside an HTTP 500/503 envelope, with the real code only surfaced in the JSON body: {"error":{"message":"...high demand...","type":"upstream_error", "param":"","code":429}} Previously LiteLLM only looked at the HTTP status and mapped this to InternalServerError, which Router treats as non-retryable for many configs — so users got hard 500s instead of fallback/retry. Now the Gemini/Vertex exception mapper parses error.code from the body and routes code 429 to RateLimitError before falling through to the HTTP-status branches. Other body codes fall through unchanged. Tests cover: - new-api gateway's `code:429` payload now maps to RateLimitError - Genuine 500-body responses stay InternalServerError - Non-JSON body strings fall through to status-code mapping unchanged * fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes Addresses greptile P1/P2 + @Sameerlite's review on #30417. The new elif branch was firing for any HTTP status, so a gateway response of HTTP 400 with body {"error":{"code":429,...}} would be incorrectly promoted to RateLimitError (retryable) instead of falling through to BadRequestError. Same trap for 401 -> AuthenticationError. Scoped the body-code 429 check to `500 <= status_code < 600` — covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx envelope) without inviting the 4xx misclassification. Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401), and the existing fall-through cases, asserting each maps to the exception type that matches the HTTP status code. 50/50 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (#30418) * feat(router)!: redact internal model_group/fallback names from exception messages The Router was unconditionally appending internal config names onto exception.message: - "Received Model Group=..." - "Available Model Group Fallbacks=..." - "No fallback model group found... Fallbacks={...}" - "context_window_fallbacks={...}" - Deployment-timeout messages including model_group - Fallback failure detail listing fallback chain ProxyException forwards .message verbatim to clients, so gateways were leaking their model_name / fallback wiring in every failed call. Fix: gate all five mutation sites on a new `litellm.expose_router_debug_in_errors` flag (default False). Set to True to restore upstream debug behavior for local debugging. Why: matches the redaction posture this codebase already has for upstream model identifiers (cf. _litellm_returned_model_name) and removes the last common error-path leak of internal model_group names. Breaking change marker (!): if anything parses "Received Model Group=" out of client error messages, flip the flag on or migrate to the x-litellm-* response headers instead. Tests: 7 cases covering each of the 5 redaction sites + the flag-on inverse path, plus a "default off" sanity check. * test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate Addresses Greptile / codecov feedback on #30418: patch coverage was 55.6% with 4 lines uncovered in litellm/router.py. The existing tests exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found), and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3 were declared in the PR description as covered by "site 5 also fires" but the gate body lines for each (the `e.message +=` inside the `if litellm.expose_router_debug_in_errors:` branch) only execute when the flag is on AND the specific exception path is taken, which neither existing test triggered. Added 4 new tests (default + flag-on × 2 sites): - test_default_does_not_leak_deployment_timeout_debug - test_flag_on_leaks_deployment_timeout_debug - test_default_does_not_leak_content_policy_fallback_hint - test_flag_on_leaks_content_policy_fallback_hint Trigger details: - Site 1 (litellm.Timeout in _acompletion) is reached via the Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on `acompletion(...)`. Cannot embed a Timeout instance in model_list because Router.__init__ deep-copies it and Timeout.__reduce__ does not preserve the required positional args. - Site 3 (ContentPolicyViolationError without content_policy_fallbacks set, in async_function_with_fallbacks_common_utils) is reached by passing a `mock_response=litellm.ContentPolicyViolationError(...)` instance via the call-site kwarg — same deepcopy-avoidance reason. 11/11 tests pass locally. Patch coverage on litellm/router.py for this PR's diff should now be 100%. * chore(router): flip expose_router_debug_in_errors default to True Addresses @Sameerlite's review on #30418 — maintain backward compat on the wire. Redact becomes opt-in via setting the flag to False; the historical behavior (leak internal model_group / fallback wiring through exception messages) is preserved as the default. - litellm/__init__.py: default flipped to True, docstring rewritten with deprecation note pointing at a future flip to False (redact by default) in a major release. - tests/test_litellm/test_router_exception_redaction.py: fixture resets to True (was False); the "off" tests now explicitly set False; the "default_leaks_*" tests rely on the fixture default. test_flag_defaults_off -> test_flag_defaults_on. - No router.py change needed; the gate keys off the same flag, only the default changes. - PR title no longer needs the breaking-change `!` marker — no client sees a behavior change at default settings. 11/11 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(guardrails): integrate Repelloai Argus guardrail (#30465) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486) * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its result is carried in `provider_specific_fields.web_search_results` — PRs #17746 / #17798 restore it for callers that round-trip provider_specific_fields. A generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on replay and instead sends back an assistant `tool_call` + a `tool` message both keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use` (with no following *_tool_result) plus a user `tool_result` for the same id — both invalid, so the next turn 400s: messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks: srvtoolu_... Each `tool_result` block must have a corresponding `tool_use` block in the previous message. This is the commonly-reported vertex_ai symptom where Gemini works but Claude 400s on the 2nd turn of a web-search chat. Fix (litellm/litellm_core_utils/prompt_templates/factory.py): - convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching *_tool_result is available to pair with it; otherwise skip it (a bare server_tool_use is itself rejected). - anthropic_messages_pt: drop a replayed `tool`/`function` message whose tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client result; a user tool_result for it is invalid). The existing reconstruction path (provider_specific_fields present, e.g. the litellm SDK) is unchanged, as is regular client tool_use/tool_result. Tests (tests/llm_translation/test_prompt_factory.py): - update test_convert_to_anthropic_tool_invoke_server_tool -> test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped - add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool Follow-up to #17746 / #17798; addresses the generic-client (no provider_specific_fields) case of #17737. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite The regression tests added in tests/llm_translation/test_prompt_factory.py aren't run by the coverage CI job (it runs tests/test_litellm), so the new factory.py branches showed as uncovered (codecov patch coverage). Add equivalent focused tests in the unit suite so both new branches are exercised there: - convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no matching *_tool_result is available. - anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic OpenAI client replays. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the server_tool_use + result valid-pair path in unit suite Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke emitting server_tool_use followed by its web_search_tool_result when the matching result is present (the litellm-SDK round-trip path). Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(anthropic): flatten srvtoolu_ tool-message guard to a negated if Addresses the Greptile style nit: replace the if-pass/else with a single negated `if not (...)` guard around the tool_result append. Behavior unchanged. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(proxy): require premium only when enabling premium metadata fields (#30285) (#30506) Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (#30488) * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR #18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate. Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions). Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate). Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced. * style(perplexity): drop redundant type annotation on else branch to satisfy mypy mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file. * fix(perplexity): update integration test expected costs for non-double-billed math Three tests in test_perplexity_integration.py asserted the old buggy expectation that reasoning_tokens are billed in addition to the full completion_tokens count. After the fix in cost_per_token, reasoning_tokens are billed at the reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the standard output rate, matching OpenAI/Perplexity convention (PR #18607). Updates: test_end_to_end_cost_calculation_with_transformation, test_main_cost_calculator_integration, test_high_volume_cost_calculation. The high-volume sanity threshold drops to 0.25 to reflect the corrected total. * fix(ui): use dynamic proxy base URL in MCP usage examples (#30487) Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the MCP server usage example and copy-to-clipboard snippet so the generated configuration works for non-local deployments. Fixes #30466 * feat: add missing UK PII entity types to Presidio guardrail (#30537) * feat: add missing UK PII entity types to Presidio guardrail Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection. * test: remove fragile hardcoded entity count test Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions. * style: apply Black formatting to match CI requirements * fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357) Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0 Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry Fixes #30346 * feat(mcp): make MCP gateway name and description configurable via env vars (#30473) * feat(mcp): make MCP gateway name and description configurable via env vars * Rename function _restore_env to _apply_env * docs(mcp): document import-time capture of env-backed identity constants Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module reload to observe env changes after import. Generated with AI assistance Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Yevhen Luhovtsov <yevhen.luhovtsov@intapp.com> Co-authored-by: Claude <noreply@anthropic.com> * fix(mcp): preserve native tools in semantic filter hook (#26650) * fix(mcp): preserve native tools in semantic filter hook The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP + native) to filter_tools(), which only knows MCP-registered tool names. Native tools silently failed the name match in _get_tools_by_names() and were dropped from the request. Fix: partition tools into native and MCP-registered before filtering. Run the semantic filter only on MCP tools, then merge native tools back unconditionally. Changes: - Robust _is_mcp_tool() using shape-based detection for OpenAI-format dicts, safe regardless of future _extract_tool_info changes - Single-pass partition loop (no double _is_mcp_tool calls) - Preserve native tools in MCP expansion path (mixed requests) - Track MCP expansion to prevent expanded tools bypassing filtering - filter_stats reports MCP-only counts for accurate metrics - Extracted _emit_filter_metadata() helper - Skip spurious filter headers for all-native tool requests Closes #26212 * remove stale docstring note referencing tools_expanded_from_mcp * fix: handle Responses API name collision and preserve tool ordering - Classify Responses API tools ({type: 'function', name: '...'}) as native to prevent name collisions with MCP canonical names - Preserve original request tool ordering using id()-based merge instead of naive native+mcp concatenation - Add 2 regression tests: name collision and ordering preservation * style: apply black formatting * fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge * lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention) * ci: retrigger checks after rebase onto litellm_internal_staging * feat(fireworks): sync Fireworks AI model registry with current platform catalog (#30616) Adds 12 new Fireworks serverless models and updates 3 existing entries in model_prices_and_context_window.json and its bundled backup to match the current Fireworks platform model list. New direct models: glm-5p2, qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6, deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast, kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and gpt-oss-20b now carry correct output token caps, cache-read pricing, and explicit capability flags max_tokens is set equal to max_output_tokens (not the full context window) for models whose generation cap is below their context window. This avoids the shared input+output budget path in get_modified_max_tokens, which would otherwise let callers request output sizes the model cannot produce. The same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b entries that had max_tokens equal to the full context window Short-form aliases (fireworks_ai/<model>) are added for every direct accounts/fireworks/models/ entry so cost attribution works for callers using bare model names. Router endpoints get short-form aliases too, and transform_request now routes bare names ending in -fast to the accounts/fireworks/routers/ path instead of defaulting every bare name to models/. This keeps the kimi-k2p6-fast router from being misrouted to the nonexistent models/kimi-k2p6-fast endpoint kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its replacement. Context windows for deepseek-v4 and kimi models use the power-of-two values (1048576 and 262144) published on the Fireworks model pages, matching the convention already used by existing entries Two regression tests in test_utils.py assert the exact per-token costs, token limits, capability flags, and short-form-to-long-form equality for all 15 models against both the main and backup cost maps. Two routing tests in test_fireworks_ai_chat_transformation.py verify bare -fast names route to routers/ and bare direct-model names route to models/ * fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443) * feat(anthropic): hoist leading in-array system to top-level (helper) * test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control * test(anthropic): mid-conversation system normalization cases * feat: add supports_mid_conversation_system flag to Claude Opus 4.8 Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost map and the bundled package backup, since the runtime helper and tests read the backup in local/offline mode. Pin the mid-system passthrough regression test to the local cost map via the existing local_model_cost_map fixture so it reads the branch-local flag rather than the network-fetched main copy. * fix(bedrock): normalize in-array system in /v1/messages handler (#29698) Wire normalize_system_messages_for_anthropic into anthropic_messages_handler so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform / Converse-bridge) hoist leading in-array system entries (and demote mid-conversation ones on models lacking supports_mid_conversation_system) into the top-level system field. The normalized messages/system are written back into the local_vars snapshot the base_llm branch reads from, otherwise the Invoke/Mantle fix would silently no-op. Also fix the helper to resolve supports_mid_conversation_system through the prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw _supports_factory could not see the flag once get_llm_provider left the invoke/ prefix on the model id, which would have wrongly demoted mid-conversation system on a Bedrock invoke opus-4-8 path. * fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param * fix(types): widen system param to Union[str, List] for hoisted system blocks * refactor(bedrock): drop dead local_vars messages writeback * fix(bedrock/converse): translate in-array system in anthropic->openai adapter (#29698) * fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty * fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches * fix(types): register supports_mid_conversation_system in model-info schema The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid) rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8 cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a typed, first-class capability flag alongside its peers (supports_output_config, etc.). --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (#28885) * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload By default the agentcore provider flattens the last message to a text-only {"prompt": "..."} payload via convert_content_list_to_str, silently dropping OpenAI multimodal blocks (image_url, file, input_audio, ...). This adds an opt-in `forward_multimodal_content` litellm param. When truthy and the last message's content is a list containing a non-text block, the original OpenAI content list is forwarded verbatim under a new "content" field so an attachment-aware AgentCore agent can read it. Default off keeps the payload byte-identical to the legacy {"prompt": "..."} shape — existing agents are unaffected. The flag is read from optional_params (where other AgentCore params land) with a litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...). AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint parses arbitrary JSON up to 100 MB (per https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html), so this is a purely upstream change; no AgentCore-side schema is asserted. * fix(bedrock/agentcore): shallow-copy forwarded multimodal content list Address review feedback (Sameerlite): payload["content"] = last_content aliased the caller's mutable messages[-1]["content"] list. Harmless today because the payload is JSON-serialized immediately, but a latent footgun if a future caller mutates the returned payload before serialization. Forward list(last_content) so the payload owns its own list. Block dicts stay shared on purpose — a deep copy would clone potentially large base64 media on the request hot path, and the flagged risk was the shared list, not the blocks. Update the passthrough tests to assert equality + distinct identity, and add a regression test that mutating the payload list can't leak back into the original message content. * Revert "fix(mcp): preserve native tools in semantic filter hook (#26650)" This reverts commit |
||
|
yuneng-jiang
|
df92c7fd07
|
fix(proxy): support SMTP implicit SSL (port 465) (#30395)
* fix: add smtp ssl (#30248) * add smtp ssl * fix comments' * fix(proxy): verify SMTP server certificate on starttls * dont read ssl from env * test(proxy): restore regression test for SMTP_TLS=False starttls skip --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> |
||
|
yuneng-jiang
|
5047eaf7f0
|
fix(proxy): return deprecated-key lookup result directly in get_data combined view (#30327)
The grace-period branch assigned the recursive get_data result (a finished LiteLLM_VerificationTokenView) back into the variable that the combined-view dict normalization then subscripts, raising TypeError on every request made with a rotated key inside its grace window; auth surfaced that as a 401. Return the recursive result directly instead. Regression test drives the full get_data flow: old hash misses the view, deprecated table resolves to the active token, and the call must return the view object |