mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-05 08:07:05 +00:00
86 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
devin-ai-integration[bot]
|
e25ce85dbc
|
fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging (#38102)
* fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): keep threaded sync failure handler so CustomLogger sync callbacks still run Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore: remove redundant comments per review Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
c04b5dba32 |
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix_model_router_spend_log_model
# Conflicts: # tests/test_litellm/litellm_core_utils/test_litellm_logging.py # tests/test_litellm/llms/azure_ai/chat/test_azure_ai_transformation.py |
||
|
ryan-crabbe-berri
|
0ea6f5e159 |
fix(azure_ai): stamp the model router's selected model instead of matching on the model name
The model Azure Model Router served was recovered by checking whether the text "model_router" or "model-router" appeared in a model string. Spend logs applied that check to the litellm model path, where the route prefix guarantees a match, but the proxy applied it to the client's model group alias, which carries no prefix. A model group named anything else therefore lost the selected model in both the response and the spend row. AzureModelRouterConfig now stamps the served model onto _hidden_params, and the spend log payload and the proxy's response restamping read that stamp. The name heuristic survives as a fallback for callers with no response in hand, routed through get_azure_ai_route so it lives in one place. |
||
|
Yassin Kortam
|
a44bb47563
|
fix(prometheus): fold auth/pre-call time into litellm_request_total_latency_metric (#37958)
litellm_request_total_latency_metric's start_time is set inside common_processing_pre_call_logic, which only runs after user_api_key_auth has already succeeded, so the metric silently excluded authentication and pre-call setup time despite being documented as total request latency. The sibling litellm_request_queue_time_seconds metric had the same problem: its arrival_time was captured after auth too, despite its own comment claiming to track when the request arrived at the proxy. request.state.litellm_received_at is now stamped unconditionally at the very first line of user_api_key_auth (previously only when OTEL was configured), giving a timestamp that precedes all auth work. Both metrics now derive from it: queue_time_seconds genuinely spans arrival through the start of pre-call processing, and the total-latency metric adds that queue time on top of its existing start/end window so it becomes true end-to-end latency. queue_time_seconds ends exactly at start_time rather than a separately captured timestamp, so its window and the total-latency window share a boundary instead of overlapping and double-counting a few lines of setup work on every request. |
||
|
Yassin Kortam
|
9349b22c64
|
fix(guardrails): stop PII/PCI masking gaps in SpendLogs, debug logs, and logging_only response (#37965)
The Presidio guardrail masks messages in place inside pre_call_hook, but three paths independently persisted or emitted the raw pre-guardrail data: the SpendLogs proxy_server_request body snapshot (taken before the hook runs), a verbose_proxy_logger.debug dump of the raw request, and logging_only mode's async_logging_hook, which never masked the model's response before it reached external logging callbacks. Resolves LIT-6015 |
||
|
ryan-crabbe-berri
|
ed02a121dd
|
Merge pull request #37878 from BerriAI/litellm_ruff_no_duplicate_definitions
test: enforce F811 so a duplicate definition cannot silently replace the first |
||
|
ryan-crabbe-berri
|
e9d40a8f73 |
test: enforce F811 so a duplicate definition cannot silently replace the first
A name bound twice keeps only the second binding. In `tests/` that is nearly always a repeated import, harmless but misleading, and the same rule is what catches the cases that are not harmless: a local that shadows an import the module still calls, and a second `def test_x` that quietly replaces the first. 311 of the 344 sites were repeated imports and came out with ruff's own fix. The remaining 33 needed a decision. Four modules imported a name they never used because a local definition below already shadowed it. Two comprehensions bound `call` over `unittest.mock.call`, which those modules import and use. One test rebound the two module handles its nested reload closure had captured. One class attribute shadowed an unused `status` import. The load-test fixtures move to a conftest, which is how pytest is meant to share them, so the test module no longer imports three fixture names it never calls. The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that fixture by name before the body runs, so the parameter never shadows anything. |
||
|
tin-berri
|
f9a8c96b82
|
feat(proxy): add router_model_name to auto-routed response bodies (#37725)
The auto-routed model group was only reachable through the x-litellm-model-id response header. SDK and framework callers that do not expose response headers had no way to read it, and under streaming there was no body surface at all. The response body `model` field is deliberately restamped back to the client-requested alias on both paths, which is correct OpenAI semantics, so this adds a separate namespaced `router_model_name` key instead of redefining `model`. The key is written on non-streaming bodies and on every SSE chunk, including the streaming fast path, and is emitted only when an auto-routing strategy actually selected the deployment. After a mid-stream fallback moves the request off the group the router picked, the key is omitted rather than continuing to claim the original tier. The router marker already supports per-chunk fallback signals via `x-litellm-attempted-fallbacks` headers; this wires that signal into the gate so no stale tier is claimed after a fallback fires. Also removes a redundant function-local import in the streaming generator that shadowed the module-level one for the whole function. |
||
|
ryan-crabbe-berri
|
b76def0e5d
|
test: require a match= on broad pytest.raises, and drop duplicate parametrize cases (#37769)
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A TypeError from a refactor, a botched fixture, an import that moved: all of them read as the rejection the test claims to police, so the test goes green for the wrong reason and stays green after the behaviour it guards is gone. PT011 closes that gap for the 317 sites B017 could not reach, because B017 only fires on a single-statement body with no `as e` binding. Each pattern here is the message the code actually raised, recorded by running the sites under a plugin that logged the concrete type and text per call site, so the assertions describe observed behaviour rather than a guess. Where a site raises more than one message across its parametrize cases, the pattern is an alternation of what was seen; where the exception carries an empty `str()` and puts the text on `.message`, the site keeps a narrow `noqa` with the reason. PT014 removes four parametrize cases that were listed twice. The duplicate re-runs an assertion that already passed, and it usually marks a case someone meant to vary and forgot to edit. |
||
|
Mateo Wang
|
a66a10b1b5
|
Merge pull request #37734 from BerriAI/litellm_fix_partial_stream_spend_rows
fix(streaming): price partial-stream spend rows at the real model and keep prompt and cache fields |
||
|
mateo-berri
|
0801493347 |
Match the client-name check to the name the proxy actually stamps
Pre-call processing rewrites request_data["model"] for aliasing and routing, so matching either key let a routed model count as the client's own name and put the wrapper model back on an Azure Model Router row. |
||
|
mateo-berri
|
c010bd6a7c |
Only keep the builder's model when the client did not ask for it
A chunk carrying usage is stored as a pre-restamp copy, so an alias-restamped stream reaches disconnect billing with its first chunk still on the deployment model and every later chunk on the client's name. That is the same shape Azure Model Router produces, and the previous guard read it as a routed model and left the alias on the row, which is the unpriced name this PR set out to stop. Compare the assembled model against the name the proxy stamps chunks with, so the alias goes back to the deployment's model and the routed model stays. |
||
|
mateo-berri
|
655d10775c |
fix(cost): keep mid-stream pricing from leaking a breakdown into the spend log
Pricing a frame through the request's own logging object is what makes custom deployment pricing work, but _response_cost_calculator does not only return a number. It also stamps cost_breakdown onto the live logging object, and on a pricing failure it writes response_cost_failure_debug_information into model_call_details. On an ordinary proxy stream that is harmless, because the success handler recomputes cost_breakdown at end of stream and overwrites whatever the frames left behind. The pass-through handlers are the problem: they compute their final cost with a bare completion_cost call and never touch cost_breakdown again, so a breakdown derived from one mid-stream frame would survive to the end and land in the spend log's metadata. response_cost itself is unaffected either way, so this was a reporting surface bug rather than a billing one, but the spend row would have gone from null to a populated breakdown for a partial frame. Snapshot both writes and put them back once the cost is read, so pricing a frame stays a read as far as the rest of the request is concerned. The returned cost is unchanged, so nothing about the injected usage.cost moves. |
||
|
mateo-berri
|
101ef7e167 |
test(cost): cover the chat.completion.chunk and raising-pricer branches
The logging-object pricing applies to streamed /v1/chat/completions too, not just Anthropic message_delta, so a deployment with negotiated per-token prices now gets that price in the streamed usage.cost there as well. Nothing asserted that half. Adds the discounted and the sticker-fallback case for the OpenAI chunk shape, plus the branch where the pricer raises and the frame falls back to model-name pricing instead of breaking the stream. |
||
|
mateo-berri
|
03a253a1f9 |
Keep the model Azure Model Router recovered from later chunks
The disconnect billing path was stamping the wrapper's model over whatever stream_chunk_builder assembled. For Azure Model Router that throws away the routed model: the proxy deliberately leaves those chunks unrestamped so the builder can pick the real model off a later chunk, and overwriting it prices the row at the router alias instead. Only apply the wrapper's model when the builder did not find a model beyond the first chunk's, which is every case except Model Router. |
||
|
mateo-berri
|
12ed364e47 | Merge branch 'litellm_internal_staging' into litellm_fix_messages_stream_cost_cache_tokens | ||
|
mateo-berri
|
de7dcbbc67 |
Carry real cache counts up instead of zeroing them on partial rows
cache_read_input_tokens and cache_creation_input_tokens are pydantic extras on Usage, not declared fields, so filling them in created keys that were not there before rather than replacing a None. Readers that test for presence then took the new zero as authoritative: the spend log writer skipped its own copy from prompt_tokens_details, turning a real cache read of 500 into 0, and the prometheus provider cache counters stopped incrementing. Carry the prompt_tokens_details counts up before defaulting to zero, so a partial row reports the same cache numbers a complete one does. Renamed the helper to say what it now does. |
||
|
mateo-berri
|
a3b6762788 |
fix(streaming): price partial-stream spend rows at the real model and keep prompt and cache fields
A streaming chat completion that ends early (client disconnect, or the proxy cutting the stream at LITELLM_MAX_STREAMING_DURATION_SECONDS) wrote a spend log row with spend 0.0, prompt_tokens 0 on the proxy-cut path, and no cache fields in usage_object. The proxy restamps chunk.model in place to the client-facing alias, so the partial response rebuilt from those chunks priced the unmapped alias and came out at 0. The failure path also rebuilt usage without the request messages, so prompt tokens counted to 0, and a cut stream never sees the final usage event that normally zero-fills the cache fields. Restamp the rebuilt partial response with the wrapper's real model before cost calculation on both the disconnect and the failure paths, pass the request messages when rebuilding usage on the failure path, and zero-fill missing cache usage fields the way completed streams already do. |
||
|
ryan-crabbe-berri
|
4af59d7c6e
|
ci: lint the test tree for undefined names and fix all 30 (#37671)
ruff.toml excludes tests/* from `ruff check`, so nothing has ever checked the test tree for names that do not exist. That matters more in tests than in product code: a NameError inside a test whose body is wrapped in `except Exception: pass` is swallowed, and the test reports green forever. Adds ruff-tests.toml selecting F821 alone, wired into the lint workflow and `make lint-ruff`, and clears every existing violation: - 4 tests interpolated an unbound `e` into a `pytest.fail` message reached only on the failure path, so the NameError, not the assertion, is what ran. test_llm_guard_error_raising is the worst: it passes today with content safety disabled entirely. It now asserts the 400 and its detail body. - 5 sites construct BaseExceptionGroup, a 3.11 builtin, in a tree that still supports 3.10. Guarded behind the exceptiongroup backport that anyio already pulls in below 3.11. - 9 missing imports (json, openai, Any, Final, HTTPException), including one in a helper that catches HTTPException by a name it never imported, so the challenge path it exists to detect raises NameError instead. - 5 annotations naming types imported inside the function body, hoisted to module scope or TYPE_CHECKING. - 2 blocks of dead code: everything after a pytest.fail in test_claude_agent_sdk, and an unused helper in test_end_users calling a function defined in a different module. - 1 error-path f-string in the router-settings doc test that masked the real FileNotFoundError behind a NameError. Only F821 for now. Widening the select list means ratcheting thousands of pre-existing findings, so rules go in one at a time with their violations already fixed. |
||
|
mateo
|
5d5dc4523f |
fix(cost): price streamed Messages usage via calculate_usage and the logging obj
Streamed `/v1/messages` `usage.cost` disagreed with the cost the logging callback recorded in three ways: `input_tokens` was read as the whole prompt total, but Anthropic reports it excluding cache tokens, so the non-cached input went unbilled on cache hits; the `cache_creation` 5m/1h split was dropped, billing 1h writes at the 5m rate; and costing by model name alone ignored the deployment's custom pricing, so a negotiated discount still streamed sticker price. Anthropic usage now goes through `AnthropicConfig.calculate_usage`, the same transformation the non-streaming path uses, and the chunk is priced through the call's logging object when there is one so it inherits `custom_pricing`, `custom_llm_provider`, `base_model` and `router_model_id`, falling back to `completion_cost` by model name. `calculate_usage` only reads its `usage_object`, so it now takes a `Mapping`. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
1857f5d04b
|
fix(proxy): send SSE keepalives while a slow upstream is still silent (#37322)
A model with a long time-to-first-token leaves the proxy's response completely idle, so any hop with an idle read timeout (AWS ALB and nginx both default to 60s) drops a connection that is perfectly healthy and would have delivered its tokens shortly after. The keepalive engines LiteLLM already ships wrap the response object, so they fill a gap once the upstream has answered and then gone quiet. They cannot fill the gap before it answers at all, and that is where the whole wait is spent: measured against api.openai.com/v1/chat/completions with gpt-5.6 at reasoning_effort high, the response headers and the first body byte both arrive at 37.90s. Nothing has entered the ASGI response phase by then. The upstream call is now raced against the keepalive interval, and when it loses, the SSE response is opened immediately and ": ping" comments, which every conformant SSE client ignores, fill the wire until the real response is ready to be replayed onto it. One seam per funnel: base_process_llm_request covers every native route, create_pass_through_route covers every passthrough route. Committing the status line that early is the cost. A failure discovered after the first ping reaches the client as an SSE error frame under a 200 rather than as an HTTP error status, and LiteLLM's own x-litellm-* response headers are not yet known. keepalive_ping_has_fired already documents the same trade-off for the existing engines. Both are why this stays off until an operator sets litellm_settings.sse_keepalive_ping_interval_seconds. Separately, the passthrough relay reached neither engine even for mid-stream gaps, which is the shape of #32491 and #24929, so the relayed bytes get the same treatment, gated on the upstream declaring text/event-stream and only emitted between complete frames so a binary transport (AWS event streams on /bedrock) and a stall halfway through a frame are both left alone. Fixes #34819 |
||
|
mateo-berri
|
d5a4c14577 |
docs(proxy): pre-fix passthrough streams omitted content-type, not octet-stream
Some checks failed
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
Terraform Modules / fmt, validate, test (aws) (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
|
||
|
mateo-berri
|
f86dc8f54e | test(proxy): assert non-Bedrock passthrough stream emits no content-type header | ||
|
mateo-berri
|
8ced9f56a1 | Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_lit_4561_bedrock_passthrough_content_type | ||
|
mateo-berri
|
05beb7abb5 | fix(proxy): emit uncached input cost so component headers sum to the total | ||
|
mateo-berri
|
7562445273 | test(proxy): assert production nesting semantics for component cost headers | ||
|
mateo-berri
|
9079e4c47b | fix(proxy): return cost breakdown header values as a named tuple | ||
|
abhinav
|
1241bd5ce1 |
feat(proxy): add per-component response cost headers
- Extract input_cost, output_cost, cache_read_cost, cache_creation_cost, reasoning_cost, and tool_usage_cost from logging object cost breakdown - Populate x-litellm-response-cost-* component headers in ProxyBaseLLMRequestProcessing.get_custom_headers - Ensure headers are omitted when cost breakdown is absent or values are None - Add comprehensive test suite covering component headers, math invariants, caching, reasoning, and discounts/margins |
||
|
mateo-berri
|
426b909447 | fix(proxy): inject streaming usage cost on openai passthrough streams | ||
|
tin-berri
|
55e666a05f
|
feat(complexity_router): report LLM classifier cost per request via routing_decision and x-litellm-classifier-cost header (#36015) | ||
|
Deepanshu Lulla
|
e2950a8995
|
fix(router): eagerly fetch Vertex AI deferred stream to surface HTTP errors in _acompletion fallback path (#34627)
* fix(router): eagerly fetch deferred stream to surface HTTP errors in fallback path
Providers like Vertex AI and Bedrock defer their HTTP call until the first
__anext__ on the returned CustomStreamWrapper (completion_stream=None,
make_call set). Errors raised inside __anext__ (e.g. 429, 503) escape the
_acompletion try/except block, so fail_calls is never incremented, deployment
cooldown does not fire, and the standard fallback chain is bypassed.
Call fetch_stream() on the wrapper before delegating to
_acompletion_streaming_iterator when completion_stream is None and make_call
is set. Any HTTP error now propagates through _acompletion's except block,
increments fail_calls, and enters the normal retry/fallback chain.
Strip Content-Length, Transfer-Encoding, Content-Encoding, and Content-Type
from exception headers at the same point to prevent HTTP framing mismatches
when LiteLLM builds its own error response body.
Add a re-raise guard in _acompletion_streaming_iterator (async and sync paths)
so MidStreamFallbackError with already-generated content re-raises to the
caller instead of silently injecting a continuation prompt into a fresh request
to a fallback model.
Apply logging cleanup in async_function_with_fallbacks_common_utils: use
%s-style formatting and exc_info=True instead of f-strings with
traceback.format_exc().
* fix(router): undo success_calls on deferred-stream fetch failure; broaden header strip
* fix(router): extract header-strip helper to keep _acompletion under strict C901 threshold
* test(router): add unit tests for _strip_http_framing_headers to satisfy router coverage gate
* test(router): add sync _completion_streaming_iterator re-raise test for mid-chunk MidStreamFallbackError
* fix(router): restore Fallbacks context in no-fallback log; document update_team mcp_rpm_limit
The log and debug message when no fallback model group is found was missing
the Fallbacks list, making it hard to understand why routing failed.
Also adds the missing mcp_rpm_limit documentation to update_team to fix
the documentation_test_api_docs CI check.
* fix(router): preserve original traceback in deferred stream fetch error re-raise
Using bare `raise` instead of `raise fetch_err` keeps the full inner
traceback from fetch_stream() intact so the error origin is visible in
logs and debuggers without being anchored to this line.
* style(test): restore black-style formatting in test_router.py
An earlier commit on this branch collapsed the file's pre-existing
multi-line formatting into single lines while adding the deferred-stream
tests, producing a diff full of unrelated reformatting noise. Restores
the untouched code to its original formatting; the actual new/changed
test content is unaffected (verified via AST comparison).
* fix(router): re-raise mid-stream fallback on any generated content, not just text
The re-raise guard added for MidStreamFallbackError only checked
generated_content, which tracks text deltas alone. A stream that emitted a
tool-call or reasoning-only chunk before failing had generated_content=""
despite already streaming to the client, so the router silently retried
and the client saw duplicated/inconsistent output. The guard now also
inspects the wrapper's raw chunks for tool_calls/reasoning_content.
Also moves the deferred-stream HTTP-framing-header stripping out of
Router._acompletion into the proxy's _handle_llm_api_exception: Router is
used directly as an SDK as well as by the proxy, and stripping headers
there dropped legitimate provider metadata (content-type,
proxy-authenticate) for direct SDK callers who never see the proxy's own
response construction.
schema.d.ts regenerated via make pre-commit; unrelated to this change.
* test(router): add direct coverage for _stream_chunks_have_generated_content
CI's router_code_coverage check flags any router.py function never referenced
by name in a test file; the new helper was only exercised indirectly through
the mid-stream re-raise guard tests.
* revert(ui): drop incidental schema.d.ts regeneration
Committing router.py/common_request_processing.py touched
pre_commit_lint.sh's litellm/proxy trigger for the API-type-sync check,
which force-regenerated schema.d.ts even though neither file changes any
route or model. The regenerated ordering of two unrelated Union/enum
fields (stream_timeout, user_role) isn't stable across process
invocations even against completely unmodified backend code (confirmed
by regenerating twice against the pre-existing committed code and getting
the same diff both times), so this reverts to the original committed
file rather than chase non-deterministic output.
* fix(proxy): strip framing headers on the pre-existing ProxyException branch too
_handle_llm_api_exception filtered framing headers into a local `headers`
dict, but for an exception that's already a ProxyException, it merged
{**e.headers, **headers}: the original e.headers came first, so a framing
header present there but absent from the filtered `headers` (because it
was just stripped) was never overwritten and survived into the response
unfiltered. Filters the merged result instead of relying on the merge
order to do it implicitly.
* chore: retrigger CI (no GitHub Actions check-suite was created for the previous two pushes)
* fix(router): detect thinking_blocks as generated content in mid-stream guard
Greptile flagged that a thinking-only delta (Anthropic extended thinking,
Delta.thinking_blocks) wasn't recognized as already-streamed content, so
a stream that emitted only thinking blocks before failing could still
restart via fallback and append an unrelated response after content the
client already received.
* fix(proxy): strip browser-facing security headers from provider exceptions too
veria-ai flagged that the framing-header denylist still let a malicious or
misconfigured provider set browser-facing headers (Access-Control-Allow-Origin,
Content-Security-Policy, Clear-Site-Data, etc.) on the proxy's own error
response. Adds a dedicated _BROWSER_SECURITY_HEADERS set alongside the
existing framing one and strips both wherever provider exception headers
reach the client response.
* refactor(router): address maintainer review mechanicals
- List[ModelResponseStream] -> list[ModelResponseStream] in
_stream_chunks_have_generated_content (ruff UP006 strict-budget gate)
- drop _strip_http_framing_headers and its 3 tests: the proxy inlines the
filter directly now, so the helper has had no production caller since
the header-stripping was moved out of Router
- move HTTP_FRAMING_HEADERS/BROWSER_SECURITY_HEADERS/
UNSAFE_PROXY_RESPONSE_HEADERS from router.py into litellm/constants.py,
removing the router.py <-> proxy import path the two CodeQL
cyclic-import alerts were pointing at
- move the eager fetch_stream() call before success_calls/logging/
_track_deployment_metrics instead of incrementing then compensating
with a manual decrement on failure
- fix a dead assert message: `mock_fallback.assert_not_called(), "..."`
built a tuple, not an assert-with-message; assert_not_called() already
raises on its own so this just drops the inert string
* revert(router): pull mid-stream continuation-removal out of this PR
Removing the continuation-prompt fallback (retrying with the partial
response as a prefixed assistant message) so a stream failing after
partial content always re-raises instead was a scope decision beyond
what this PR's title/issue (#31874) describe, and it directly conflicts
with #30242/#30743, which are already fixing the same code path for
Anthropic's removal of assistant-message prefill on Sonnet 4.6+/Opus
4.6+. Landing this PR's version first would delete the branch those PRs
are patching; landing theirs first would have this PR undo their fix on
rebase.
Restores the original prefill-based continuation-resume behavior
(including the is_pre_first_chunk guard already in litellm_internal_staging)
in both _acompletion_streaming_iterator and _completion_streaming_iterator,
and removes _stream_chunks_have_generated_content along with the tests
that only existed to cover the guard. This PR now only touches the
deferred-stream eager-fetch fix and the header-stripping fixes; the
non-text-content re-raise idea becomes a follow-up PR built on top of
whichever of #30242/#30743 lands.
* fix(proxy): re-filter unsafe headers after the response-headers hook merge
_handle_llm_api_exception filtered provider/framing headers once, then
merged in post_call_response_headers_hook's return value afterward
without re-filtering. The ProxyException branch happened to re-filter
after its own header merge, but the HTTPException/httpx.HTTPStatusError/
generic-exception branches passed the post-hook headers straight through
unfiltered, so a callback hook (any custom guardrail/logging plugin)
returning an unsafe header would bypass the strip entirely for those
paths. Filters once, right after the hook merge, so every branch gets
the same guarantee.
* Revert "revert(router): pull mid-stream continuation-removal out of this PR"
This reverts commit
|
||
|
Yassin Kortam
|
8cf2e2eb43
|
fix(proxy): apply key/team router_settings.model_group_alias (#35486)
Key and team `router_settings.model_group_alias` was accepted, persisted and echoed back by `/key/info`, but never applied at request time, so the request ran on the group the caller asked for. `route_request` forwards only the settings the Router accepts as per-request kwargs, and `model_group_alias` is not one of them: the Router resolves aliases from its own instance attribute, which holds the global config map and is shared across requests. Resolve the alias in the proxy instead, alongside the existing model-alias rewrites and ahead of the pre-call hooks, so per-model limits and guardrails key off the group that actually serves the request. Authorize the alias target before the rewrite; model access was checked against the requested group, so a key whose alias points at a group it cannot call gets the usual 403 rather than being quietly served it. Resolves LIT-4879 |
||
|
mateo-berri
|
c0de87d08d |
fix(proxy): resolve team-alias models in the stream usage support gate
Team-scoped models store an internal model_name_{team_id}_{uuid} name
with the public alias only in team_public_model_name, so resolving them
through get_model_list without team_id returned no deployments and the
gate skipped injection, leaving those streams on tiktoken estimates.
Thread user_api_key_dict.team_id through the gate.
|
||
|
mateo-berri
|
a770b437d5 |
fix(proxy): gate default stream usage injection on provider support and neutralize client-sent strip marker
Bytez and OCI param maps raise on stream_options when drop_params is unset, so the default injection would have broken every streamed chat completion routed to them. Injection now only happens when every router deployment behind the requested model (wildcards and aliases included) declares stream_options in its supported OpenAI params; providers that do not declare it either reject the param or already stream usage natively, so skipping them keeps old behavior instead of erroring. _litellm_strip_stream_usage arriving in the client request body is now overwritten at ingress (and popped in the experimental queue endpoint), so a client can no longer suppress the usage chunk it explicitly requested by planting the internal marker. |
||
|
mateo-berri
|
43efd02d35 |
fix(proxy): request stream usage upstream by default and strip it from client streams
Streamed chat completions that did not opt into stream_options.include_usage were logged with tiktoken estimates over the visible text, so hidden reasoning tokens (billed as output by OpenAI-compatible providers) were never counted and SpendLogs could undercount output tokens by 90%+ on reasoning models. The proxy now injects include_usage upstream for /v1/chat/completions streams by default and strips the injection artifacts (the final usage chunk and the empty prompt-filter chunk) from the client-facing SSE stream, so accounting uses provider-billed usage while the client-visible stream stays byte-identical to today. always_include_stream_usage keeps its existing semantics: true forwards the usage chunk to clients as before, and an explicit false now acts as a kill switch that disables the upstream injection for OpenAI-compatible backends that reject stream_options. |
||
|
devin-ai-integration[bot]
|
cc45d18e9c
|
feat(complexity-router): add return_raw_model_name toggle for response model field (#33875)
* feat(complexity-router): optionally return raw model name Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): restore asyncio import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(tests): preserve staging asyncio import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): drop unused local asyncio import Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(dashboard): add complexity router raw model toggle Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(complexity-router): move metadata key constant to constants.py Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy-tests): preserve module spacing Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Krrish Dholakia <krrishdholakia@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
shivam
|
371fa670d6 |
fix(proxy): forward Bedrock event-stream content-type on unbuffered passthrough
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yassin Kortam
|
ae92e511f1
|
fix(proxy): bill partial streamed spend when the client disconnects mid-stream (#33736)
* fix(proxy): bill partial streamed spend when the client disconnects mid-stream * fix(router): guard FallbackStreamWrapper chunks alias for non-CSW streams * fix(proxy): await disconnect billing dispatch instead of unrooted create_task * fix(proxy): make disconnect slot release single-owner to avoid double release * fix(proxy): use union syntax for disconnect cleanup params (UP045 budget) |
||
|
Yuneng Jiang
|
eae1d2aa79
|
test(proxy): cover per-key per-model TPM limit triggering gateway fallback
Drive the real parallel_request_limiter through _pre_call_with_fallbacks for the LIT-3890 customer scenario: a key-level model_tpm_limit raises ProxyRateLimitError from the pre-call hook and the configured gateway fallback serves the request instead of returning a 429. Unlike the existing tests, this exercises the actual limiter rather than a hand-built error. Also switch the new _pre_call_with_fallbacks return annotation to builtin tuple to stay within the ruff UP006 strict-rule budget. |
||
|
Yuneng Jiang
|
e5103e0290
|
Merge branch 'litellm_internal_staging' into litellm_local-rate-limit-fallbacks | ||
|
Mateo Wang
|
2f0cdb35bf
|
fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI (#32258)
* fix(responses-bridge): custom tool round-trip and allowlist preservation for Codex CLI
Convert Responses API custom tools to Chat Completions function tools and map
function_call responses back to custom_tool_call output items so Codex CLI gets
the apply_patch round-trip it expects. Preserve and validate allowed_callers
during the custom->function conversion so the Anthropic adapter's caller
allowlist is not silently dropped, which would let a tool meant to be callable
only by another tool be invoked directly by the model. Use modern type
annotations (list/dict/set/X | None) throughout to keep the ruff strict budget
within its ratcheted ceilings.
* fix(responses-bridge): address review feedback on custom tool bridge
Type convert_custom_tool_to_function_tool against Mapping/ChatCompletionToolParam
and validate allowed_callers with a strict TypeAdapter so the two new cast()
calls that tripped the LIT006 ceiling are gone. Warn when dropping Responses-only
tool types (computer_use, image_generation, namespace, shell) instead of
discarding them silently. Return output items as Pydantic models instead of
model_dump()ing every item to a dict, matching the declared return type. Apply
the same None-safe metadata pattern to the request_data paths that still used
setdefault, and drop the unused build_custom_tool_call_item helper.
* fix(responses-bridge): recover custom tool input when arguments is empty
* fix(auth): extract custom tool names for allowlist enforcement on responses route
The Responses guardrail translation handler only extracted function and mcp
tool names, so a key or team restricted by metadata.allowed_tools could invoke
a disallowed tool by declaring it with type custom now that the bridge converts
custom tools into callable Chat Completions function tools. Extract custom tool
names through the same path so check_tools_allowlist rejects them.
* fix(responses-bridge): scope input payload recovery to custom_tool_call items
Recovering tool arguments from the input field on any falsy arguments value
made plain function_call input items with empty arguments and a stray input
key get rewritten into a {"content": ...} envelope, corrupting multi-turn
replay for normal function tools. Gate the recovery on the item type so it
only applies to custom_tool_call items, which are the ones that store their
payload in input.
* fix(responses-bridge): default missing function_call arguments to empty string
With input recovery scoped to custom_tool_call items, a plain function_call
input item without an arguments key left raw_arguments as None and the
downstream str() turned it into the literal string None. Coerce to an empty
string instead, matching the pre-bridge behavior.
---------
Co-authored-by: duanhongyi <duanhongyi@doopai.com>
|
||
|
Sameer Kankute
|
5b93ba0ada
|
feat(router): add separate ITPM/OTPM deployment rate limits (#31952)
Some checks are pending
GitHub Actions Security Analysis / zizmor (push) Waiting to run
* feat(router): add separate ITPM/OTPM deployment rate limits Support input/output tokens per minute on deployments via enforce_model_rate_limits, with reservation, reconciliation, refund on failure, and rate-limit headers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(router): keep ITPM/OTPM diff minimal in router.py Drop unrelated Black reformatting from router.py and types/router.py so the PR only contains functional ITPM/OTPM changes. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): make ITPM/OTPM limits separate and atomic Address Greptile review on separate ITPM/OTPM deployment rate limits. - OTPM is now reserved atomically pre-call with rollback, matching the ITPM path, so concurrent requests can no longer overshoot the configured output limit before reconciliation - ITPM counts input tokens only; it no longer accumulates completion tokens, so the input-token limit and x-ratelimit-limit-input-tokens header describe input usage as their names imply - _read_reservation_from_kwargs only falls back to litellm_params.metadata when the top-level metadata channel is absent, so production requests carrying a litellm_params.metadata dict still reconcile and refund their reservation Adds regression tests for OTPM atomicity under concurrency, input-only ITPM enforcement, and reservation lookup when litellm_params.metadata is present. * fix(router): subtract input tokens only from remaining-input-tokens header The in-flight replay for x-ratelimit-remaining-input-tokens subtracted total tokens (input + output) instead of input tokens only, so clients saw remaining input quota understated by the completion token count on every response. Now consistent with the input-only ITPM counter. * fix(router): make itpm/otpm vs tpm/rpm precedence explicit When a deployment configures itpm/otpm alongside tpm/rpm, the io-token path takes over and the tpm/rpm limits are not enforced. Log a warning the first time such a conflicting deployment is seen so the supersession is not silent, and document the mutual exclusivity. Post-call reconciliation now only trues up a counter that was actually reserved against, so the itpm/otpm keys are no longer incremented for deployments that never configured that limit. * fix(router): track actual io-token usage on the reservation-minute key Post-call reconciliation now keys off the exact cache key stashed at pre-call time rather than one recomputed from the response-time minute. This fixes two issues: a request whose pre-call estimate was 0 now still writes its actual billable input to the ITPM counter (previously it was skipped, leaving the limit unenforceable for that request), and a call that finishes in a later minute reconciles against the minute it reserved against instead of pushing a negative delta into the next minute. Counters are only touched when their limit is configured. * fix(router): run io-token reconciliation before the model_id guard async_log_success_event gated IO reconciliation behind the model_id guard that only the TPM tracking path needs. Since reconciliation works entirely from the cache keys stashed in kwargs, a success event whose standard_logging_object lacks model_id would skip reconciliation and leave the reservation on the counter until the TTL expired, wasting quota. Route the IO path first. * fix(router): don't replay in-flight delta for itpm/otpm headers For ITPM/OTPM model groups the counter is incremented at reservation time (pre-call), so the remaining values returned by get_remaining_model_group_usage already account for the current request. Replaying the in-flight delta on top double-counted it and understated x-ratelimit-remaining-input/output-tokens by up to max_tokens on every response. Skip the delta for io-token groups; the legacy TPM/RPM replay path is unchanged. * fix(router): clear io-token reservation after reconcile/refund async_io_token_refund_failure and async_io_token_reconcile_success now clear the stashed reservation keys from the request metadata once done. Otherwise, on a model group mixing IO-limited and non-IO deployments, a failed IO call that retries on a non-IO fallback left the stale sentinel in the shared request metadata; the fallback's success handler would divert into IO reconciliation against the already-refunded key, driving the ITPM counter negative and skipping the non-IO deployment's TPM tracking. * fix(router): tidy reservation channel lookup and header guard Consolidate the reservation channel lookup into a single ordered helper shared by read and clear, so top-level metadata always wins over litellm_params metadata without the tangled per-iteration fallback. Also stop gating the router rate-limit header block on the presence of x-ratelimit-remaining-input/output-tokens. That block only emits those headers for ITPM/OTPM groups; for a non-IO group backed by a provider that natively returns input/output token headers, the extra conditions suppressed the router's own remaining-tokens/requests headers. * fix(router): strip client-supplied io-token reservation keys The reservation sentinels (_litellm_itpm_reserved, _litellm_itpm_cache_key, and the otpm equivalents) are server-only, but metadata is caller-controlled on proxy requests. An authenticated caller could forge these fields with an arbitrary cache key so the post-call reconcile/refund path would decrement any deployment's ITPM/OTPM counter and let it exceed the configured limit. Strip the reserved keys from the request metadata in set_io_token_rate_limit_request_kwargs, which runs before the router stashes its own reservation, so only a genuine server-side reservation is ever read post-call. * fix(router): track TPM routing load for io-limited deployments deployment_callback_on_success early-returned for any deployment with itpm/otpm set, so its total-token usage never landed in the router's TPM routing counter. TPM-aware routing strategies then saw 0 load for IO deployments and over-routed to them in mixed model groups. Only skip tracking when neither tpm/rpm nor itpm/otpm are configured; itpm/otpm enforcement still runs separately in ModelRateLimitingCheck, so the routing counter and the enforcement counters stay independent. * fix(router): expose standard tpm/rpm headers for io-limited groups get_remaining_model_group_usage returned early for ITPM/OTPM groups, so a group that also set tpm/rpm never emitted x-ratelimit-remaining-tokens / -requests; clients and prometheus gauges reading those saw no data. Build both header sets instead of returning early. Also simplify the in-flight header replay: only the tpm/rpm counters are incremented post-response, so the delta now adjusts just those. The itpm/otpm counters are incremented at reservation time (pre-call), so the input/output token headers already reflect the request and are left untouched - which removes the need for the separate io-group special case. * fix(router): roll back ITPM on any OTPM reservation error; dedup warning per instance Two follow-ups from review. The pre-call OTPM reservation only rolled back the ITPM reservation on a RateLimitError, so a transient cache error while reserving OTPM left the ITPM counter inflated until the TTL expired; catch any exception, release the ITPM reservation, then re-raise. Replace the module-level lru_cache warn-once (caching a logging side effect, which never re-warns in a long-lived process) with an instance-scoped set of already-warned deployment ids on ModelRateLimitingCheck. * fix(router): always clear reservation stash on reconcile; don't collapse id-less warning dedup Clear the reservation in a finally block so a mid-reconciliation cache error still removes the stash and a duplicate success event can't re-process it. Dedup the itpm/otpm-vs-tpm/rpm conflict warning per real deployment id; a deployment with no id no longer collapses every id-less deployment onto the str(None) key (which would suppress all but the first warning). * fix(router): skip io reservation when deployment can't be keyed _get_cache_keys returned a shared 'global_router:None:None:...' key when a deployment was missing model_info.id or litellm_params.model, so misconfigured deployments could share one rate-limit bucket. Return None in that case and skip io reservation for the request. * fix(router): honor explicit max_tokens=0 in io reservation _resolve_max_tokens used 'max_tokens or max_completion_tokens', so an explicit max_tokens=0 fell through to the model default. Only fall back to max_completion_tokens when max_tokens is absent. * fix(ci): satisfy lint budget, router coverage, and dashboard schema sync - Modernize the new itpm/otpm module's type hints to PEP 585 lowercase generics (Dict/Tuple/List -> dict/tuple/list) to clear the added UP006 violations; ratchet ruff-strict-budget.json's UP006 ceiling down to match. - Replace three try/except Exception blocks that must stay broad by design (token_counter and litellm.get_model_info raise untyped exceptions, and an io-token refund failure must never break the logging pipeline) with contextlib.suppress(Exception), matching the codebase's existing resolution for this exact BLE001 pattern. - Add direct unit tests for get_model_group_io_token_usage (multi-deployment aggregation and the empty-model-list case) in test_router_helper_utils.py, satisfying the router function-coverage check. - Regenerate the dashboard's schema.d.ts so the new itpm/otpm fields on GenericLiteLLMParams and ModelGroupInfo are reflected in the OpenAPI types. * fix: enforce io token rate limits consistently * fix: honor zero max tokens in otpm reservation * fix(lint): fix UP007 violation and resync ruff-strict-budget.json to base Convert Union[_Span, Any] to _Span | Any (safe on this repo's Python >=3.10 floor) to clear the new UP007 violation from the TYPE_CHECKING-gated Span alias. The previously committed ruff-strict-budget.json ratcheted UP006 down from a stale base; litellm_internal_staging has since tightened that same ceiling further on its own. Reset the file to the current base's committed values and re-ratchet from there so the budget only ever moves down relative to the actual merge-base, never against a stale snapshot. * fix(router): attach ITPM/OTPM headers on dict responses and harden reservation Strip itpm/otpm from provider kwargs, ensure messages are available for ITPM estimation, honor max_output_tokens on /v1/responses, and propagate rate-limit headers through /v1/messages dict responses via _hidden_params. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(router): attach ITPM/OTPM headers to streaming /v1/messages responses Wrap bare async iterators in HiddenParamsAsyncIteratorWrapper so set_response_headers can attach rate-limit headers to streaming Anthropic messages responses that lack a _hidden_params slot. Co-authored-by: Cursor <cursoragent@cursor.com> * style: ruff format add_retry_fallback_headers.py Fix CI ruff format check failure on get_hidden_params_dict call site. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(router): extract set_response_headers helpers to fix C901 budget Move header-attachment logic into add_retry_fallback_headers helpers so set_response_headers stays under the strict complexity ceiling. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: keep IO token reservation when response usage is missing Missing usage was reconciled as zero and fully refunded the pre-call reservation, allowing limit bypass on repeated successful calls. Only adjust counters when usage is resolved from the response or standard logging fields; otherwise keep the reservation until TTL expires. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: enforce RPM/TPM alongside IO-token limits on mixed deployments Deployments with both itpm/otpm and tpm/rpm previously returned after the IO reservation and skipped RPM/TPM checks. Run both paths and refund the IO reservation only when RPM/TPM rejects after a successful reservation. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: track TPM usage on success for mixed IO+TPM deployments The early return after IO-token reconciliation in log_success_event and async_log_success_event skipped the TPM counter increment, so the tpm_key the pre-call check reads was never written and tpm_limit was never actually enforced on deployments that also configure itpm/otpm. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: treat total-only usage as unresolved in IO-token reconcile usage/standard_logging_object entries carrying only total_tokens (no prompt/completion or input/output breakdown) were treated as resolved usage, resolving to (0, 0) and refunding the full reservation. Both _usage_is_present and the standard_logging_object fallback now require an actual input/output breakdown before reconciling, keeping the reservation otherwise. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: reserve minimal token when input/output estimation fails _reservation_value(0, limit) reserved the entire limit whenever token estimation failed (empty/unsupported input, tokenizer error), letting one such request claim the whole bucket and 429 every concurrent request to the deployment until it completed. Reserve 1 token instead so estimation failures no longer serialize traffic. Co-authored-by: Cursor <cursoragent@cursor.com> * fix: refund IO reservation synchronously before retry deployment pick On retry, set_io_token_rate_limit_request_kwargs clears reservation sentinels from the shared kwargs dict before a background failure handler can refund them, stranding the counter until TTL. Refund and clear any stale reservation in _update_kwargs_with_deployment before stripping sentinels for the next attempt. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(io_token_rate_limit_check): use model-specific tokenizer for ITPM estimate; document sync-refund Redis ceiling Pass the deployment litellm_params.model to token_counter so it uses the model's native tokenizer instead of the generic fallback, narrowing the reservation over/under-estimate window between pre-call and post-call reconcile. Add a ponytail: comment to refund_stale_reservation_before_retry explaining the known ceiling: the synchronous DualCache.increment_cache issues a blocking Redis INCR when a Redis backend is configured. This only fires on streaming mid-stream retries (non-streaming failures await their failure handler before the retry picks a new deployment, leaving no sentinels to refund). Upgrade path: make _update_kwargs_with_deployment async. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
devin-ai-integration[bot]
|
b768b62067
|
fix(proxy): restore model state on non-rate-limit exceptions in fallback loop
Addresses Greptile review feedback: wrap the fallback loop in try/except BaseException to always restore self.data['model'] to the original value when a non-ProxyRateLimitError exception escapes a fallback attempt. Add regression test for this edge case |
||
|
devin-ai-integration[bot]
|
9ea149b49e
|
refactor: remove getattr, unused param, and unnecessary comments | ||
|
devin-ai-integration[bot]
|
1d89e65731
|
fix(proxy): trigger gateway fallbacks on local rate limit errors
When pre-call hooks (parallel_request_limiter, dynamic_rate_limiter_v3) reject a request with ProxyRateLimitError, the router's fallback logic was never reached because the exception was raised before route_request was called. Add _pre_call_with_fallbacks that catches ProxyRateLimitError, resolves configured fallbacks (key-level router_settings -> router-level), and retries with each fallback model in order. If all fallbacks are also rate-limited, the original error is re-raised. |
||
|
Mateo Wang
|
375659ef04
|
fix(proxy): emit x-litellm-response-cost header on /messages and /generateContent (LIT-4076) (#31675)
* fix(proxy): emit x-litellm-response-cost header on /messages and /generateContent (LIT-4076) The Anthropic /v1/messages and Google native :generateContent routes return TypedDict results (AnthropicMessagesResponse, GenerateContentResponseBody) that are plain dicts at runtime and cannot hold a _hidden_params attribute. The cost is computed by update_response_metadata, but ResponseMetadata.apply() only persists _hidden_params back when the result object has that attribute, so for those two routes the computed response_cost was dropped. The non-streaming header build in base_process_llm_request then saw an empty response_cost and get_custom_headers filtered the x-litellm-response-cost header out, even though the other x-litellm-* headers still appeared. The non-streaming success path now recovers the cost from the logging object when the response cannot carry _hidden_params, preferring the value already stored in model_call_details and recomputing from the same calculator only when it has not been stored yet. Object responses (ModelResponse, ResponsesAPIResponse) keep their existing behavior, so chat/completions, /responses, and the Anthropic error path that intentionally emits a zero cost are unaffected. Streaming stays out of scope because the header is emitted at stream start, before the cost is known. * fix(proxy): also recover response cost header for /generateContent responses with _hidden_params (LIT-4076) * fix(proxy): compute generateContent response cost synchronously so cost header is emitted (LIT-4076) * fix(lint): suppress BLE001 on generate_content cost normalization guard The defensive blind except keeps cost normalization from ever breaking the response path; mark it noqa so it does not breach the strict-rule budget. |
||
|
Yassin Kortam
|
c33a7f8757
|
fix(proxy): cancel upstream LLM stream when client disconnects during time-to-first-token (#31499)
create_response buffers the first streamed chunk (to detect error-only streams) before handing the StreamingResponse to Starlette. Starlette only starts listening for client disconnects once it is serving that response, so a disconnect during a long time-to-first-token left the upstream LLM call running until the request timeout. This races the first-chunk fetch against an http.disconnect monitor; on disconnect it cancels the fetch, which propagates into async_streaming_data_generator's cleanup (records the 499 and closes the upstream stream), and returns a 499. Resolves LIT-3568 |
||
|
Sameer Kankute
|
e5da5a3b6d
|
fix(proxy): skip model override when response has no model field (#31183)
* fix(proxy): skip OpenAI model override for search responses Search responses omit a model field by spec but still set model on the request for routing, which caused noisy errors and dict injection. * fix(proxy): drop redundant search-specific model override skip The silent return for responses without a model field already covers SearchResponse objects; remove the extra search type check. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): skip model override for dict responses without model key Dict-shaped responses (e.g. search) must not get a spurious model field injected when they never had one; only override when model is present. Co-authored-by: Cursor <cursoragent@cursor.com> * test(proxy): cover swallowed setattr failure in model override --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
b5fcd859be
|
fix(guardrails): return 400 not 500 when AIM blocks a request (#30573)
* fix(guardrails): return 400 not 500 when AIM blocks a request AIM guardrail blocks raised a bare HTTPException whose type and param serialized as the literal string "None", which broke OpenAI-SDK error parsing for downstream consumers. Switching AIM to raise a ProxyException surfaced a second bug: the shared error funnel re-derived the HTTP status from a nonexistent status_code attribute and downgraded the 400 to a 500. The funnel now honors an already-normalized ProxyException rather than rebuilding it, and ProxyException is excluded from llm_exceptions alerting so a content-policy block no longer pages on-call as an LLM API failure Resolves LIT-3751 * fix(guardrails): route all AIM rejection paths through ProxyException The block-action fix left two AIM rejection paths raising a bare HTTPException: the multimodal anonymize rejection and the output-side block. Both serialized type and param as the literal string "None", the same malformed shape the block fix removed. Funnel all three through a shared _rejection helper so they return a conformant OpenAI error body. The output block carries content_policy_violation; the multimodal rejection stays a plain invalid_request_error because it is a usage error, not a policy violation Resolves LIT-3751 * fix(guardrails): record AIM ProxyException blocks in failure logs Switching AIM blocks from HTTPException to ProxyException made _is_proxy_only_llm_api_error return False for them, so _handle_logging_proxy_only_error was skipped and the blocked prompt was dropped from the configured failure loggers. Classify ProxyException as a proxy-only error alongside HTTPException so guardrail blocks are recorded again, matching the prior behavior. The llm_exceptions alert suppression is a separate check and stays in place Resolves LIT-3751 * style(guardrails): use str | None over Optional[str] in AIM _rejection * style(guardrails): collapse AIM _rejection signature per black |
||
|
Sameer Kankute
|
1ccc1e5b23
|
chore: litellm oss staging160626 (#30527)
* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit |