mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-07 02:59:05 +00:00
203 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mateo-berri
|
d9aaa95978 |
Gate OpenAI prompt cache breakpoints on the real target and carry them through /v1/responses
The cache control hook also runs on litellm.responses() input. On a
GPT-5.6 deployment it wrapped a string-content item into a chat-shaped
{"type": "text"} part, which the Responses API rejects, and it never
marked input_text, input_image or input_file parts, so no breakpoint and
no prompt_cache_options reached the provider. Add the Responses part
types to the eligible block set and translate chat-shaped text parts on
non-assistant items to input_text in
ResponsesAPIRequestUtils.merge_prompt_management_input, which both the
async and the sync prompt management sites go through.
The dialect also fired for any GPT-5.6 name that resolved to provider
openai, including deployments pointed at a custom api_base that does not
understand prompt_cache_breakpoint. Decide it once per request from the
provider, the model map and the resolved api_base (request, then
litellm.api_base, then OPENAI_BASE_URL / OPENAI_API_BASE): only
api.openai.com and *.api.openai.com hosts speak the dialect, a top-level
prompt_cache_options opts a custom target in, and litellm_proxy/ targets
never get it. maybe_seed_default_injection_points takes api_base and
stamps the finished decision on the points as _litellm_openai_dialect so
the sync completion() path, whose hook params do not carry api_base,
honors it; maybe_inject_cache_control takes api_base from the
/v1/messages handler.
Eligibility now comes from a supports_prompt_cache_breakpoint model map
flag on the OpenAI gpt-5.6 entries, exposed through
litellm.utils.supports_prompt_cache_breakpoint, with the GPT version rule
kept only for models the map does not know. The OpenAI dialect no longer
reserves a slot for tool_config points, which OpenAI has no cache block
for, and with_prompt_cache_breakpoint plus the chat bridge helper return
a new block instead of mutating their input.
|
||
|
mateo-berri
|
7b6f537855 |
fix(anthropic_messages): price native /v1/messages vertex calls on the deployment location
The proxy pre-creates the logging object before the router picks a deployment, and the native /v1/messages handler never copied the deployment's vertex_location into the logging params it updates, so cost resolution fell back to the environment or the default region and priced every call on this surface with the regional endpoint uplift. Copy the explicitly configured location from the request's litellm params, the same source dispatch builds the request URL from, and register the new regional_endpoint_uplift_multiplier field in the cost map schema test. |
||
|
Mateo Wang
|
d58b1c8558
|
Merge pull request #37516 from BerriAI/litellm_gemini_prompt_cache_min_tokens_4096
fix(model_prices): set prompt_cache_min_tokens=4096 for Gemini 3.5/3.6/3.7 Flash and 3.1 Pro Preview |
||
|
mateo-berri
|
77716eeaed | fix(model_prices): set prompt_cache_min_tokens=4096 for Gemini 3.5/3.6/3.7 Flash and 3.1 Pro Preview | ||
|
Devin AI
|
dcd8bb3f38 |
test(model_prices): update DeepSeek V4 pricing expectations
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
mateo-berri
|
138c77023a |
fix: accept bool thinking param instead of crashing with AttributeError
litellm.completion(thinking=True) crashed pre-network in is_thinking_enabled
with a retryable APIConnectionError ('bool' object has no attribute 'get'),
so the router burned retries on a deterministic failure and proxy clients got
a traceback instead of a usable response.
validate_and_fix_thinking_param now coerces thinking=True to the enabled dict
with the default medium budget and drops thinking=False, and the remaining
dict-assuming thinking accessors (base config, bedrock converse, deepseek)
guard with isinstance so raw bools can never crash a transform.
|
||
|
Mateo Wang
|
e75b4b1c2a
|
Merge pull request #37362 from BerriAI/litellm_lit_5651_bedrock_guardrail_cost
feat(guardrails): count bedrock guardrail cost against spend and budgets |
||
|
mateo-berri
|
354b0c3a45 |
fix(guardrails): bill all chunks on mid-chunking block, strip client guardrail cost metadata, add cost map schema keys
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail call AWS billed for the logical request, not just the blocking chunk. Client-supplied metadata.standard_logging_guardrail_information is stripped at the proxy boundary so callers cannot forge (even negative) guardrail cost into spend, and guardrail_information_cost ignores negative or non-finite entry costs as defense in depth. The cost map schema test now allows guardrail_cost_per_unit and the guardrail mode. |
||
|
mateo-berri
|
e3a93c40be | fix(proxy): stop leaking the client_side_timeout marker to providers | ||
|
mateo-berri
|
ce82a440d3 | fix: register rust as a litellm param so it never leaks into provider request bodies | ||
|
mateo-berri
|
71d951bfc0 | chore(types): drop redundant comments around the bedrock batch params | ||
|
Marty Sullivan
|
0c5c9c79d7 |
fix(bedrock): carry s3_output_bucket_name and bedrock_tags through credential normalization
Registering the five managed-batch fields in all_litellm_params stops them leaking into extra_body, but two of them never reached the transformation that reads them. CredentialLiteLLMParams is a whitelist, so get_deployment_credentials_with_provider round-tripped the deployment and silently dropped s3_output_bucket_name and bedrock_tags before the files/batch/passthrough callers saw them. s3_bucket_name, s3_region_name and aws_batch_role_arn were added to that model for #25104; these two are the remainder of the same deployment config bedrock_tags is typed as a plain list rather than a stricter shape so a malformed value still reaches _validate_bedrock_tags and gets its own error message instead of a Pydantic one The preservation assertion previously round-tripped through GenericLiteLLMParams, which is extra="allow" and would hold even for a field nothing declares. It now also reproduces the CredentialLiteLLMParams normalization the proxy actually performs, and fails naming exactly the dropped fields without this change |
||
|
Marty Sullivan
|
b84dd6922e |
fix(bedrock): stop leaking managed-batch litellm_params to the provider
A Bedrock managed-batch deployment carries aws_batch_role_arn, s3_bucket_name, s3_region_name, s3_output_bucket_name and bedrock_tags in its litellm_params, and the batch and files transformations read all five from there. None was registered in all_litellm_params, so the param builder swept them into extra_body on every other route that deployment serves: Bedrock answers "aws_batch_role_arn: Extra inputs are not permitted" on Anthropic models and "extraneous key [aws_batch_role_arn] is not permitted" on Nova, Llama and Titan, so configuring batch turns every chat and embedding request to that model into a 400. Register them alongside the agentic-loop and callback-credential fields, which are listed for exactly this reason. The batch path is unaffected because GenericLiteLLMParams is extra="allow" and preserves them into litellm_params for the transformations that consume them. Before this, batch could only be configured on a deployment dedicated to batch; the same model group could not serve both. |
||
|
mateo
|
6517c1dc06 |
fix(cost): support cache_creation_input_token_cost in tiered pricing and make tier selection all-or-nothing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Yuneng Jiang
|
075781568d
|
test: remove tests that never execute
Three groups, all verified by running the suite rather than by inspection. 18 files whose every test function carries an unconditional @pytest.mark.skip, 39 test functions in total. They are collected on every CI run and always skip, so they advertise coverage the suite does not have. Reasons on the marks include "AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to using 'otel' for logging"; 26 of the marks predate 2025. 30 test functions with a byte-identical body and identical decorators to a sibling in the same file and class, differing only in name. Deleting one of each pair removes no coverage. Four further candidates were excluded because they override an inherited test, where deleting the override un-shadows the base class implementation instead of removing a duplicate. 9 test functions that a later definition of the same name shadows, so Python never binds them and pytest cannot collect them. One file that is a demo script rather than a test; its own docstring says to run it with python. Verification: collecting the 26 edited files gives 2,492 node IDs before and 2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9 shadowed deletions account for 0 (confirming at runtime that they were never collectable), nothing unexplained disappeared, and nothing new appeared. No other test or module imports any deleted symbol. |
||
|
mateo-berri
|
df5425675e | fix(schema): declare supports_tool_search in the model prices schemas | ||
|
Yassin Kortam
|
d0c65f83f1
|
fix(websearch): stop leaking interception control fields to providers (#36480)
The web-search interception hooks stamp _websearch_interception_emit_native_blocks and _websearch_interception_converted_stream onto kwargs to carry state across the agentic loop, but neither was registered in all_litellm_params. The param builder sweeps anything it does not recognize into the outbound request, so a provider that validates its body rejects the whole call: Bedrock Converse answers "_websearch_interception_emit_native_blocks: Extra inputs are not permitted" with a 400, which breaks every request interception touches on that route. Register both alongside their code-interpreter counterparts, which were already listed for exactly this reason. Resolves LIT-5391 |
||
|
Deepanshu Lulla
|
9ce96c2d34
|
feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418)
* feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars
Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and
two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both
setters after assigning litellm_trace_id so every JSON log record emitted within the
async request context carries trace_id and, when provided, session_id — enabling log
correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without
any changes to individual log call sites.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields
* fix(logging): always reset session_id_var to empty string when no session_id provided
* feat: gate request correlation IDs in logs behind request_correlation_in_logs flag
* refactor: move correlation ID injection into CorrelationContextFilter
* feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload
Plaintext log lines (json_logs off) now get the same trace_id/session_id
suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a
visible effect regardless of log format.
StandardLoggingPayload gets a new independent session_id field, populated
from litellm_session_id. trace_id's existing session_id-first fallback is
preserved when request_correlation_in_logs is off; with the flag on, an
explicit litellm_trace_id now takes priority over litellm_session_id so
the two fields carry genuinely independent values.
* fix(logging): restore correlation context after nested calls; sanitize correlation ids
Addresses two review findings on this PR.
CorrelationContextFilter's trace_id/session_id contextvars were set on every
Logging.__init__ but never reset, so a nested LiteLLM call sharing the same
asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call,
an MCP sampling call) would leave the outer request's subsequent log lines
stamped with the nested call's ids instead of its own. set_trace_id/
set_session_id now return their contextvars.Token, and Logging stores them
and resets both once its own success/failure handler actually completes,
via a new idempotent _restore_correlation_context() called from all four
terminal handlers.
set_trace_id/set_session_id also now strip control characters and bound
length before storing a caller-controlled trace_id/session_id, since these
values can originate from request input (litellm_session_id, x-litellm-
trace-id) and get interpolated into plain-text log lines - without this, a
caller could embed \r/\n or escape sequences to forge fake log entries.
* fix(logging): restore correlation context after nested calls, not before
The previous commit called _restore_correlation_context() as the first
line of each terminal handler, before that handler's own callback
dispatch loop runs. That's backwards: a nested LiteLLM call triggered
from within a callback (e.g. a guardrail's own LLM-as-judge call) would
then capture the *already-reset* value as its own pre-call baseline,
and its own reset would restore to that instead of the true outer
value - verified live to still leak.
success_handler/async_success_handler/failure_handler/async_failure_handler
are now thin wrappers: the original bodies move to
_success_handler_body/etc, called inside a try/finally that restores
context only once the full body - including any nested calls its own
callback dispatch triggers - has actually finished, mirroring proper
stack-scoped nesting semantics.
* test(logging): cover async_failure_handler's correlation-context restore
Codecov flagged the new async_failure_handler wrapper (try/finally around
_async_failure_handler_body) as uncovered - the method had no direct test
at all before this PR's refactor split it into a wrapper. Adds a test that
awaits it directly and asserts both that async_log_failure_event still
fires and that _restore_correlation_context() puts the pre-call
trace_id/session_id back.
* fix(logging): restore correlation context by value, not by contextvars.Token
veria-ai correctly flagged that contextvars.Token.reset() only works in the
exact Context it was created in, and litellm's async success path (and
streaming failure path) dispatch async_success_handler/async_failure_handler
via asyncio.create_task and the global logging worker - a different Context
than Logging.__init__ ran in. reset_trace_id/reset_session_id silently
swallowed the resulting ValueError, so the restore was a no-op for exactly
those paths. Verified independently: reproduced the raw contextvars
behavior, then confirmed litellm's async success dispatch really does go
through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py).
Logging now captures the pre-call *value* (not a Token) and restores via a
plain set_trace_id()/set_session_id() call, which works regardless of which
Task/Context calls it. reset_trace_id/reset_session_id are removed as
dead/unreliable code. Added a regression test that spawns __init__ and the
restore in different asyncio Tasks - confirmed it fails against the prior
Token-based commit and passes here.
* fix(logging): restore correlation context in the originating task too
Greptile's re-review correctly identified a remaining gap: for a
successful acompletion(), async_success_handler is dispatched via
asyncio.create_task + the global logging worker into a *different* Task
than the one wrapper_async/Logging.__init__ ran in. The prior fix (43c164a)
only restored the handler's own (detached, throwaway) Task - it never
touched the originating request Task, which keeps this call's trace_id/
session_id set for the rest of its own execution (e.g. nested calls made
via the same Task).
wrapper()/wrapper_async() in litellm/utils.py now restore the originating
Task's correlation context in a finally block once the whole call is done,
regardless of what detached logging tasks it spawned along the way. Since
the wrapped body rebinds its own `kwargs` local via function_setup(),
sharing the dict object doesn't work here; a small mutable holder carries
the constructed Logging instance back out to the outer wrapper instead.
_restore_correlation_context() is no longer guarded against repeat calls:
with value-based (not Token-based) restoration, each distinct Task that
calls it needs its own restore to take effect in that Task's own view of
the contextvars, so multiple calls (once per Task involved in an attempt)
are required, not just tolerated.
Added a regression test using mock_response to exercise the real success
dispatch path (asyncio.create_task + GLOBAL_LOGGING_WORKER) without a live
provider call, asserting the *test's own* (originating) task context is
restored after the call - this is exactly the case Greptile flagged and
the prior commit didn't cover.
* fix(logging): restore correlation context when function_setup itself fails
Greptile's 4th finding: if function_setup() constructs Logging() (whose
__init__ already mutates trace_id_var/session_id_var) and then raises
before returning - e.g. update_environment_variables() throws - the
caller's wrapper()/wrapper_async() never receives a logging_obj reference,
so its own restore-on-finally never fires. The correlation ids leak into
every subsequent log line on that thread/task with no way to clear them.
function_setup()'s own except block now restores the context itself in
that case, using whatever logging_obj it managed to construct before
failing (locals().get(), safe against the earlier failure modes where
logging_obj was never assigned at all).
Added a regression test that monkeypatches Logging.update_environment_variables
to raise after construction, confirmed it fails without this fix (the
leaked ids show up directly in the raised exception's own log line) and
passes with it. Broader sweep (test_utils.py, test_router.py,
test_main_module_header.py, streaming handler tests, plus all
logging-specific tests): 722 passed.
* fix(logging): don't assume every litellm_logging_obj is a real Logging instance
CI caught a real regression from the last commit: tests/test_litellm/llms/xai/test_xai_key_fallback.py
injects a minimal FakeLogging stand-in (only implementing
update_from_kwargs) as litellm_logging_obj for a narrow realtime-config
unit test, bypassing the real Logging class entirely. wrapper()/
wrapper_async()'s finally block and function_setup()'s except block both
unconditionally called _restore_correlation_context() on whatever ended up
in the holder, which doesn't exist on that stand-in.
_restore_correlation_context is new plumbing specific to this PR's
feature, not part of any pre-existing stand-in's expected interface, so
callers of it can't assume every object playing the litellm_logging_obj
role implements it. Added _restore_correlation_context_if_supported(),
a small getattr-guarded helper, and used it at all three call sites.
* fix(logging): don't restore context too early on setup failure or streaming
Two more findings from Greptile's 5th review round.
1. function_setup()'s except block restored correlation context *after*
logging the "Error in function_setup" exception, so that diagnostic log
line itself was stamped with the doomed call's ids instead of the outer
ids - misleading, since the failed call never produces anything else to
attribute those ids to. Restore now happens before the log call.
2. wrapper()/wrapper_async() restored the originating task's context as
soon as a streaming call returned, before the caller ever starts
iterating the CustomStreamWrapper it just got back. Any log lines
emitted while iterating (in the same thread/task) incorrectly showed
the pre-call ids instead of this call's own ones. The wrapper finally
block now skips the restore when the return value is a stream wrapper,
deferring to the terminal handler that already fires once the stream is
actually assembled/exhausted.
Both verified with tests that fail against the prior commit and pass
against this one. Broader sweep unchanged at 829 passing.
* fix(logging): best-effort correlation cleanup on abandoned streams
Greptile's 7th finding: if a caller returns a streaming response and never
fully consumes it - stops iterating early, drops the reference, cancels
it - the terminal handler that normally restores the originating task's
trace_id/session_id never fires, since it only runs once the stream is
actually assembled/exhausted. The ids leak into every subsequent log line
in that thread/task with no bound.
There's no reliable Python hook for "this was abandoned without being
closed" - CustomStreamWrapper has no close()/__aexit__/context-manager
convention today, and the only automatic option is __del__, whose timing
is inherently unpredictable (delayed by cyclic GC, not guaranteed at
interpreter shutdown, can run on a different thread). This is a best-effort
safety net, not a guarantee, and is documented as such in the docstring.
Testing this via real garbage collection proved unreliable in practice:
per-chunk logging submits work to a thread pool executor whose worker
thread transiently holds its own bound-method reference to the wrapper
until that task completes, so refcount doesn't hit zero on a
deterministic schedule even with polling. Tests call __del__ directly
instead - a plain method, safe to invoke early - which exercises exactly
the restore logic real garbage collection would eventually trigger,
plus a case confirming a broken logging_obj can never make __del__ raise.
* fix(logging): restore consumer's context at every real stream exit point
Two more findings from this round.
Veria AI: even a *fully consumed* stream never restored the actual
consuming thread/task's correlation context. The terminal success dispatch
(dispatch_success_handlers via asyncio.create_task for async, or
success_handler via the shared executor for sync) only restores whatever
detached context it runs in - never the caller's own thread/task that's
running the for/async for loop. Same root cause as the wrapper-level fix
two rounds ago, just missed for the streaming-completion path.
Greptile: explicit aclose() (client disconnect, router fallback aborting
a partial stream) closed the underlying stream without restoring
correlation context either, since request wrappers intentionally skip
restoration for returned streams and no terminal handler runs on this
path.
Added CustomStreamWrapper._restore_consumer_correlation_context(), called
from every point control genuinely returns to the consumer: the final
raise StopIteration/StopAsyncIteration on natural exhaustion (both sync
branches, both async branches), _handle_stream_fallback_error (the shared
choke point for all three failure-raising call sites), and aclose(). __del__
now delegates to the same helper instead of duplicating it.
Verified with tests extending the existing streaming-exhaustion cases to
assert the consuming context is restored after the loop completes (fails
against the prior commit, passes now), plus a dedicated aclose() test.
Broader sweep: 832 passing.
* fix(logging): don't let a delayed __del__ finalizer clobber a newer active call
If an abandoned stream's __del__ fires late (after cyclic GC delay), a
different call may have already taken over the correlation contextvars in
the same Task/thread. Restoring unconditionally would stomp that active
call's trace_id/session_id with the abandoned stream's stale pre-call
snapshot. __del__ now only restores when the contextvars still hold the
ids this call itself set.
* fix(logging): compare sanitized ids in the __del__ ownership guard
set_trace_id()/set_session_id() sanitize (strip control chars, bound length)
before storing, so the contextvar's value can differ from the raw
litellm_trace_id/litellm_session_id. The __del__ ownership guard was
comparing against the raw values, so a caller-supplied id containing control
characters or exceeding 256 chars would never match, permanently skipping
cleanup. Capture what set_trace_id()/set_session_id() actually stored and
compare against that instead.
* fix(logging): restore consumer context on the synthesized finish_reason chunk
Both __next__ and _finalize_completed_stream() have a branch that fires when
the underlying stream ends without ever emitting an explicit finish_reason
chunk: they synthesize one via finish_reason_handler() and return it. A
consumer that stops as soon as it sees finish_reason - a common pattern -
never calls __next__()/__anext__() again, so the existing restore in the
sent_last_chunk-is-True StopIteration branch never runs for them. The
underlying stream is already exhausted at this point regardless of whether
the caller keeps iterating, so restoring here is safe.
* fix(logging): don't restore correlation context before the caller receives the final chunk
The previous fix (5147c69186) restored context immediately before returning
the synthesized finish_reason chunk from __next__/_finalize_completed_stream,
reasoning that completion_stream was already exhausted. But that chunk is
still this call's own data, and the caller's own application-level log
statements processing it run in the same synchronous frame right after the
return - restoring first made those lines carry the wrong (outer) ids,
exactly what wrapper()/wrapper_async() deliberately avoid by not restoring
while a stream is being iterated.
Revert to not restoring there. A caller that keeps iterating still gets a
correct, deterministic restore on its very next __next__()/__anext__() call
(completion_stream is exhausted, so that immediately re-raises
StopIteration/StopAsyncIteration through the already-restoring branch). A
caller that stops right after finish_reason relies on aclose() or the
best-effort __del__ guard, same as any other stream the caller doesn't fully
exhaust.
* refactor(logging): hoist a safely-hoistable function-body import to module top
CorrelationContextFilter.filter()'s `import litellm` was a function-body
import; verified it can move to module top without a circular-import failure
(litellm/__init__.py already imports from litellm._logging before setting
request_correlation_in_logs, but a bare `import litellm` only binds the
already-in-sys.modules module object - the attribute itself isn't read until
filter() actually runs, by which point litellm is fully initialized).
* test(logging): move correlation tests into their conventionally-mapped files
tests/test_litellm/ mirrors litellm/ in a parallel path. Correlation tests
for the Logging class (litellm_logging.py), function_setup/wrapper_async
(utils.py), and CustomStreamWrapper (streaming_handler.py) had all landed in
test_logging.py, which only maps to litellm/_logging.py itself. Moving each
group to its correctly-mapped file: test_litellm_logging.py (Logging class
init/restore), test_utils.py (function_setup, wrapper_async), and
test_streaming_handler.py (CustomStreamWrapper) in the next commit.
test_logging.py keeps only what actually exercises _logging.py's own
contextvars/filters/formatters/sanitization. No behavior change - same
assertions, same coverage, just relocated.
* fix(logging): restore correlation context unconditionally in wrapper()'s sync path
Blocking finding from review: a caller-visible correlation feature was
silently misattributing one request's logs to a different, unrelated one on
the sync/threaded path. wrapper()/wrapper_async() both left trace_id/session_id
"open" across a stream's entire iteration so the caller's own log lines while
consuming it would carry the right ids. That's safe for wrapper_async(): each
async call gets its own asyncio Task with its own copy of the contextvars,
and Tasks are never recycled across requests, so a leftover value can only
ever affect that one already-abandoned Task.
It is not safe for wrapper() (sync): a plain OS thread has no such per-call
isolation, and a thread pool's worker threads *are* recycled across unrelated
requests. If a sync stream was abandoned (client disconnect, early break, an
uncaught exception) without ever being exhausted or closed, nothing restored
its contextvars, and a pool could later hand that same thread to a completely
different call, which would inherit the abandoned request's ids as its own
"pre-call" baseline and then restore back to that poison when it finished -
permanently misattributing every subsequent log line on that thread,
including its own, to the abandoned request. Strengthening the __del__
finalizer already added for this can't fix it: finalizer timing is exactly
what a permanently-reused thread can't rely on.
wrapper() now restores unconditionally in its own finally, before a sync
stream is ever handed back to the caller. The trade-off: a sync stream
consumer's own application-level log statements while iterating no longer
automatically carry this call's ids (litellm's own internal per-chunk
logging is unaffected, since it's dispatched separately). That's an
acceptable cost for eliminating a silent cross-request misattribution bug.
wrapper_async() keeps the existing conditional (skip-if-streaming) behavior,
justified by the Task-isolation argument above; CustomStreamWrapper's
__del__/aclose()/next-iteration restore machinery remains meaningful and
necessary there.
This also simplifies wrapper()/wrapper_async() back toward their original
shape: both previously used a mutable-dict-holder split into a separate
_body function to smuggle logging_obj/result out to an outer finally,
working around function_setup() rebinding its own local `kwargs`. That
restructuring is no longer needed - `logging_obj` (and, for wrapper_async(),
`result`) were already function-level locals in scope for a plain
try/finally; three of wrapper_async()'s retry-return statements now assign
through `result` first so it accurately reflects what's actually returned
even on a retry path.
Regression test: test_abandoned_sync_stream_does_not_contaminate_a_later_call_on_the_same_thread
in test_streaming_handler.py reproduces the exact reported scenario with a
real single-worker ThreadPoolExecutor - confirmed it fails with the prior
(skip-restore-on-stream) wrapper() and passes with this fix.
* refactor(logging): use Mapping instead of bare dict for read-only params
_get_standard_logging_payload_trace_id/_session_id only read litellm_params
(.get() calls, no mutation) - annotate it as Mapping[str, Any] rather than a
bare mutable dict, per the repo's no-mutable-collection-in-annotation rule.
* fix(logging): scope request_correlation_in_logs to the async/proxy path only
Blocking review finding: wrapper() (the sync entry point) used the same
skip-restore-on-stream design as wrapper_async(), but a plain OS thread has
no per-call context isolation the way an asyncio Task does, and a thread
pool's worker threads are recycled across unrelated requests - an abandoned
sync stream could leave its ids stuck on a thread a pool later hands to a
completely different request, misattributing that request's logs. A fix
existed and was tested (restore unconditionally in wrapper()'s own finally),
but it doesn't benefit this feature's primary consumer - the proxy only ever
calls the async entry point - and carries sync-specific complexity this PR
doesn't need.
Scope the feature to async only instead: Logging.__init__() takes a new
supports_correlation_logging parameter (default True), threaded down from a
new function_setup(..., is_async_call: bool = True) parameter. wrapper() is
the one caller that passes is_async_call=False; every other function_setup()
call site (wrapper_async(), the router, and proxy/MCP-internal call sites)
is already async and keeps the default. With
supports_correlation_logging=False, Logging.__init__() never calls
set_trace_id()/set_session_id() at all, so a sync call has nothing to leak
in the first place. wrapper() reverts to its pre-review shape with no
correlation-specific code at all.
StandardLoggingPayload's own trace_id/session_id fields are unaffected
either way - they're a deterministic per-call read of
self.litellm_trace_id/self.litellm_session_id, not ambient contextvar state,
so they were never exposed to the cross-request bug.
Full sync/direct-SDK support (stamping + its own safe-restore mechanism) is
deferred to a follow-up PR; the fix and its regression test already exist in
this branch's history at commit 9f3a20f4b2 and can be resurrected there.
Tests: replaced the two wrapper()-level tests with ones proving the new
invariant (sync calls, streaming and non-streaming, never touch
trace_id_var/session_id_var even when the caller explicitly passes
litellm_trace_id/litellm_session_id), and added a direct unit test for the
supports_correlation_logging=False gate on Logging.__init__ itself. Verified
live: a real proxy (Postgres-backed, real OpenAI calls) shows clean
trace_id/session_id isolation across two concurrent sessions with no
cross-contamination; a standalone script confirms real sync SDK calls
against a real model never touch the correlation contextvars.
* feat(logging): fall back to W3C traceparent/baggage for trace_id/session_id
request_correlation_in_logs previously only resolved trace_id/session_id from
litellm-specific sources: x-litellm-trace-id/x-litellm-session-id headers, a
generic x-<vendor>-session-id header, or Anthropic-style metadata.user_id. If
none were present, trace_id fell back to an auto-generated UUID unrelated to
anything else, and session_id stayed empty - even when the caller already had
real distributed-tracing instrumentation sending the actual industry-standard
headers for this.
Add a fallback to the W3C Trace Context traceparent header (trace-id
component) and W3C Baggage header (session.id entry), so a request already
carrying real OpenTelemetry trace context correlates litellm's own logs with
the same trace in the caller's observability backend (Datadog, Honeycomb,
Tempo, etc.) instead of getting an unrelated generated id. Precedence is
unchanged for existing sources: explicit litellm headers and the Anthropic
metadata path both still win over this new fallback, which only fires when
neither found anything. trace_id and session_id are resolved independently
here (unlike the existing chain_id mechanism, which uses one shared value for
both), since traceparent and baggage are semantically distinct W3C concepts.
New helpers _trace_id_from_traceparent/_session_id_from_baggage in
litellm_pre_call_utils.py parse the header formats directly (no new
dependency - both are simple fixed-width/delimited strings), wired into
LiteLLMProxyRequestSetup.add_litellm_metadata_from_request_headers() only
when the corresponding litellm_trace_id/litellm_session_id key isn't already
set by the existing paths.
Verified live against a real proxy: a bare traceparent header produces a log
trace_id exactly matching its trace-id component; a traceparent alongside an
explicit x-litellm-trace-id header (different value) produces a log showing
the explicit header's value, proving precedence.
* fix(logging): reserve trace_id/session_id in JsonFormatter against message-content spoofing
JsonFormatter merges keys parsed from the message body before applying extra
record attributes, and the extra-attributes loop skips a key that's already
present. A caller-controlled log message that happens to parse as JSON/dict
with a "trace_id"/"session_id" key (e.g. the proxy logging a raw request-header
dict) could therefore make the JSON record carry the attacker-supplied value
instead of the real correlation context set via CorrelationContextFilter.
trace_id/session_id are now applied from the LogRecord's own attributes after
message-content parsing, unconditionally overwriting anything the message body
claimed for those two keys.
* style(logging): fix import order (ruff I001) in _logging.py and litellm_logging.py
- _logging.py: import litellm belongs after the stdlib from-imports, grouped
with the other litellm.* imports, not before them.
- litellm_logging.py: the refactor to Mapping introduced a second, separate
`from collections.abc import Mapping` instead of merging it into the
existing `from collections.abc import Callable` import.
Caught by the strict-rule budget gate (ruff-strict-budget.json caps I001 at
0 new violations); both auto-fixed with `ruff check --fix --select I001`.
* style(logging): freeze mutable-collection constructions flagged by LIT002
Five sites in this PR's diff built a mutable list/dict literal instead of a
frozen value: a plain list of optional strings in CorrelationPlainFormatter,
a `kwargs or {}` fallback, a `metadata or {}` fallback, two `[...]` candidate
orderings, and a `dict(headers)` copy feeding a dict comprehension. Each is
build-once/read-only, so this rewrites them as tuples, MappingProxyType, or a
plain conditional `.get()` instead of seeding then reading a fresh mutable
collection - no behavior change, confirmed by the existing test suite.
Caught by the type-discipline budget gate (LIT002 capped at 0 new
violations).
* fix(logging): reserve trace_id/session_id even when no correlation context is active
Live-proxy verification surfaced a gap in the earlier message-content-spoofing
fix (7f390a57fc): that fix only overwrites trace_id/session_id from the
LogRecord's own attribute, so it does nothing for a log line emitted before
CorrelationContextFilter has stamped anything on this record (e.g. the
"Request Headers" debug line, which fires before Logging.__init__() runs for
the request). On such a record, a caller-supplied header literally named
trace_id/session_id still got promoted into the JSON output via the embedded
JSON/dict-repr parser, since there was no genuine value to protect.
Fixed at the source: trace_id/session_id are now excluded unconditionally from
the message-content-parsing promotion step, not just superseded afterward.
Verified live against a real proxy - the exact adversarial request (headers
literally named trace_id/session_id) no longer leaks into any JSON log record.
Added a regression test for this no-active-context variant specifically,
confirmed it fails against the prior commit and passes now.
Also fixes an unrelated basedpyright regression from an earlier rebase's
conflict resolution: litellm/utils.py's `logging_obj` was incorrectly
re-annotated `Final` at its second assignment in function_setup() (it's first
declared `None` a few lines earlier), which basedpyright correctly rejects.
* fix(proxy): stop logging the raw W3C baggage session_id value
_session_id_from_baggage() extracts the caller-controlled session.id entry
verbatim - it isn't sanitized until set_session_id() runs later in
Logging.__init__(). The debug log line for this extraction interpolated the
raw value directly, so a caller could embed terminal control characters or
ANSI escape sequences that forge/alter plaintext log output for anyone
tailing the proxy's logs.
Verified live: a baggage header with an embedded ANSI escape reached the
terminal as a real, unescaped control sequence before this fix. Drops the
value from the log line entirely (the extraction succeeding is enough signal
on its own) rather than sanitizing-then-logging, matching veria-ai's
suggestion. Added a regression test using caplog that fails against the prior
commit and passes now.
* fix(logging): restore consumer context only after stream-failure exception mapping
_map_anthropic_exception/_map_aleph_alpha_exception synchronously log a debug
diagnostic (the raw status code) as part of exception_type()'s mapping.
_handle_stream_fallback_error restored the consumer's outer correlation
context before calling exception_type(), so that diagnostic log line carried
the outer (or empty) trace_id/session_id instead of the failing stream's own -
flagged by Greptile.
Moved the restore to run after mapping completes, matching the same
restore-after-not-before pattern already applied elsewhere in this file for
success/finish_reason handling. Added a regression test that captures the
correlation context live during a mocked exception_type() call; fails against
the prior commit, passes now.
* fix(logging): restore consumer context only after aclose()'s stream close completes
aclose() restored the consumer's outer correlation context as its first
statement, before awaiting the underlying provider stream's own aclose()/
close(). If that close attempt raises, the except branch's debug diagnostic
ran under the already-restored outer context instead of the closing stream's
own trace_id/session_id - flagged by Greptile, same restore-too-early pattern
as the stream-failure fix in f1cf9589d6.
Moved the restore to the end of aclose(), after the close attempt (and its
diagnostic logging) completes. Added a regression test with a fake stream
whose aclose() raises, capturing the correlation context live during the
diagnostic log call; fails against the prior commit, passes now.
* style(logging): satisfy new strict-lint budgets introduced upstream (Final, ANN401, S110, TRY300, kwargs typing)
Rebasing onto litellm_internal_staging pulled in 116 upstream commits that
introduced/tightened several lint gates this PR's own code now trips:
- LIT010 (every local/module-level variable must be Final): added Final
annotations across _logging.py, litellm_logging.py, streaming_handler.py,
litellm_pre_call_utils.py, and utils.py. Where a name is genuinely
reassigned (logging_obj: starts None, later set to the real object) or
branch-assigned, either restructured into a single ternary expression
(ordered_candidates) or suppressed with `# rebind-ok: <reason>` matching
this repo's documented escape hatch.
- LIT011 (parameter mutation): suppressed the two new `data[key] = value`
writes in litellm_pre_call_utils.py with `# rebind-ok`, matching the
unsuppressed precedent already used for every other `data[...]` write in
the same function - `data` is an intentional out-param there.
- ANN001/ANN003/ANN202 (missing parameter/return type annotations): fully
typed success_handler/_success_handler_body, their async twins, and
failure_handler/_failure_handler_body/async variants in litellm_logging.py,
plus function_setup in utils.py (added Rules to its existing TYPE_CHECKING
block for the rules_obj: Rules annotation).
- ANN401 (explicit Any disallowed): suppressed with `# noqa: ANN401` on the
handful of genuinely-heterogeneous result/*args/**kwargs parameters, since
ordinary suppression is this repo's documented path.
- S110 (try/except/pass): added to the existing BLE001 noqa on the one
best-effort correlation-cleanup try/except this PR added.
- TRY300 (return inside try): moved two `return result` statements into
`else:` blocks in the retry-fallback paths this PR's own diff touched.
- reportPrivateUsage (basedpyright): renamed the two new
StandardLoggingPayloadSetup static methods (get_standard_logging_payload_
trace_id/session_id) to drop their leading underscore, since they're
genuinely called from a sibling module-level function in the same file.
No behavior change - confirmed by the full existing test suite (819 passed)
plus all four lint gates (ruff format, ruff-strict, type-discipline,
basedpyright) passing clean.
* fix(lint): stop RUF100 flagging noqa suppressions the strict gate needs
CI's plain "ruff check" job uses the default ruff.toml, a narrower config
than ruff-strict.toml (used only by the strict-rule budget gate). ANN401 and
S110 aren't enabled in the default config, so RUF100 (unused-noqa) flagged
the `# noqa: ANN401`/`# noqa: ...,S110` suppressions this PR added as pointless
under that config, even though they're genuinely needed under ruff-strict.toml.
- ANN401: added to ruff.toml's existing `lint.external` list (same mechanism
already used for C901/TID251, enforced by the strict gate but not by this
config) - these Any usages are genuinely dynamic/forwarded, so the
suppression itself is correct and just needed registering.
- S110: fixed the underlying code instead of registering another external
code - the try/except/pass in
CustomStreamWrapper._restore_consumer_correlation_context now logs at
debug level on failure (matching the existing best-effort-cleanup pattern
in _record_partial_usage_for_failure elsewhere in this file), which
satisfies S110's own suggestion directly and needs no suppression at all.
Verified against both ruff.toml and ruff-strict.toml directly, plus all
three other gates (ruff format, type-discipline, basedpyright) and the full
test suite (821 passed).
* fix(lint): scope the ANN401 exemption to file level instead of a repo-wide noqa
Ruff has no per-line-scoped way to register a noqa code across configs (that
requires the default ruff.toml's lint.external list, which is repo-wide in
scope even though the noqa itself is per-line). Since ruff does support
file-level exemptions via per-file-ignores, and ANN401 only needed exempting
in exactly two files, moved the exemption there instead:
- ruff-strict.toml: added [lint.per-file-ignores] disabling ANN401 for
litellm_logging.py and utils.py specifically, with a comment explaining
why (heterogeneous response/forwarded-args parameters with no fitting
concrete type - already verified by trying CostResponseTypes and hitting
a real basedpyright mismatch).
- ruff.toml: reverted the ANN401 entry from lint.external - no longer
needed, since there's no `# noqa: ANN401` left anywhere for RUF100 to
second-guess.
- Removed the now-redundant `# noqa: ANN401` from the 10 affected
parameters in both files, keeping the existing kwargs-ok reasons and
adding a short inline comment on the `result`/`*args` lines pointing at
the ruff-strict.toml exemption for context.
Verified against both configs directly (ANN401 clean under ruff-strict.toml
for these files, RUF100 clean under the default config), all four gates
(ruff format, ruff-strict, type-discipline, basedpyright), and the full
test suite (821 passed).
* fix(logging): redact credential-shaped trace_id/session_id before stamping log records
CorrelationContextFilter stamps trace_id/session_id onto a LogRecord after
SecretRedactionFilter has already run, so a caller-controlled value (e.g. via
x-litellm-trace-id or a W3C baggage header) that happens to look like a real
credential reached JSON and plaintext logs unredacted. Apply the same
credential redaction already used elsewhere in this module at
_sanitize_correlation_id(), the single choke point both set_trace_id() and
set_session_id() route through, so every caller-facing entry point is covered
without depending on filter ordering.
* fix(logging): restore correlation context when a stream's max-duration timeout fires
CustomStreamWrapper.__anext__() called _check_max_streaming_duration() before
entering its try block, so the litellm.Timeout it raises bypassed the except
Exception -> _handle_stream_fallback_error path entirely, leaking the timed-out
stream's own trace_id/session_id into whatever the consumer's task logs next.
Move the check inside the try so it flows through the same restoration path
every other stream failure already uses.
* test(streaming): make dispatch_failure_handlers mock awaitable for the async max-duration test
Moving _check_max_streaming_duration() inside __anext__()'s try block (prior
commit) means a max-duration Timeout now dispatches failure handlers through
the same path every other stream failure already uses, instead of bypassing
it entirely. dispatch_failure_handlers is async on the real Logging class;
the test's plain MagicMock logging_obj made asyncio.create_task() choke on a
non-coroutine return value once that path actually got exercised.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Yassin Kortam
|
87dbb632b2
|
test(utils): pin the register_model replay test to the recorded half (#35994)
test_reapply_runtime_registrations_replays_register_model_overrides asserts that a fetched catalog value survives the replay for a key an operator override does not mention. Any Router still alive in the process re-asserts its own deployments first, so a router serving openai/gpt-4o writes its model_info over that catalog value and the assertion reads the router's number instead. Routers built by earlier tests stay in the weak set until they are collected, which made the test depend on collection timing and fail intermittently in shards that run the router tests alongside it. The live-router rebuild is covered in test_router_model_cost_isolation.py, so this test now runs with the replay callback unset and exercises the recorded registrations it is about. |
||
|
Yassin Kortam
|
8ec562f279
|
fix(ai21): resolve the documented AI21_API_KEY instead of a misspelled name (#35985)
get_api_key resolved the ai21 key from AI211_API_KEY, with a doubled 1. Every other ai21 code path reads AI21_API_KEY, including the validate_environment branches that report it as the missing one, so the name a user is told to set was ignored here. No user path reaches this branch today, since every provider-resolution site rewrites custom_llm_provider to ai21_chat and sets the key from a correctly spelled read first, so this is a correctness fix rather than a bug fix. It is worth making because the env-var documentation gate reads this call site: leaving the misspelling in place would require a row for AI211_API_KEY in the environment variables reference table, which would turn a typo into public API |
||
|
Yassin Kortam
|
347798b80e
|
fix(router): keep custom model_info across a price data reload (#35491)
A price data reload replaced litellm.model_cost wholesale, discarding every runtime registration: the deployment model_info the Router registers from model_list, and pricing overrides passed to litellm.register_model. Custom model groups lost max_input_tokens / max_output_tokens in /model_group/info, and a deployment whose backend model is in the catalog silently reverted to upstream values. Runtime registrations are now recorded and replayed on top of the freshly fetched catalog. Router._pre_call_checks resolved the per-deployment model name only after the model-info lookup, so an unregistered model left it unset and the supported params check ran against the bare model group name, raising "LLM Provider NOT provided" out of deployment selection. The name is now resolved first, and an unresolvable provider skips that check rather than failing the request. Resolves LIT-4675 |
||
|
Mateo Wang
|
bf1a8fe403
|
Merge pull request #35270 from BerriAI/litellm_gpt_pricing_change
fix(pricing): correct gpt-5.6 prices for openai, bedrock, and flex long context |
||
|
mubashir1osmani
|
62aebaf035 |
fix(pricing): bill gpt-5.6 flex requests above 272k at the flex long-context rate
OpenAI publishes a long-context column on the Flex tier, at half the standard long-context rate. We had no field for it, so a >272k flex request fell through to the standard long-context price and billed 2x: Terra $4/$18 instead of $2/$9, Luna $0.40/$1.80 instead of $0.20/$0.90, Sol $10/$45 instead of $5/$22.50. Adding the values to the cost map alone does nothing, because get_model_info builds ModelInfoBase from an explicit kwargs list and silently drops any key not named there. Declare the four *_above_272k_tokens_flex fields and wire them through, then add the values for sol, terra, luna, and the gpt-5.6 alias. That same gap was already swallowing cache_creation_input_token_cost_flex, _priority, and _above_272k_tokens, which were present in the cost map but never reached the calculator; they are wired through here too. Fast mode (ex-Priority) publishes no long-context column, so nothing is added there rather than deriving a rate by analogy. |
||
|
mateo
|
b0a48d516c |
test(fireworks_ai): align Kimi output-limit expectations with cost map fix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
a7e665620b
|
fix: match exact class in callback dedup so a custom subclass does not block a built-in logger (#34804) | ||
|
Krrish Dholakia
|
2f502a1bfc |
fix(cost_tracking): map cache_write_tokens on Responses API usage path
The Responses API (/v1/responses) usage transform rebuilt prompt token details and dropped OpenAI's input_tokens_details.cache_write_tokens, so gpt-5.6 cache-creation tokens were never logged or billed via that route. Map it in the transform, and make PromptTokensDetailsWrapper keep cache_write_tokens and cache_creation_tokens in sync on assignment. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Krrish Dholakia
|
e6ec153243 |
fix(cost_tracking): map OpenAI cache_write_tokens for prompt cache creation billing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
Tin Chi Lo
|
d966122249 |
fix(fireworks_ai): correct glm-5p2 prompt-cache read price to $0.14/1M
glm-5p2 (and its fireworks_ai/glm-5p2 alias) carried cache_read_input_token_cost of 2.6e-07, the GLM 5.1 rate; the entry was seeded from the wrong row. Fireworks' standard serverless rate for GLM 5.2 is $0.14/1M = 1.4e-07, so every prompt-cache hit was billed at nearly double the real rate. Corrects the value in both the canonical map and the bundled backup. The existing fireworks cost-calculator test now reads the cached rate from the map instead of hardcoding it, so it tracks the shipped value. |
||
|
yuneng-jiang
|
04a5ebb94d
|
chore(ci): merge oss branch (#33784)
* fix(embeddings): accept encoding_format='float' for vertex_ai/gemini embeddings (#33617)
OpenAI SDKs (and litellm's own client since ~1.84) send
encoding_format='float' by default, but the vertex embedding config only
supports ['dimensions'], so get_optional_params_embeddings raised
UnsupportedParamsError at the provider default value. Any
OpenAI-compatible client talking to a litellm proxy with vertex
embedding models got a 400 unless the operator set proxy-wide
drop_params: true.
Float lists are exactly what the vertex API returns, so the param is a
no-op: pop it before validation. Other values (e.g. 'base64') keep the
existing unsupported-param behavior (dropped with drop_params, raise
otherwise).
Fixes #33173
Co-authored-by: Mihidum Hettiyahandi <55163074+mihidumh@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add Singulr guardrail integration for LiteLLM gateway (#31302)
* singulr guardrail support for litellm gateway
* Update litellm/proxy/guardrails/guardrail_hooks/singulr/singulr.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix comments
* improvement
* fix: resolve review comments and implement requested improvements
* fix:Guardrail bypass through uninspected messages
* fix:tool text scanning
* fix: Legacy function definitions bypass scanning by adding indirect message scaning
* chore: remove unintended basedpyright budget file
* fix:Response schema bypasses guardrail scanning (response_format.json_schema)
* chore: restore basedpyright-code-budget.json and update lint baselines
Restores the file deleted in
|
||
|
Tin Chi Lo
|
ba70189e32 |
fix(router): resolve prompt cache minimum per model instead of a flat 1024
MINIMUM_PROMPT_CACHE_TOKEN_COUNT was a flat 1024 described as "minimum number of tokens to cache a prompt by Anthropic". Anthropic's minimum cacheable prefix is per-model and ranges from 512 to 4096, and it can differ per platform for the same model, so one constant is wrong in both directions is_prompt_caching_valid_prompt gates PromptCachingDeploymentCheck, which is what optional_pre_call_checks: ["prompt_caching"] turns on. When it believes a prompt is cacheable, async_filter_deployments pins routing to whichever deployment previously served that prefix. For a prompt between 1024 and 4096 tokens on Opus 4.6, Opus 4.5 or Haiku 4.5, litellm judged it cacheable and constrained routing while the provider never cached it, so the pin cost load balancing for nothing. In the other direction Fable 5 caches from 512 tokens, so a 512 to 1024 token prefix was refused a pin it had earned The minimum now resolves from prompt_cache_min_tokens in the model cost map, which keeps it current with new models and lets the Bedrock override for Fable 5 fall out of the existing per-entry keys with no special casing. MINIMUM_PROMPT_CACHE_TOKEN_COUNT stays as a global escape hatch when explicitly set, and as the fallback for models the cost map has no entry for async_filter_deployments only ever receives the model group alias, never a model name, so it resolves the threshold from healthy_deployments instead. A group may mix models with different minimums, so it takes the max: a prompt is only treated as cacheable when it clears every member's minimum, because an unnecessary pin is the defect being fixed while a missed pin only forfeits an optimization Gemini context caching shares this gate and has the same defect; its entries are left unset so they keep today's behavior, tracked separately in LIT-4525 |
||
|
mateo-berri
|
598fa9d64d | feat(pricing): add gemini-omni-flash-preview with video output token pricing | ||
|
Mateo Wang
|
8447cd3ad3
|
Merge pull request #32836 from BerriAI/litellm_gemini_image_supports_reasoning_31766 | ||
|
Mateo Wang
|
5e23a5ab05
|
fix(bedrock): gate in-place system role messages on model support for Claude Invoke (#32831)
* fix(bedrock): gate in-place system role messages on model support for Claude Invoke * feat(bedrock): default unmapped Claude 4.8+ to in-place system role handling via fallback rule |
||
|
Mateo Wang
|
4737e75c86
|
fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map (#32840)
* fix(bedrock): add jp.anthropic.claude-opus-4-8 to model cost map * test: use apac regional profile for cost-map fallback test since jp now has an entry |
||
|
mateo-berri
|
e5421bfe1e
|
fix(model_cost): add missing backup entries for gemini image models
gemini/gemini-3.1-flash-image, vertex_ai/gemini-3-pro-image, and vertex_ai/gemini-3.1-flash-image existed in the root pricing JSON but not in litellm/model_prices_and_context_window_backup.json, leaving deployments with LITELLM_LOCAL_MODEL_COST_MAP=True unprotected. Copies the root entries into the backup verbatim and extends the regression test to cover all ten gemini image models, asserting each exists in the local cost map so a missing backup entry fails the test instead of passing vacuously |
||
|
Deepanshu
|
fd862bb2b8
|
fix(model_cost): add supports_reasoning: false to gemini/gemini-3-pro-image | ||
|
Deepanshu
|
75dd70a678
|
fix(model_cost): add supports_reasoning: false to Gemini image generation models
vertex_ai/gemini-2.5-flash-image, vertex_ai/gemini-3-pro-image-preview, vertex_ai/gemini-3.1-flash-image-preview, gemini/gemini-3-pro-image-preview, and gemini/gemini-3.1-flash-image-preview were missing supports_reasoning entries; _supports_factory then fell through to the vertex_ai provider-level config which returns true, causing requests with reasoning_effort to be sent to an API that rejects them. |
||
|
tin-berri
|
bf02a4a47f
|
test: add /v1/messages to supported_endpoints schema enum (#32739) | ||
|
devin-ai-integration[bot]
|
a874de6ac6
|
feat(models): add GPT-5.6 (sol/terra/luna) pricing and metadata (#32659)
* feat(models): add GPT-5.6 (sol/terra/luna) pricing and metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: allow gpt-5.6 service-tier cache-write keys in model prices schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix: floating point entry errors --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
734fd29e00
|
fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) (#32389)
* fix(utils): resolve bedrock regional inference profiles to regional pricing in get_model_info (LIT-4056) * test(register_model): use a triple provider prefix as the unresolvable-key fixture get_model_info now resolves bedrock/bedrock/... like a routing prefix, so the double-prefix fixture stopped exercising the register_model fallback path. Lock the new double-prefix resolution in as a model-info regression test |
||
|
Mateo Wang
|
8bb4e62412
|
feat(tencent): add Tencent TokenHub as a provider (#31903)
* feat(tencent): add Tencent TokenHub as a provider Tencent TokenHub is OpenAI- and Anthropic-compatible. This registers it as a new provider: TencentChatConfig routes /v1/chat/completions and gates the thinking/reasoning_effort params behind supports_reasoning, and TencentAnthropicMessagesConfig routes the Anthropic-compatible Messages API. Adds cost tracking, the deepseek-v4-pro/flash model entries, and provider endpoint support metadata. * test(tencent): add unit tests for Tencent TokenHub provider Covers TencentChatConfig (chat completions) and TencentAnthropicMessagesConfig (messages API) across transformation, param mapping, URL building, and header validation, plus get_optional_params routing. Tests mock supports_reasoning to stay independent of remote model cost data. * fix(tencent): correct max_output_tokens and reuse parent messages env validation Raise max_output_tokens/max_tokens for tencent/deepseek-v4-pro and tencent/deepseek-v4-flash from 8192 to 384000, matching Tencent TokenHub's published DeepSeek-V4 output limit; the 8192 value mirrored the native DeepSeek default and would have rejected valid larger requests before they reached Tencent Delegate validate_anthropic_messages_environment to the parent via super() so the Tencent messages endpoint keeps content-type and anthropic-beta header injection instead of dropping them, keeping only the TENCENT_API_KEY resolution overridden Add regression tests covering beta-header injection, the cost-calculator delegation, provider-info secret resolution, and validate_environment key handling * fix(tencent): normalize messages URL when TENCENT_API_BASE has chat completions suffix * fix(tencent): register tencent in models_by_provider The provider was added to the LlmProviders enum and cost map but not to the models_by_provider lookup, so test_models_by_provider (which asserts every litellm_provider present in the cost map is registered) failed once the tencent models were loaded. Add the tencent_models set, populate it from the cost map, and expose it under the tencent key, mirroring deepseek. * fix(tencent): import generic_cost_per_token from its canonical module Import generic_cost_per_token from litellm.litellm_core_utils.llm_cost_calc.utils instead of the top-level litellm.cost_calculator dispatcher, which imports the tencent cost module at load time. Removing the back-reference avoids the circular import and matches how deepseek and the other providers source the helper. --------- Co-authored-by: Felipe Rodrigues Gare Carnielli <felipe.gare@hotmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
Shivam Rawat
|
1543725916
|
fix(bedrock): honor ttl for tool_config cache injection points (#31929)
* fix(bedrock): honor ttl for tool_config cache injection points Pass cache_control_injection_points control.ttl through to Bedrock toolConfig cachePoint blocks, matching message/system cache behavior. Co-authored-by: Cursor <cursoragent@cursor.com> * refactor(bedrock): drive Claude 4.5+ ttl support from pricing JSON, not regex is_claude_4_5_on_bedrock hardcoded a model-name pattern list that needed a manual update for every new Claude release (it already silently missed Sonnet 5 and Fable 5). Replace it with a lookup against cache_creation_input_token_cost_above_1hr in model_prices_and_context_window.json, which AWS docs confirm tracks the same 1h-TTL-capable model set. Also fixes two bedrock Claude 3.5 Sonnet entries that incorrectly carried that pricing field (their own regional variants didn't have it), which would have made the JSON-driven check wrongly grant them 1h TTL support. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(tests): use real Claude Sonnet 4.5 release id in ttl cache-point tests test_add_cache_point_tool_block_passes_ttl_for_claude_4_5 and test_bedrock_tools_pt_passes_ttl_for_claude_4_5 used a fabricated model id (...-20250514-v1:0) that never shipped. This passed under the old regex-based is_claude_4_5_on_bedrock, which matched on substring alone, but fails now that it looks up cache_creation_input_token_cost_above_1hr in litellm.model_cost, since the fake id has no pricing entry. Also force the bundled local cost map in both tests so ttl eligibility reads this branch's pricing data instead of the network-fetched main copy, which lacks the fix until merge. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(bedrock): restore cache and tool config compatibility * fix(bedrock): preserve Sonnet 5 parallel tool config * fix(bedrock): decouple parallel tool support from cache ttl * refactor(bedrock): drive parallel tool use config from JSON, not hardcoded patterns Replace the hardcoded _CLAUDE_BEDROCK_PARALLEL_TOOL_USE_PATTERNS tuple and bedrock_converse_supports_strict_tool_schemas (dead code) with a supports_parallel_tool_use_config key in model_prices_and_context_window.json, matching how is_claude_4_5_on_bedrock already reads cache_creation_input_token_cost_above_1hr from the pricing JSON. New models pick up parallel tool use support automatically when their pricing entry ships with the key set, with no code change required Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): use real model id in parallel-tool-use-without-ttl-pricing test anthropic.claude-opus-4-7-unlisted-v1:0 has no entry in model_prices_and_context_window.json, so bedrock_converse_supports_parallel_tool_use_config returned False and the test died with KeyError on additionalModelRequestFields. Use jp.anthropic.claude-opus-4-7, a real entry that carries supports_parallel_tool_use_config without 1h-TTL cache pricing, which is exactly the decoupling this test exists to cover * test(utils): allow supports_parallel_tool_use_config in pricing schema The misc unit test job validates model_prices_and_context_window.json against the INTENDED_SCHEMA allowlist in test_utils.py, which rejects unknown keys. Add the supports_parallel_tool_use_config key this PR introduced so test_aaamodel_prices_and_context_window_json_is_valid passes again * fix(bedrock): preserve ttl for regional claude models * fix(bedrock): fall back to base model entry when regional pricing lacks capability fields Regional model_cost entries like jp.anthropic.claude-opus-4-7 that omit cache_creation_input_token_cost_above_1hr shadowed the base entry that has it, so is_claude_4_5_on_bedrock returned False and requested cache ttl values were dropped for those deployments. Both capability lookups now consult the full model id and the region-stripped base entry, matching the coverage of the old name-pattern list. Also restores ToolBlock keyword construction for the tool_config cachePoint; PEP 589 TypedDict keyword instantiation works on every supported Python version --------- Co-authored-by: Shivam Rawat <shivamrawat@Shivams-MacBook-Pro.local> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
85f924148a
|
fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8 (#31582) (#31923)
* fix(bedrock/converse): drop toolSpec.strict for Opus 4.7/4.8
Bedrock Converse routes Claude Opus 4.7/4.8 through an Anthropic-compatible
validator that maps toolSpec to the native tool shape and rejects the extra
`strict` key with `tools.N.custom.strict: Extra inputs are not permitted`,
even though Anthropic's native API accepts `strict` as a top-level tool field
for the same models. Sonnet 4.5/4.6 and Opus <=4.6 accept `toolSpec.strict`
unchanged.
The existing gate `get_bedrock_base_model(model).startswith("anthropic")`
(introduced in #29814 to forward `strict` for Claude on Bedrock Converse) is
too broad and regressed Opus 4.7/4.8 callers — see #31582.
Replace the inline check with a small `bedrock_converse_supports_strict_tools`
helper that excludes the Opus 4.7/4.8 family from strict forwarding. All
other Anthropic models on Bedrock keep the existing behavior.
Closes #31582.
* fix(bedrock/converse): move strict-tools regression to a clean test file
The original regression test was added to
test_litellm_core_utils_prompt_templates_factory.py, which has
pre-existing ruff-format violations throughout (multi-line asserts that
fit on one line). The lint workflow runs `ruff format --check` on
changed files only, so touching that file surfaces those pre-existing
violations and fails CI for unrelated reasons.
Move the #31582 regression coverage into a new dedicated test file so
the format check stays green. Also collapses the helper's `not any(...)`
onto a single line to satisfy ruff format.
Covers: #31582
* refactor(bedrock/converse): drive strict-tools gate from model cost map
Replace the hardcoded Opus 4.7/4.8 pattern list with a
bedrock_converse_supports_strict_tools flag on the affected entries in
model_prices_and_context_window.json, resolved via get_model_info with a
local cost map fallback, so future models with the same restriction only
need a JSON update
* chore: revert unrelated credential_migration.py reformat
---------
Co-authored-by: ly-wang19 <ly-wang19@users.noreply.github.com>
|
||
|
Mateo Wang
|
fde4c7c97a
|
feat(gdc): implement Google Distributed Cloud (GDC) Gemini provider (#31895)
* feat(gdc): add Google Distributed Cloud Gemini provider support Introduce support for the Google Distributed Cloud (GDC) Gemini provider by adding "gdc" to the list of chat providers and enabling the gdc/ model prefix. The implementation defines a new GDCGeminiConfig class which handles authentication via Google Distributed Cloud service account credentials, manages token generation, formats GDC Gemini request URLs, and transforms request structures accordingly The PreProcessNonDefaultParams class is also updated to exclude vertex parameters from filtering when the custom LLM provider is GDC, allowing vertex parameters to be passed properly during GDC initialization * fix: resolve issues identified in PR #30702 * fix(gdc): harden credentials, fix vertex param filtering, add tests The supports_vertex_params branch regressed vertex_ai and vertex_ai_beta: the `if custom_llm_provider in [...]: pass` was a no-op, so those providers fell through to the config lookup, found no supports_vertex_params, and had their vertex_ params stripped. The check is now a single _provider_supports_vertex_params helper that keeps vertex_ params for the vertex family and for any config that opts in, and only swallows the expected ValueError from an unknown provider string instead of a blanket except GDC project and location now resolve from the deployment's litellm_params and the litellm.vertex_project / litellm.vertex_location globals before falling back to request optional_params, matching how vertex_ai resolves them, so a proxy caller can no longer route a request to a project the deployment did not expose A request api_key is no longer treated as a filesystem path, so a caller can't make the host open a local service-account file; api_key must be a literal service-account JSON string or a bearer token The opt-in token cache is hardened: the lock and cache dict are created in __init__ instead of via a racy hasattr lazy-init, the token is read inside the lock, and the audience is stripped of a trailing slash once so the cached and non-cached paths agree Also declares gdc_api_base, switches the lazy-import entry to the relative path every other entry uses, adds the missing trailing comma in the provider config map, and drops the api_base fallback that only ran when api_key was None Adds unit tests covering the vertex-param filter, deployment-over-request precedence, the api_key file-path rejection, URL construction branches, environment validation, token caching, and the gdc completion dispatch; transformation.py is fully covered * fix(gdc): prefer GDC-specific config, honor vertex_ai aliases, harden URL and bool parsing * fix(gdc): mint the GDCH token audience from the host, not the full base When api_base embedded /v1/projects/... and the deployment set project/location, get_complete_url rebuilt the request URL from the host while validate_environment still derived the token audience from the full original api_base, so the bearer token could target a different audience than the URL actually called. The audience is now the scheme://host of api_base in every case, matching the host get_complete_url builds against * fix(gdc): restrict JSON api_key to GDCH service accounts Only accept a credential whose type is gdch_service_account before calling google.auth.load_credentials_from_dict, so a caller-supplied external_account/identity_pool/pluggable credential carrying arbitrary token or credential_source endpoints is rejected before any token refresh runs. GDC only ever uses GDCH service accounts, and non-GDCH credentials could not have completed auth anyway (with_gdch_audience is GDCH-only), so this narrows the credential-refresh surface without changing valid GDC behavior. * fix(gdc): validate project and location as plain identifiers vertex_project and vertex_location can come from request params and were interpolated as raw path text into the GDC request URL and the x-goog-user-project header. A caller-supplied value containing / ? # or .. could reshape the path and make the proxy send its GDC-authorized request to a different endpoint under the configured host. Validate both against a strict identifier pattern before building the URL or header and raise an auth error otherwise; GCP project ids and locations are plain identifiers so valid deployments are unaffected. * fix(gdc): bind x-goog-user-project quota header to the deployment The quota project header was resolved with request-level vertex_project taking effect, so with a preformed deployment api_base a caller could set vertex_project to a different project and have it sent under the proxy's GDC credential, misattributing quota or billing. Resolve the header project the same way the URL is resolved: a preformed api_base without a deployment override binds to the project embedded in the URL, otherwise deployment and global config win over request params. This keeps the URL and the quota header consistent. * fix(gdc): always rebind x-goog-user-project, stripping caller-forwarded values The quota project header was only set when absent, so with client header forwarding an authenticated caller could send their own x-goog-user-project (any casing) and have it ride on the proxy's GDC credential, bypassing the deployment-derived binding. Strip every casing of the header and always set it from _effective_project before the request is signed. * fix(gdc): make a preformed api_base authoritative for project routing get_litellm_params copies caller-supplied vertex_project and vertex_location into litellm_params via OPTIONAL_KWARGS_KEYS, so litellm_params cannot be treated as a deployment-only source. The previous _deployment_overrides_path inference let an authenticated caller flip a pinned preformed api_base such as /v1/projects/pinned/... to /v1/projects/attacker/..., driving requests to a caller-chosen project with the proxy's configured GDC credentials and quota header A preformed /v1/projects/ api_base is now authoritative; get_complete_url returns it unchanged and _effective_project binds the x-goog-user-project quota header to the project embedded in that URL, so a caller can no longer redirect a pinned deployment or move the quota header off it. The two tests that asserted the override behavior are now regression tests that fail if the rewrite is reintroduced * fix(gdc): make a preformed api_base self-sufficient in get_complete_url get_complete_url resolved and required a params-derived vertex_project before returning a preformed /v1/projects/ api_base, so a deployment that pins its project in the api_base path was forced to also pass vertex_project or hit 'project is required'. validate_environment already extracts the project from a preformed URL and needs no such param, so the two paths disagreed The preformed-URL early return now runs before project/location resolution, matching validate_environment: a preformed api_base is returned as-is with no redundant param, and non-preformed bases still require vertex_project and vertex_location as before. Adds a regression test that a preformed base with no project/location params returns the URL unchanged --------- Co-authored-by: Paige O'Connor <lostpaige@google.com> Co-authored-by: Tim Laubach <tlaubach@google.com> |
||
|
Mateo Wang
|
b76a858826
|
feat: declarative fallback generalizations for unknown models (#29718)
* feat: declarative fallback generalizations for unknown models Unknown or newly-released models previously degraded (missed cost lookups, wrong supports_* flags, broken provider routing) and were patched with one-off hardcoded regexes scattered across Python. This adds a single data-driven source of truth: a fallback_generalizations block in model_prices_and_context_window.json holding ordered, case-insensitive regex rules that map a model name to the metadata to apply when it has no exact entry. A new fallback_generalizations module owns the rules and a compiled-regex cache that is built once and invalidated on reload, so the O(n) scan runs only on a cache miss. get_llm_provider now routes an otherwise-unknown model via the first matching rule's litellm_provider, replacing the hardcoded _CLAUDE_PATTERN and _matches_claude_model_pattern. _get_model_info_helper falls back to a matching rule's model_info after the exact lookups miss, so get_model_info and the supports_* helpers resolve unknown models from the same rule. get_model_cost_map extracts the block out of the returned map, and the integrity check now counts real model entries (excluding reserved meta keys) so the new key cannot mask a genuinely shrunk upstream file. The top level of the file stays a flat map of models so existing litellm releases that fetch the live file keep working and keep receiving updates; the block ships in both the root file and the bundled backup. An anthropic-claude rule reproduces the old future-claude routing and additionally supplies capability flags and a context window https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * refactor(anthropic): derive adaptive-thinking from a version threshold; harden generalizations Replace the per-minor-version _is_claude_4_6_model / _is_claude_4_7_model substring matchers with a single _claude_version_at_least predicate that parses the Claude family version from the model name and compares against 4.6. This covers 4.8/4.9/5.x without a code change (the old matchers missed 4.8 entirely) while keeping an explicit supports_adaptive_thinking flag authoritative when present, so there is one source of truth. The two direct call sites in the chat transformation now route through _is_adaptive_thinking_model instead of the deleted matchers. Also address review feedback on the generalizations module: return a copy of the matched model_info so a future caller cannot mutate the compiled-rule cache, document that patterns are matched with re.search and must anchor with ^ and $, and reindent the fallback_generalizations block to the file's 2-space style in both JSON files. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): surface adaptive-thinking from the cost map; fix date misparse supports_adaptive_thinking shipped in the model cost map but was never declared on ModelInfo nor copied during construction, so get_model_info (and the supports_* factory) silently dropped it for every provider-prefixed or generalized name; only a bare base entry resolved. Wire it through ModelInfo like the other capability flags and backfill the flag onto the genuine Claude 4.6/4.7/4.8 entries across providers so the data, not code, declares the capability. The anthropic-claude fallback rule also carries the flag (and now accepts a dotted minor, e.g. 4.6) so an unmapped future Claude degrades to adaptive thinking without a code change. Tighten the Claude version parser so an eight-digit date suffix (claude-opus-4-20250514, the non-adaptive Opus 4.0) is no longer read as minor 4.20250514. The cost map stays authoritative; the version check is only a fallback for provider-prefixed names (bedrock/invoke routes, -v1-less ids) that resolve to no mapped entry and so cannot be reached by an exact lookup or the bare-name rule. https://claude.ai/code/session_01G8Jro8dPLktwnaaSJwVDpo * fix(anthropic): date-safe adaptive-thinking version fallback, conservative fallback pricing, ruff strict gate Reconcile adaptive-thinking detection after merging litellm_internal_staging. Keep the cost-map resolver (_supports_model_capability) as the source of truth and add a date-safe opus/sonnet/haiku >= 4.6 name version as a fallback for provider-prefixed ids the cost map cannot resolve (e.g. bedrock/invoke/us.anthropic.claude-opus-4-6). A two-digit cap on the minor keeps an eight-digit date suffix from being misread as a minor version, so the dated Claude 4.0 release stays non-adaptive Price the shipped anthropic-claude fallback rule at the Opus tier so an unknown or newly released Claude is over-costed rather than billed as free Drop the module-level global state in fallback_generalizations (PLW0603) in favor of a small registry object, and switch its annotations plus the new utils helper to builtin generics (UP006), bringing the ruff strict-rule totals back under ceiling * refactor(anthropic): drive adaptive-thinking version gate from a declarative rule Replace the bespoke _claude_version_at_least heuristic with a version-gated fallback_generalizations rule. Unmapped Claude ids now resolve adaptive thinking purely from the cost map: an explicit entry, or the new self-contained anthropic-claude-adaptive-thinking rule that matches opus/sonnet/haiku >= 4.6 (covering 5.x, 6.x and beyond with no code change). New families ship via Price Data Reload instead of a code edit The rule carries the same Opus-tier pricing as the broad anthropic-claude rule plus supports_adaptive_thinking, and is matched first; the broad rule stays version-neutral, so an unmapped >= 4.6 Claude resolves to full pricing and the adaptive flag from one rule, while a sub-4.6 alias such as claude-opus-4-0 is still priced yet stays non-adaptive. The regex caps the minor at two digits so a dated 4.0 id (...-4-20250514) is never read as a >= 4.6 minor * refactor(anthropic): dedupe adaptive-thinking rule via declarative extends The version-gated anthropic-claude-adaptive-thinking rule duplicated the broad anthropic-claude rule's entire Opus-tier price block because rules do not merge: first match wins and returns one rule's whole model_info, so the adaptive rule had to be self-contained. Add a declarative extends field to fallback_generalizations: a rule names a parent and inherits its model_info, with its own keys overriding. Inheritance is resolved once at install time against each rule's raw model_info, so the adaptive rule now carries only its delta (supports_adaptive_thinking) and inherits pricing from the broad rule. Runtime matching, provider routing and gating are unchanged; the broad rule stays anchored and first-match-wins still holds. * docs(anthropic): add ignored description key documenting each generalization regex * fix(anthropic): drop fabricated pricing from the anthropic-claude fallback rule Per review feedback, the base rule no longer carries input/output/cache costs, and the adaptive-thinking rule that extends it inherits that no-pricing model_info. Pricing an unmapped model at a guessed tier reports a confidently-wrong cost without the caller knowing; dropping it keeps the standard unpriced behavior (zero, not a fabricated number) so a missing price stays visible. The rules still supply provider routing, context window, and capability flags, so a brand-new Claude can still be called and its capabilities (including adaptive thinking for >= 4.6) resolved. Description and tests updated to match |
||
|
Mateo Wang
|
ef3dcf91a2
|
chore: remove unused keys from model cost map (#31528) | ||
|
Sameer Kankute
|
4476923ac4
|
test: add realtime proxy e2e suite across providers (#30960)
* tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * test: multi-window budgets coverage * fix: p0 issues, added types and shared functions for each test suite * chore: add config.yml * test: passthrough endpoints stream/non-stream e2e * style: carry clearer status_code comparison into renamed e2e dir * fix: rename cost breakdown function * fix: pydantic validation for budget info, dont allow explicit type cast * refactor: migrate to gateway client * test: add custom pricing tests * chore: change master key * test(e2e): address greptile review feedback Remove the duplicate cache/cache_params block in the gateway config so the two can't silently diverge under future edits. Reorder the soft-budget test to assert the call isn't a budget block before require_successful_call, since that helper hard-fails any non-2xx and left the budget-block check unreachable; the misleading "skip" comment is corrected. Add a deferred delete in test_budget_delete_removes_it so a failed delete doesn't leak a budget on the shared proxy. Scope the spend_tracking sys.path insertion in pytest_sessionfinish to just the cleanup import so a broader "pytest tests/" run isn't left with a mutated path. * test(e2e): drop misleading skip comment on require_successful_call require_successful_call fails hard, it does not skip; the trailing comment was factually wrong. The function name already states intent, so the comment is removed in both per-model and tag budget helpers. * test(e2e): assert budget-isolation invariant before success check On the should-still-succeed path of the per-model and tag isolation tests, check is_budget_block before require_successful_call. If the isolation bug fires the unaffected model/tag is blocked, so asserting the specific 'blocked by X' invariant first yields the diagnostic message instead of a generic upstream-failure. Matches the ordering in test_soft_budget_e2e.py. * fix(e2e): guard spend-log truncate on skip and stop returning unrelated priced rows * fix(e2e): run case init() inside try so partial-init failures tear down run_case called case.init() outside the try/finally that runs teardown(), so a case that registers cleanups progressively (create team, then user, then key) and then fails partway through init() would leak the already-created entities on the long-lived shared proxy. Move init() inside the try so teardown always runs. Add a regression test that registers a cleanup then raises mid-init and asserts the resource is still released. * test(e2e): mark known pricing-leak isolation test xfail(strict) test_custom_pricing_is_isolated_from_sibling_deployment documents a real proxy gap (a deployment's custom per-token pricing leaks into the shared cost map for sibling deployments of the same underlying model) and was left unconditionally failing, which pollutes the suite's pass/fail signal. Mark it xfail(strict=True) so the suite stays green while the leak persists and turns into a failure the moment isolation is fixed, prompting the marker's removal. * refactor(e2e): make suite pass its shipped strict basedpyright config The suite ships tests/pyrightconfig.json (strict, no Any), but basedpyright --project tests reported four errors in it: three reportAny on the parametrize ids=lambda c: c.__name__, and one reportUnusedFunction on the underscore-prefixed autouse fixture _require_live_proxy. Replace the untyped lambda with a typed _case_id(case_cls: Type[_BudgetCase]) -> str so the ids are no longer Any, and rename the fixture to require_live_proxy so basedpyright no longer treats it as an unused private function (it is referenced only by pytest's autouse machinery). basedpyright --project tests now reports zero errors. * fix(tests/e2e): gate spend-log truncate on e2e marker, not test directory * test(e2e): run harness unit tests without a live proxy The autouse session fixture skipped the whole tests/e2e session when no proxy answered, which also skipped test_lifecycle.py, a pure unit test of run_case that never touches the proxy. A regression test that silently skips gives no signal, so the skip now lives in pytest_runtest_setup gated on the same e2e marker the spend-log truncate guard already uses: live tests skip when no proxy is up while harness unit coverage always runs. The liveness probe is cached with lru_cache so it still runs once per session * test(e2e): clean up gateway config comment debris Fix the typo on the header comment and drop the orphaned namespace/ttl comment remnants left indented under cache_params; the active values are already set above. Flagged by greptile review. * fix: add new tests, split gateway * test(e2e): type the redis spend-counter probe for strict basedpyright The new cold-counter reseed test drove its redis client untyped, so the strict tests/pyrightconfig.json (reportUnknown*, reportAny) flagged ten errors once the file landed: scan_iter/get came back unknown and the pool.map lambda had an untyped parameter. Annotate the client as redis.Redis[str] via a TYPE_CHECKING import (the runtime import stays lazy so the suite still skips, not errors, when redis is absent), which resolves scan_iter to Iterator[str] and get to str | None, and replace the lambda with a typed inner function mirroring _burst. basedpyright --project tests is back to zero errors. * test(e2e): xfail the known team multi-window failure and isolate member teardown Greptile flagged two issues in the mirrored split-gateway commit. The team multi-window budget test documents a real /team/new write bug (budget_limits go straight to the Json? column and Prisma 500s, unlike the json.dumps'd key and /team/update paths) and was left as an unconditional hard failure, which would turn any live-proxy CI run red; mark it xfail(strict=True) like the custom-pricing isolation test so the suite stays green while the bug persists and flips to a failure the moment the write is fixed and the marker should go. The class-scoped member fixture in test_team_member_budget_e2e.py tore down its key, user, and team sequentially with no exception isolation, so a failed delete_key would strand the user and team on the long-lived shared proxy. Route cleanup through a ResourceManager: register each delete progressively and run them LIFO best-effort in a finally, so a partial-setup failure still releases what came before and one failed delete never blocks the rest. * test: add realtime proxy e2e suite across providers Add tests/realtime_e2e covering the proxy realtime websocket endpoint end to end against live providers (openai, azure, gemini, vertex_ai, bedrock, xai). Two layers: a raw-websocket suite asserting the normalized OpenAI GA event sequence, delta/transcript consistency, usage, and a full tool-call round-trip; and a pipecat smoke driving the proxy through the GA OpenAIRealtimeLLMService. Tests carry a new realtime_e2e marker and skip cleanly when the proxy or provider creds are absent, so they stay out of the default unit run. * test: move realtime e2e suite into tests/e2e harness Replace the standalone tests/realtime_e2e with a tests/e2e/realtime suite that follows the existing e2e conventions: a session-scoped client fixture, a frozen-dataclass RealtimeClient wrapping the shared Gateway, pydantic models for every sent and received event, and the e2e marker with the parent harness's liveness skip. The suite opens the proxy realtime websocket (websockets.sync to stay synchronous like the rest of the harness) and asserts the normalized OpenAI GA event sequence for a text conversation plus a full tool-call round-trip, parametrized across providers. A provider whose realtime alias is not configured on the proxy skips via /model/info. Adds a gemini realtime model to the gateway config and fixes the openai realtime model id. * test: add pipecat realism layer to realtime e2e suite Add test_realtime_pipecat_e2e driving the same providers through pipecat's GA OpenAIRealtimeLLMService with base_url pointed at the proxy, as a coarse realism check on top of the raw-websocket suite. Each test stays synchronous and runs the async pipecat pipeline via asyncio.run, and the module skips unless pipecat-ai is installed. Lift the shared provider matrix, ws-url helper, and skip helper into realtime_client so both suites use them. * fix(e2e): parse GA realtime transcript events in e2e client The realtime e2e client speaks the GA protocol, but transcript() only aggregated beta delta event names. Handle GA deltas, fall back to response.done output, and accept nested usage details on response.done. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): address realtime code-review findings - Use the real openai/gpt-4o-realtime-preview model ID in the gateway config (gpt-realtime-2 does not exist and would fail every live test) - Pass a bare base_url to pipecat's OpenAIRealtimeLLMService so pipecat can append ?model= itself; the previous realtime_ws_url already contained ?model= causing a malformed duplicated query parameter - Wrap connection.recv() in a try/except TimeoutError in collect_until so a deadline expiry inside recv preserves the collected-events diagnostic instead of raising a bare, message-free exception Co-authored-by: Cursor <cursoragent@cursor.com> * fix(e2e): filter configured_models to mode:realtime entries only ModelInfoEntry.model_info used CustomPricing (extra="ignore") so the mode field from /model/info was silently dropped, making it impossible to distinguish realtime from non-realtime deployments. Add an optional mode field to CustomPricing and filter configured_models() to entries whose model_info.mode == "realtime" so skip_if_unconfigured never accidentally skips a realtime test due to a naming-pattern collision with a non-realtime deployment. Co-authored-by: Cursor <cursoragent@cursor.com> * Update litellm-config.yml * fix(e2e): use TypeVar instead of PEP 695 generic in realtime parse_last PEP 695 type-parameter syntax (def f[T: Bound](...)) is only parseable on Python 3.12+, but the project declares requires-python >=3.10. Importing the realtime e2e client on 3.10/3.11 raised a SyntaxError before any test could run. Switch parse_last to the backport-safe TypeVar idiom so the suite imports across the full supported range. * fix(e2e/realtime): use GA openai/gpt-realtime model id The realtime gateway config used openai/gpt-realtime-2, which is not a real OpenAI model id and would 404 once live OpenAI realtime credentials are wired in. The GA speech-to-speech model is openai/gpt-realtime (snapshot gpt-realtime-2025-08-28); switch the openai-realtime alias to it. * fix(realtime): harden Gemini/Vertex Live for audio-native e2e Coerce TEXT responseModalities to AUDIO on native-audio and flash-live models, suppress the orphan turnComplete response.done that arrives immediately after tool results, omit function_response.id on Vertex, stop appending client query params to Gemini/Vertex WSS URLs, and add regression tests for these paths. Co-authored-by: Cursor <cursoragent@cursor.com> * Add xai full compatibility * Add working vertex ai realtime tests * Add audio + server vad e2e tests * Add config for e2e testing models * Add fix xai server vad * fix: use correct OpenAI realtime model ID in e2e gateway config openai/gpt-realtime is not a valid model; replace with the correct openai/gpt-4o-realtime-preview model ID to prevent model-not-found errors when running the openai-realtime e2e tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore openai/gpt-realtime model ID gpt-realtime is a valid model; reverting the unnecessary change. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: resolve UP006 violations, mock test failures, and stale spec field - Guard gemini setup-without-tools deferral with litellm.gemini_live_defer_setup flag so the default (False) path sends setup immediately, fixing two failing mock tests: test_client_ack_caches_setup_to_prevent_duplicate_session_update_setup and test_deferred_setup_sends_session_update_before_buffered_audio - Replace deprecated typing generics (Dict, List, Tuple, Optional) with builtin equivalents in xai/realtime/transformation.py, gemini/realtime/transformation.py, and realtime_streaming.py to satisfy the UP006 ruff-strict ceiling - Remove 'role' from OpenAPI compliance test expected fields; Google removed it from the Interaction schema in their live spec Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use Optional[dict] in xai normalizer to preserve Black line-split dict[str, Any] | None is shorter than Optional[Dict[str, Any]] by enough that Black collapses the _normalize_usage signature to a single line (86 chars), conflicting with the existing multiline format. Using Optional[dict[str, Any]] keeps the line at 90 chars (> 88 limit) so Black preserves the multiline shape, while still satisfying UP006 by replacing Dict with dict. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: remove proxy-level setup-tools deferral, delegate to transformer The _gemini_setup_deferred / _gemini_pre_setup_buffer block in _send_to_backend was double-deferring: GeminiRealtimeConfig already handles the session.update-to-setup mapping internally and always returns a ready-to-send setup on the first session.update call (session_configuration_request=None). The proxy layer was incorrectly holding back that setup waiting for tools that the transformer had already incorporated. Removing the block fixes two failing tests: test_client_ack_caches_setup_to_prevent_duplicate_session_update_setup test_deferred_setup_sends_session_update_before_buffered_audio Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: abstract Gemini protocol keys out of core and use cost map for live model detection Move Gemini-specific message key knowledge (setup, realtimeInput, clientContent, toolResponse) out of the core RealTimeStreaming module into provider-level methods. BaseRealtimeConfig gains is_setup_message and is_content_message (both default False); GeminiRealtimeConfig overrides them with the actual Gemini key checks. Add gemini_native_audio and gemini_audio_only_live capability flags to the 10 affected model entries in the cost map. _is_audio_only_live_model and _is_native_audio_model now read from the cost map first and fall back to the existing string markers for models not in the map. * fix: apply black formatting and register gemini capability fields in schema * refactor: drop string-marker fallback; resolve audio-only live models via cost map only * fix: use registered cost-map model name in vertex realtime tests * fix: patch cost map in tests so they don't depend on remote main branch state * fix: align gateway config vertex-realtime model ID with cost-map registered name * fix: patch gemini-2.5-flash-native-audio in cost map fixture for CI * fix(e2e): use correct OpenAI realtime model id in gateway config Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e): add budget rescheduler short intervals to gateway config Without proxy_budget_rescheduler_min/max_time set, the rescheduler defaults to ~600s, causing all budget-reset e2e tests to timeout before the reset fires. Set to 5–10s so tests complete within 90s. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * chore(e2e): strip non-realtime files from PR scope Restore budget, spend-tracking, and custom-pricing test files to their litellm_internal_staging state. Keep the mode field addition to CustomPricing in models.py (needed by realtime configured_models filter). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(tests): restore async_realtime regression test and add missing fixture - Restore the end-to-end async_realtime regression test for Vertex query-param forwarding; the previous unit-only version did not exercise the code path where the original bug lived - Add patch_gemini_audio_cost_map_entries fixture to test_gemini_audio_only_live_models_drop_text_from_text_audio_combo so it does not depend on the cost map having gemini_audio_only_live set in CI Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): resolve ANN401 violations in realtime streaming code Define RealtimeEventNormalizer Protocol and replace bare Any annotations with typed alternatives (object for event/value params, the Protocol for the normalizer) to stay within the strict-rule budget. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: black format realtime_streaming.py * fix(tests): add gemini_native_audio and gemini_audio_only_live to model prices schema * fix(lint): fix I001 import sort order in realtime_streaming.py * fix(lint): restore import litellm to correct position before from-litellm imports * undo budget removal * test(e2e): pin explicit credentials for gemini and vertex realtime models * test(e2e): share keepalive-safe LiteLLMRealtimeLLMService across pipecat suites The pipecat smoke test drove the proxy through the stock OpenAIRealtimeLLMService, which sends websocket keepalive pings at its default interval. The proxy does not answer them, so the connection is closed with a 1011 before the run completes. Move the proxy-aware LiteLLMRealtimeLLMService (keepalive disabled) into a shared pipecat_service module and use it from both the smoke and audio suites. * test(e2e): document that LiteLLMRealtimeLLMService._connect keeps the ?model= param The proxy routes realtime websockets on the ?model= query param, and pipecat's OpenAIRealtimeLLMService.__init__ bakes it into self.base_url before _connect runs. Passing self.base_url through preserves it; spell that out so the override is not misread as dropping the param. * fix(realtime): set _content_sent_after_setup only after the backend send succeeds A failed content send used to flip _content_sent_after_setup to True before the send was confirmed, mirroring the correct-on-failure ordering the adjacent session-config cache already follows. If the send raised, the flag stayed True and a later session.update that produced a setup frame was silently dropped even though the backend never received any content. Set the flag after the send succeeds and add a regression test that fails if the ordering is reverted. * fix: normalize realtime passthrough events * refactor(realtime): declare patch_outgoing_session on normalizer Protocol; fix wav chunk return type The RealtimeEventNormalizer Protocol only declared should_drop and normalize, so the outgoing session.update patch went through a getattr(..., None) lookup even though should_drop/normalize are called directly. The sole implementer (XAIRealtimeNormalizer) already provides patch_outgoing_session, so declare it on the Protocol and call it directly for consistent, fully-typed dispatch. Also correct _load_wav_chunks' return annotation from list[bytes] to tuple[list[bytes], int]; it returns (chunks, sample_rate) and the caller unpacks both. --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Krrish Dholakia
|
d0706c17fe
|
fix(anthropic): drop unsupported speed param with drop_params (#31152)
* fix(anthropic): drop unsupported speed param with drop_params Anthropic fast mode (speed) is Opus 4.6/4.7/4.8 on the direct API only. Strip speed when the model map lacks supports_speed and drop_params is set, for both chat completions and /v1/messages passthrough. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(ci): allow supports_speed in model map schema The new supports_speed flag on Opus entries must pass JSON schema validation in test_aaamodel_prices_and_context_window_json_is_valid. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(review): raise on unsupported speed without drop_params Passthrough /v1/messages now raises UnsupportedParamsError when speed is unsupported and drop_params is false. Emit drop warning from map_openai_params when speed is silently skipped. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): gate speed param by routed provider, not just model id Vertex, Azure, and Bedrock reuse the shared Anthropic transform and strip their provider prefix first, so a bare `claude-opus-4-8` resolved to the direct-API model-map entry (`supports_speed: true`) and forwarded `speed` upstream, producing the same 400 that drop_params is meant to prevent. Gate fast mode on `custom_llm_provider == "anthropic"` so it stays on the direct Anthropic API across both the chat completions and `/v1/messages` passthrough paths, and collapse the duplicated drop/raise logic in map_openai_params into the shared `_maybe_drop_speed_param` helper. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Mateo Wang
|
9f97111edd
|
feat(fireworks_ai): sync chat completions endpoint with full API surface (#30885)
* feat(fireworks_ai): sync chat completions endpoint with full API surface Add 23 missing request parameters to get_supported_openai_params(): seed, top_logprobs, min_p, typical_p, repetition_penalty, mirostat_target, mirostat_lr, logit_bias, echo, echo_last, ignore_eos, prompt_cache_key, prompt_cache_isolation_key, raw_output, perf_metrics_in_response, return_token_ids, safe_tokenization, service_tier, metadata, speculation, prediction, stream_options, sampling_mask. Also add reasoning_history gated on supports_reasoning. Fix prompt_truncate_length to prompt_truncate_len to match the actual API parameter name. The old name was never in DEFAULT_CHAT_COMPLETION_PARAM_VALUES, so it always went to extra_body and was rejected by Fireworks; it never actually worked. Normalize reasoning_effort boolean values to strings: True becomes "medium", False becomes "none". The Fireworks OpenAPI schema documents these as accepted types, but the server rejects non-string values with HTTP 400 in practice. Integers pass through as-is since the server is expected to validate them. Auto-inject stream_options.include_usage=true when stream=true and the user has not explicitly set stream_options. Without this, Fireworks returns null usage in all streaming chunks, which is inconsistent with the non-streaming behavior where usage is always present. If the user explicitly sets include_usage=false, it is preserved. Capture Fireworks-specific response fields in transform_response(): perf_metrics, prompt_token_ids, raw_output, and token_ids are now extracted from the response and stored in response._hidden_params (fireworks_perf_metrics, fireworks_prompt_token_ids, fireworks_raw_outputs, fireworks_token_ids) so they are accessible to logging, the proxy, and downstream consumers when the corresponding request parameters are enabled. Remove deprecated document inlining logic. Document inlining was deprecated on 2025-06-30 (https://docs.fireworks.ai/updates/changelog#-document-inlining-deprecation). This removes _add_transform_inline_image_block(), the file-to-image_url migration in _transform_messages_helper(), and the disable_add_transform_inline_image_block lookup. Current models that support image input do so natively as VLMs. cache_control, provider_specific_fields, and thinking_blocks stripping is retained. Update get_provider_info() to look up supports_vision and supports_pdf_input from the model cost map instead of hardcoding both to True (which was based on the now-deprecated document inlining). supports_prompt_caching remains True. API docs: https://docs.fireworks.ai/api-reference/post-chatcompletions Reasoning guide: https://docs.fireworks.ai/guides/reasoning Prompt caching: https://docs.fireworks.ai/guides/prompt-caching * fix fireworks chat api surface gaps * Scope Fireworks thinking param to reasoning models * style: fix black formatting * fix(test): update minimax-m3 expected_vision to True * test: cover non-dict content branch in transform_messages_helper * fix(fireworks_ai): remove metadata from supported params to prevent internal metadata disclosure * test(fireworks_ai): replace stale document-inlining capability test The CircleCI-only litellm_utils_tests suite still asserted the old behavior where document inlining made every Fireworks model report supports_pdf_input and supports_vision as True. That premise was removed in this change, so the test now reflects cost-map-driven capabilities: unmapped models no longer advertise vision/PDF support while mapped VLMs like minimax-m3 still do. * test(fireworks_ai): add end-to-end regression for native OpenAI params The existing coverage for the newly supported OpenAI-native params asserted list membership in get_supported_openai_params or called map_openai_params with a hand-built dict, both of which bypass the get_optional_params gate (DEFAULT_CHAT_COMPLETION_PARAM_VALUES). That gate is what previously raised UnsupportedParamsError for seed, top_logprobs, logit_bias, prompt_cache_key, service_tier and prediction when drop_params=False. Assert the full path so a revert of the supported-params additions fails the test instead of passing a shallow membership check. * test(fireworks_ai): fix test isolation in vision/inlining tests Use monkeypatch in test_fireworks_ai_vision_capability_from_cost_map so the LITELLM_LOCAL_MODEL_COST_MAP env var and litellm.model_cost are restored after the test instead of leaking global state into the rest of the process. Switch the document-inlining integration tests off deepseek-v3p1, whose supports_vision is null in the cost map, onto minimax-m3 which is explicitly supports_vision:true. The pass-through assertions no longer depend on a model incidentally not being marked non-vision. * fix(fireworks_ai): gate image rejection on exact vision capability The image_url rejection read supports_vision via _get_model_cost_capability, which falls back to hyphen-boundary substring matching when no exact cost-map entry exists. A custom or fine-tuned model id that merely contains a known non-vision model's short name (e.g. an id ending in -glm-5p2) inherited that entry's supports_vision:false and hard-failed valid image_url blocks on a vision-capable deployment. Split the exact candidate-key lookup into _get_model_cost_capability_exact and use it for the hard rejection so a fuzzy match can never block images; the substring fallback stays a soft signal for capability reporting. Also rewrites the fallback as a comprehension + max instead of an accumulating loop. * feat(fireworks_ai): surface response fields on streaming responses The Fireworks-specific response fields (perf_metrics, prompt_token_ids, per-choice raw_output and token_ids) were only captured into _hidden_params in transform_response, which runs for non-streaming completions; streaming chat went through the default OpenAI chunk handler and dropped them. Add a FireworksAIChatCompletionStreamingHandler that the provider now returns from get_model_response_iterator. It reuses one extraction helper with transform_response and attaches the fields to each streamed chunk's provider_specific_fields, which is the channel litellm preserves when it rebuilds streamed chunks (per-chunk _hidden_params is not carried through). Per-choice token_ids/raw_output ride the content chunks; response-level perf_metrics/prompt_token_ids ride the final usage chunk. Covered by an end-to-end streaming test through litellm.completion(stream=True). --------- Co-authored-by: Ahmad Shahzad <ahmad@shahzad.dev> Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com> |
||
|
Sameer Kankute
|
e33e2917c6
|
chore: litellm oss 170626 (#30637)
* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (#30089) * fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets, /realtime/calls and their /v1 + /openai/v1 variants) to LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as LLM API routes. Without this, non-admin virtual keys received 401 'Only proxy admin can be used to generate, delete, update info for new keys/users/teams' when calling these endpoints. Fixes #29923 * fix(proxy): validate session.model for realtime routes in model-access check The GA Realtime WebRTC HTTP routes resolve the effective model from the nested session.model (falling back to the top-level model), but the auth layer's get_model_from_request() only extracted the top-level model. A model-restricted virtual key could therefore place a disallowed model in session.model, leave the top-level model unset, and skip can_key_call_model() entirely - obtaining an ephemeral token for a model it is not allowed to use. Extract session.model for the realtime client_secrets/calls routes so the model-access check runs against the model the request will actually use. Legitimate callers are unaffected; their permitted model still validates. Relates to https://github.com/BerriAI/litellm/issues/29923 * fix(proxy): classify realtime transcription_sessions routes as LLM API routes Add the GA Realtime WebRTC transcription_sessions HTTP routes to openai_routes so is_llm_api_route() returns True for them, matching the client_secrets and calls routes already fixed. These endpoints are registered with user_api_key_auth in realtime_endpoints/endpoints.py, so without this a non-admin virtual key calling POST /v1/realtime/transcription_sessions would hit the admin-only 401 branch. Extends the regression test parametrization accordingly. --------- Co-authored-by: habonlaci <4699494+habonlaci@users.noreply.github.com> * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (#30272) * feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models * fix(proxy): degrade /v1/models gracefully when model-group lookup fails --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: sort tiered token-cost thresholds numerically (#30375) * fix: sort tiered token-cost thresholds numerically _get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a lexicographic sort, so for tiers whose thresholds have different digit lengths (e.g. 90k vs 128k) a request crossing both was billed at the lower tier that sorted first. Sort by the parsed numeric threshold instead, so the highest tier the request actually crosses is applied. * refactor: reuse _parse_above_token_threshold for inline threshold parse --------- Co-authored-by: Eric (GabiDevFamily) <271972409+santino18727-debug@users.noreply.github.com> * fix(openai): preserve cache_control for openai-compatible custom endpoints (#30387) * fix(openai): preserve cache_control for openai-compatible custom endpoints * fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation * fix(proxy): drain all daily-spend batches per flush cycle (#30281) (#30505) * fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (#30545) * fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers * test(types): add regression test for internal rate-limit fields in all_litellm_params * fix(init): add bool type annotation to suppress_debug_info (#30531) Module-level `suppress_debug_info = False` had no annotation, so strict type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to `True` (as done in proxy_server.py and router.py) then fails with an invalid-assignment error. Annotate it as `bool` to match every other flag in this module. * fix: coalesce null aggregates in update_metrics for no-spend keys (#29945) * feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (#30006) * feat(team_endpoints): Add query parameter key_limit to /team/info * feat(team_endpoints): update schema.d.ts to include the new query parameter * feat(team_endpoints): add tests for limitting key count in /team/info response * feat(team_endpoints): Apply suggestions from greptile * Set greater-than constraint on key-limit * Fix type * fix(router): release aiohttp connection when stream iteration ends abnormally (#30271) * fix(router): release aiohttp connection when stream iteration ends abnormally A streaming response that terminates with a mid-stream read timeout, a task cancellation (client disconnect), or GeneratorExit never closed the underlying aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body EOF, so each abnormally terminated stream permanently leaked one slot from the shared TCPConnector pool. During a backend traffic spike the pool drains; once exhausted every subsequent request to that host waits for a slot, times out and surfaces as a 408, indefinitely, even after the backend recovers. Only a proxy restart cleared the in-memory sessions, which matched the reported symptom of a router stuck returning 408 for a healthy vLLM backend. Close the response in a finally clause when iteration ends. On a fully read response the connection was already released at EOF and close() is a no-op, so keep-alive reuse for normal requests is unchanged. Fixes #30192 * test(aiohttp): cover GeneratorExit path with a mock instead of a live socket The previous slot-release test started a real aiohttp TCP server, which can flake in offline CI and does not exercise this fix's code path directly. Replace it with a dependency-injected mock that closes the stream generator (GeneratorExit) and asserts the response is closed, covering the third abnormal-exit path the finally block handles * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273) * feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery * refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils * fix(proxy): make model_list request param optional for direct callers * feat(dashscope): add Responses API support (#30286) * feat(dashscope): add Responses API support DashScope's OpenAI-compatible endpoint serves /responses, so register a DashScopeResponsesAPIConfig that routes dashscope/* responses calls to {api_base}/responses without rewriting the upstream model id, instead of falling back to the chat-completions -> responses emulation pipeline. Closes #29780 * feat(dashscope): mark responses API as not supporting native websocket Matches the hosted_vllm/perplexity/openrouter responses configs, which all override supports_native_websocket() to False since the OpenAI-compatible endpoint has no native wss:// responses transport. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): preserve error_message on ProxyException failures (#30381) * fix(spend-logs): preserve error_message on ProxyException failures `StandardLoggingPayloadSetup.get_error_information` used `str(original_exception)` to populate the human-readable error message stored in `spend_logs.metadata.error_information.error_message`. `ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in its constructor but does NOT call `super().__init__(message)` and does NOT define `__str__`. As a result, `str(ProxyException(...))` returns the empty string, and every auth/budget/quota rejection was landing in spend_logs with `error_message=""` despite a fully populated traceback. Operator impact: dashboard "LLM Failure" rows became untriageable — the only way to tell a 401 from a 429 was to manually unpack the traceback JSON via psql. Burst failure patterns (e.g. a UI session polling with a stale token) produced 20-30 indistinguishable `error_code=401` rows per second. Fix: prefer the `.message` attribute (set by ProxyException and every litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback is retained for non-litellm exception types, preserving prior behavior. Test plan: - 2 new unit tests in tests/test_litellm/litellm_core_utils/ test_litellm_logging.py: * test_get_error_information_prefers_message_attribute_over_str * test_get_error_information_falls_back_to_str_when_no_message_attr - Existing test_get_error_information_error_code_priority still passes - End-to-end verified: bad-key 401 now stores full "Authentication Error, Invalid proxy server token passed..." message in spend_logs.metadata.error_information.error_message * fix(spend-logs): preserve explicit empty .message + drop dead reference Greptile P2 on #30381. The truthiness check `if message_attr:` silently skipped an explicit empty-string `.message` and fell through to `str(original_exception)`. For ProxyException-shaped objects both produce empty, so the bug was latent; for other exception types it would inject a different string into error_information.error_message and corrupt the signal. Use `is not None` so an empty string survives verbatim. Also drop the stale `See e2e/cases/11.` comment reference — that path does not exist anywhere in the repo and confuses future readers. Regression test added: an exception with `.message=""` and a non-empty `super().__init__()` arg must yield error_message == "". * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (#30382) * fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response The non-streaming /v1/messages response carries a LiteLLM-injected usage.total_tokens = input_tokens + output_tokens that is not part of the Anthropic API spec. This caused three problems: 1. Shape divergence with streaming on the same endpoint. message_delta.usage in the SSE path never carries total_tokens. Clients parsing both paths get two different schemas from one endpoint. 2. Shape divergence with upstream. Direct calls to https://api.anthropic.com/v1/messages return no total_tokens field, so clients using the official Anthropic SDK couldn't rely on it, and clients that did rely on the LiteLLM-injected one broke when bypassing the proxy. 3. Numerical misuse. total = input + output undercounts when cache_read_input_tokens and cache_creation_input_tokens are non-zero, because cache tokens are reported in their own fields. A 100k-token cached prompt with 1 non-cache input token + 200 output tokens reports total_tokens = 201, off by ~99.8% from any reasonable definition of "total." Fix: add _strip_total_tokens_from_anthropic_response in litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the success path of anthropic_response right before returning. Only mutates dict-shaped responses; streaming (which already lacks the field) is left untouched. spend_logs / Prometheus continue to compute total_tokens internally for billing — this fix only strips the field from the wire response. Scope: only the Anthropic passthrough endpoint /v1/messages. The OpenAI-shape /v1/chat/completions is unaffected. * fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage Two P1 greptile threads on #30382: P1 — **Backwards-incompatible removal without a feature flag** Stripping `usage.total_tokens` unconditionally breaks any client currently reading the LiteLLM-shaped non-streaming /v1/messages response. Per the codebase's policy (mirrors #30418), gate behind a new flag. - `litellm.strip_anthropic_total_tokens: bool = False` (default — backward-compat: clients keep seeing total_tokens). - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`. - Docstring: planned to flip to True in a future major release; opt in early. P1 — **Silent no-op if `result` is a Pydantic model** `base_process_llm_request` may return a Pydantic-style object whose `.usage` is a plain dict (the most common shape — e.g. objects wrapping raw upstream JSON). The original `isinstance(response, dict)` guard skipped strip on those, so `total_tokens` would still hit the wire. Helper now also reads `getattr(response, "usage", None)` and strips when that's a dict. Strongly-typed Pydantic `Usage` sub-models with required `total_tokens` fields are still skipped — those impose type constraints the helper doesn't try to subvert. Tests: - `test_strips_total_tokens_on_pydantic_model_with_dict_usage` - `test_flag_defaults_off` 8/8 pass locally. * fix(anthropic): drop env var for strip flag (docs CI) Mirrors #30418's pattern (`expose_router_debug_in_errors: bool = True`, no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var introduced in the prior commit was flagged by `tests/documentation_tests/test_env_keys.py` because the documentation file `docs/my-website/docs/proxy/config_settings.md` lives in `BerriAI/litellm-docs` (separate repo) and registering a new env key requires a parallel docs PR — a friction we avoid here by exposing the flag only as a Python attribute + `litellm_settings` config key, both of which load through the existing proxy config plumbing without needing the env-var registry to be updated. No semantic change: default still False, behavior identical when set via `litellm.strip_anthropic_total_tokens = True` or `litellm_settings.strip_anthropic_total_tokens: true` in config.yaml. Verified locally: env scan no longer surfaces the key; 8/8 tests pass. * ci: retrigger workflows after base branch change to litellm_internal_staging * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (#30413) * fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 * test: resolve model prices JSON relative to test file for pip installs * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (#30417) * fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit signal from upstream inside an HTTP 500/503 envelope, with the real code only surfaced in the JSON body: {"error":{"message":"...high demand...","type":"upstream_error", "param":"","code":429}} Previously LiteLLM only looked at the HTTP status and mapped this to InternalServerError, which Router treats as non-retryable for many configs — so users got hard 500s instead of fallback/retry. Now the Gemini/Vertex exception mapper parses error.code from the body and routes code 429 to RateLimitError before falling through to the HTTP-status branches. Other body codes fall through unchanged. Tests cover: - new-api gateway's `code:429` payload now maps to RateLimitError - Genuine 500-body responses stay InternalServerError - Non-JSON body strings fall through to status-code mapping unchanged * fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes Addresses greptile P1/P2 + @Sameerlite's review on #30417. The new elif branch was firing for any HTTP status, so a gateway response of HTTP 400 with body {"error":{"code":429,...}} would be incorrectly promoted to RateLimitError (retryable) instead of falling through to BadRequestError. Same trap for 401 -> AuthenticationError. Scoped the body-code 429 check to `500 <= status_code < 600` — covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx envelope) without inviting the 4xx misclassification. Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401), and the existing fall-through cases, asserting each maps to the exception type that matches the HTTP status code. 50/50 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (#30418) * feat(router)!: redact internal model_group/fallback names from exception messages The Router was unconditionally appending internal config names onto exception.message: - "Received Model Group=..." - "Available Model Group Fallbacks=..." - "No fallback model group found... Fallbacks={...}" - "context_window_fallbacks={...}" - Deployment-timeout messages including model_group - Fallback failure detail listing fallback chain ProxyException forwards .message verbatim to clients, so gateways were leaking their model_name / fallback wiring in every failed call. Fix: gate all five mutation sites on a new `litellm.expose_router_debug_in_errors` flag (default False). Set to True to restore upstream debug behavior for local debugging. Why: matches the redaction posture this codebase already has for upstream model identifiers (cf. _litellm_returned_model_name) and removes the last common error-path leak of internal model_group names. Breaking change marker (!): if anything parses "Received Model Group=" out of client error messages, flip the flag on or migrate to the x-litellm-* response headers instead. Tests: 7 cases covering each of the 5 redaction sites + the flag-on inverse path, plus a "default off" sanity check. * test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate Addresses Greptile / codecov feedback on #30418: patch coverage was 55.6% with 4 lines uncovered in litellm/router.py. The existing tests exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found), and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3 were declared in the PR description as covered by "site 5 also fires" but the gate body lines for each (the `e.message +=` inside the `if litellm.expose_router_debug_in_errors:` branch) only execute when the flag is on AND the specific exception path is taken, which neither existing test triggered. Added 4 new tests (default + flag-on × 2 sites): - test_default_does_not_leak_deployment_timeout_debug - test_flag_on_leaks_deployment_timeout_debug - test_default_does_not_leak_content_policy_fallback_hint - test_flag_on_leaks_content_policy_fallback_hint Trigger details: - Site 1 (litellm.Timeout in _acompletion) is reached via the Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on `acompletion(...)`. Cannot embed a Timeout instance in model_list because Router.__init__ deep-copies it and Timeout.__reduce__ does not preserve the required positional args. - Site 3 (ContentPolicyViolationError without content_policy_fallbacks set, in async_function_with_fallbacks_common_utils) is reached by passing a `mock_response=litellm.ContentPolicyViolationError(...)` instance via the call-site kwarg — same deepcopy-avoidance reason. 11/11 tests pass locally. Patch coverage on litellm/router.py for this PR's diff should now be 100%. * chore(router): flip expose_router_debug_in_errors default to True Addresses @Sameerlite's review on #30418 — maintain backward compat on the wire. Redact becomes opt-in via setting the flag to False; the historical behavior (leak internal model_group / fallback wiring through exception messages) is preserved as the default. - litellm/__init__.py: default flipped to True, docstring rewritten with deprecation note pointing at a future flip to False (redact by default) in a major release. - tests/test_litellm/test_router_exception_redaction.py: fixture resets to True (was False); the "off" tests now explicitly set False; the "default_leaks_*" tests rely on the fixture default. test_flag_defaults_off -> test_flag_defaults_on. - No router.py change needed; the gate keys off the same flag, only the default changes. - PR title no longer needs the breaking-change `!` marker — no client sees a behavior change at default settings. 11/11 pass locally. * ci: retrigger workflows after base branch change to litellm_internal_staging * feat(guardrails): integrate Repelloai Argus guardrail (#30465) * feat(guardrails): add RepelloAI Argus guardrail integration (#1) * feat(guardrails): add RepelloAI Argus guardrail integration Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed asset policies enforced via an asset_id and X-API-Key auth. * fix(guardrails): harden RepelloAI Argus guardrail - scan streaming responses on output (was bypassing the guardrail) - log blocked verdicts as guardrail_intervened instead of success - treat auth/config errors (401/403/404/422) as misconfiguration that always blocks, not a fail-open-able unreachable error - default unreachable_fallback to fail_closed and read it directly; block on unknown/malformed verdicts so an API change can't silently disable enforcement - type unreachable_fallback as a Literal, drop the duplicate config model, expose unreachable_fallback in the config schema, and stop leaking the raw provider response / exception strings to the client * fix(guardrails): address RepelloAI Argus review feedback - support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback) - make asset_id required in the config model - normalize unreachable_fallback so only fail_open opens; block on 400 misconfig - correct the shared unreachable_fallback field description * docs(guardrails): add RepelloAI Argus docs page and dashboard listing - add docs page covering config, env vars, modes, verdicts, failure semantics - list RepelloAI Argus in the Guardrail Garden with provider/logo mappings - add a regression test for the provider logo and display-name resolution * fix(guardrails): keep RepelloAI asset_id optional in config model A required asset_id leaked onto the shared LitellmParams (which inherits RepelloAIGuardrailConfigModel), breaking validation for every other guardrail. Keep it optional like sibling models; the guardrail __init__ still raises when asset_id is missing, which is the real enforcement. * Add comment for last user turn scanning * feat(guardrails): harden repelloai scanning * feat(guardrails): expand repelloai scanning to include tool definitions Add extraction of tool definitions and tool call arguments to the RepelloAI guardrail scanning. Improves detection coverage by including function schemas and parameters in the prompt sent to the guardrail service. Also captures detailed error responses in logs and adds guardrail header to streaming responses. * refactor(guardrails): fix and harden repelloai schema text extraction - Fix duplicate text in _iter_schema_text: previously all dict values were re-queued onto the stack even after scalar/list keys were already extracted explicitly, causing names/descriptions to appear twice in the scanned prompt - Extract schema key frozensets to module-level constants so they are not reconstructed on every call - Change _iter_schema_text from @classmethod to @staticmethod (cls unused) - Narrow _call_analyze stage param from str to Literal["prompt", "response"] - Add HttpxResponse type annotation to _raise_for_config_error - Add LLMResponseTypes annotation to async_post_call_success_hook response param * fix(guardrails): resolve pyright type errors in repelloai guardrail - Narrow async_handler.post return from Response|None to Response with explicit None guard before calling raise_for_status/json - Fix list comprehension returning str|None by switching to explicit loop with isinstance guard so pyright tracks the narrowing - Cast model_dump() result to Dict since hasattr does not narrow object type in pyright * fix(guardrails/repello): include Responses API instructions field in prompt scan The /v1/responses top-level `instructions` field was not included in _extract_prompt_text, allowing a caller to bypass guardrail policy checks by putting blocked content in `instructions` while keeping `input` benign. * feat: add api_key to config model and read prompt from data dict * fix(guardrails/repello): plug input_text and tool-call response bypass gaps Responses API input content parts with type 'input_text' were silently dropped by build_inspection_messages (which only handles type='text'), allowing callers to send blocked content via that path without triggering the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail and call it when walking the Responses API input messages. Post-call scanning skipped responses whose choices contained only tool_calls or function_call (message.content=None), letting models put blocked output in function arguments undetected. Fix: _extract_chat_completion_text now calls _extract_tool_call_args_from_message on each choice message. Also replace typing.Dict/List with builtin dict/list to clear TID251 strict ruff violations introduced by this file. * fix(guardrails/repello): scan Responses API function_call output arguments Output items with type 'function_call' in a /v1/responses response were skipped by _extract_responses_api_text; only 'message' items were walked. A model could return blocked content in function_call.arguments undetected. Now extract arguments from function_call output items before scanning. * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486) * fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its result is carried in `provider_specific_fields.web_search_results` — PRs #17746 / #17798 restore it for callers that round-trip provider_specific_fields. A generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on replay and instead sends back an assistant `tool_call` + a `tool` message both keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use` (with no following *_tool_result) plus a user `tool_result` for the same id — both invalid, so the next turn 400s: messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks: srvtoolu_... Each `tool_result` block must have a corresponding `tool_use` block in the previous message. This is the commonly-reported vertex_ai symptom where Gemini works but Claude 400s on the 2nd turn of a web-search chat. Fix (litellm/litellm_core_utils/prompt_templates/factory.py): - convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching *_tool_result is available to pair with it; otherwise skip it (a bare server_tool_use is itself rejected). - anthropic_messages_pt: drop a replayed `tool`/`function` message whose tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client result; a user tool_result for it is invalid). The existing reconstruction path (provider_specific_fields present, e.g. the litellm SDK) is unchanged, as is regular client tool_use/tool_result. Tests (tests/llm_translation/test_prompt_factory.py): - update test_convert_to_anthropic_tool_invoke_server_tool -> test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped - add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool Follow-up to #17746 / #17798; addresses the generic-client (no provider_specific_fields) case of #17737. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite The regression tests added in tests/llm_translation/test_prompt_factory.py aren't run by the coverage CI job (it runs tests/test_litellm), so the new factory.py branches showed as uncovered (codecov patch coverage). Add equivalent focused tests in the unit suite so both new branches are exercised there: - convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no matching *_tool_result is available. - anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic OpenAI client replays. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(anthropic): cover the server_tool_use + result valid-pair path in unit suite Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke emitting server_tool_use followed by its web_search_tool_result when the matching result is present (the litellm-SDK round-trip path). Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * style(anthropic): flatten srvtoolu_ tool-message guard to a negated if Addresses the Greptile style nit: replace the if-pass/else with a single negated `if not (...)` guard around the tool_result append. Behavior unchanged. Refs #17737 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(proxy): require premium only when enabling premium metadata fields (#30285) (#30506) Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (#30488) * fix(perplexity): stop double-billing reasoning tokens in manual cost fallback When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR #18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate. Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions). Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate). Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced. * style(perplexity): drop redundant type annotation on else branch to satisfy mypy mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file. * fix(perplexity): update integration test expected costs for non-double-billed math Three tests in test_perplexity_integration.py asserted the old buggy expectation that reasoning_tokens are billed in addition to the full completion_tokens count. After the fix in cost_per_token, reasoning_tokens are billed at the reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the standard output rate, matching OpenAI/Perplexity convention (PR #18607). Updates: test_end_to_end_cost_calculation_with_transformation, test_main_cost_calculator_integration, test_high_volume_cost_calculation. The high-volume sanity threshold drops to 0.25 to reflect the corrected total. * fix(ui): use dynamic proxy base URL in MCP usage examples (#30487) Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the MCP server usage example and copy-to-clipboard snippet so the generated configuration works for non-local deployments. Fixes #30466 * feat: add missing UK PII entity types to Presidio guardrail (#30537) * feat: add missing UK PII entity types to Presidio guardrail Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection. * test: remove fragile hardcoded entity count test Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions. * style: apply Black formatting to match CI requirements * fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357) Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0 Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry Fixes #30346 * feat(mcp): make MCP gateway name and description configurable via env vars (#30473) * feat(mcp): make MCP gateway name and description configurable via env vars * Rename function _restore_env to _apply_env * docs(mcp): document import-time capture of env-backed identity constants Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module reload to observe env changes after import. Generated with AI assistance Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Yevhen Luhovtsov <yevhen.luhovtsov@intapp.com> Co-authored-by: Claude <noreply@anthropic.com> * fix(mcp): preserve native tools in semantic filter hook (#26650) * fix(mcp): preserve native tools in semantic filter hook The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP + native) to filter_tools(), which only knows MCP-registered tool names. Native tools silently failed the name match in _get_tools_by_names() and were dropped from the request. Fix: partition tools into native and MCP-registered before filtering. Run the semantic filter only on MCP tools, then merge native tools back unconditionally. Changes: - Robust _is_mcp_tool() using shape-based detection for OpenAI-format dicts, safe regardless of future _extract_tool_info changes - Single-pass partition loop (no double _is_mcp_tool calls) - Preserve native tools in MCP expansion path (mixed requests) - Track MCP expansion to prevent expanded tools bypassing filtering - filter_stats reports MCP-only counts for accurate metrics - Extracted _emit_filter_metadata() helper - Skip spurious filter headers for all-native tool requests Closes #26212 * remove stale docstring note referencing tools_expanded_from_mcp * fix: handle Responses API name collision and preserve tool ordering - Classify Responses API tools ({type: 'function', name: '...'}) as native to prevent name collisions with MCP canonical names - Preserve original request tool ordering using id()-based merge instead of naive native+mcp concatenation - Add 2 regression tests: name collision and ordering preservation * style: apply black formatting * fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge * lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention) * ci: retrigger checks after rebase onto litellm_internal_staging * feat(fireworks): sync Fireworks AI model registry with current platform catalog (#30616) Adds 12 new Fireworks serverless models and updates 3 existing entries in model_prices_and_context_window.json and its bundled backup to match the current Fireworks platform model list. New direct models: glm-5p2, qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6, deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast, kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and gpt-oss-20b now carry correct output token caps, cache-read pricing, and explicit capability flags max_tokens is set equal to max_output_tokens (not the full context window) for models whose generation cap is below their context window. This avoids the shared input+output budget path in get_modified_max_tokens, which would otherwise let callers request output sizes the model cannot produce. The same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b entries that had max_tokens equal to the full context window Short-form aliases (fireworks_ai/<model>) are added for every direct accounts/fireworks/models/ entry so cost attribution works for callers using bare model names. Router endpoints get short-form aliases too, and transform_request now routes bare names ending in -fast to the accounts/fireworks/routers/ path instead of defaulting every bare name to models/. This keeps the kimi-k2p6-fast router from being misrouted to the nonexistent models/kimi-k2p6-fast endpoint kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its replacement. Context windows for deepseek-v4 and kimi models use the power-of-two values (1048576 and 262144) published on the Fireworks model pages, matching the convention already used by existing entries Two regression tests in test_utils.py assert the exact per-token costs, token limits, capability flags, and short-form-to-long-form equality for all 15 models against both the main and backup cost maps. Two routing tests in test_fireworks_ai_chat_transformation.py verify bare -fast names route to routers/ and bare direct-model names route to models/ * fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443) * feat(anthropic): hoist leading in-array system to top-level (helper) * test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control * test(anthropic): mid-conversation system normalization cases * feat: add supports_mid_conversation_system flag to Claude Opus 4.8 Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost map and the bundled package backup, since the runtime helper and tests read the backup in local/offline mode. Pin the mid-system passthrough regression test to the local cost map via the existing local_model_cost_map fixture so it reads the branch-local flag rather than the network-fetched main copy. * fix(bedrock): normalize in-array system in /v1/messages handler (#29698) Wire normalize_system_messages_for_anthropic into anthropic_messages_handler so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform / Converse-bridge) hoist leading in-array system entries (and demote mid-conversation ones on models lacking supports_mid_conversation_system) into the top-level system field. The normalized messages/system are written back into the local_vars snapshot the base_llm branch reads from, otherwise the Invoke/Mantle fix would silently no-op. Also fix the helper to resolve supports_mid_conversation_system through the prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw _supports_factory could not see the flag once get_llm_provider left the invoke/ prefix on the model id, which would have wrongly demoted mid-conversation system on a Bedrock invoke opus-4-8 path. * fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param * fix(types): widen system param to Union[str, List] for hoisted system blocks * refactor(bedrock): drop dead local_vars messages writeback * fix(bedrock/converse): translate in-array system in anthropic->openai adapter (#29698) * fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty * fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches * fix(types): register supports_mid_conversation_system in model-info schema The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid) rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8 cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a typed, first-class capability flag alongside its peers (supports_output_config, etc.). --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (#28885) * fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload By default the agentcore provider flattens the last message to a text-only {"prompt": "..."} payload via convert_content_list_to_str, silently dropping OpenAI multimodal blocks (image_url, file, input_audio, ...). This adds an opt-in `forward_multimodal_content` litellm param. When truthy and the last message's content is a list containing a non-text block, the original OpenAI content list is forwarded verbatim under a new "content" field so an attachment-aware AgentCore agent can read it. Default off keeps the payload byte-identical to the legacy {"prompt": "..."} shape — existing agents are unaffected. The flag is read from optional_params (where other AgentCore params land) with a litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...). AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint parses arbitrary JSON up to 100 MB (per https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html), so this is a purely upstream change; no AgentCore-side schema is asserted. * fix(bedrock/agentcore): shallow-copy forwarded multimodal content list Address review feedback (Sameerlite): payload["content"] = last_content aliased the caller's mutable messages[-1]["content"] list. Harmless today because the payload is JSON-serialized immediately, but a latent footgun if a future caller mutates the returned payload before serialization. Forward list(last_content) so the payload owns its own list. Block dicts stay shared on purpose — a deep copy would clone potentially large base64 media on the request hot path, and the flagged risk was the shared list, not the blocks. Update the passthrough tests to assert equality + distinct identity, and add a regression test that mutating the payload list can't leak back into the original message content. * Revert "fix(mcp): preserve native tools in semantic filter hook (#26650)" This reverts commit |