mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-17 23:51:30 +00:00
75 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Yassin Kortam
|
7978b9f721
|
fix(proxy): stop leaking internal exception details to clients (#39380)
* fix(proxy): stop leaking internal exception details to clients Public error responses could disclose internal details in two places. A proxy-layer exception with no recognized provider status code (a bug in a custom callback, a hook, or litellm's own code) forwarded its raw str() text verbatim on a 5xx, including any embedded credential, filesystem path, or internal hostname, or a full stack trace; the same client-facing message now runs through a redaction layer built on top of the credential redaction that already runs on log output, so it also drops an embedded traceback and scrubs path-shaped and hostname-shaped substrings. It intentionally never runs on server-side logs, which must keep full detail for debugging. exception_type(), litellm's core exception mapper, is shared by direct SDK callers (litellm.completion()) and the proxy, and it deliberately embeds a traceback into an unmapped exception's message as a debugging aid for library users; a first pass at this fix stripped that traceback inside exception_type() itself and broke that convention (caught by tests asserting on the traceback frame). The traceback stays in exception_type()'s own output; only the proxy's client-facing response boundary (and the streaming response generator, which never needs to embed one at all) strips it. Full generic-message replacement for the unclassified-exception case was tried first and reverted too: several routes deliberately raise a bare exception as an informative, secret-free validation message (e.g. the OCR endpoint's rejection of provider-native file IDs), and replacing those wholesale broke that convention; targeted redaction leaves them untouched. Also stops the default uvicorn-based proxy from sending a Server response header. Resolves LIT-6747 * refactor(proxy): drop the unrelated error-message constant and trim redaction comments Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> |
||
|
yucheng-berri
|
b4f5b6aa94
|
fix(logging): redact credential query params from the uvicorn access log (#39293)
* fix(logging): redact credential query params from the uvicorn access log Raw virtual keys reached container stdout two ways: - `GET /key/info?key=sk-...`, `/global/spend/report?api_key=sk-...`, `/key/spend/report`, `/spend/logs`, `/user/daily/activity` and the Gemini passthrough routes all put the credential in the request target, and `uvicorn.access` had no redaction filter (only `uvicorn.error` did). - the key budget error interpolates `LiteLLM_VerificationToken.key_name`, a column with no enforced shape, into a message that is both logged and returned to the caller. `SecretRedactionFilter` cannot be reused on an access logger: it collapses the record into `record.msg` and clears `record.args`, and uvicorn's AccessFormatter unpacks those args at emit time, so every access line would raise TypeError. `AccessLogRedactionFilter` scrubs the positional args in place instead. An access line is the one input to the secret regex an unauthenticated caller controls end to end, so two bounds go with it. The request target is cut back to a whole query parameter under 512 characters before it is scanned, since a half parameter is too short to match its own pattern and would be logged raw, and the dropped tail is not logged at all. The connection-string pattern is bounded too, because its user half could previously re-scan the rest of the string from every `://`: a 16 KB URL of `a://` pairs took 314s and now takes 0.12s, with the caps set high enough that an RDS IAM auth token used as a DSN password still redacts. Credential query params are terminated by `&` like the existing `key=` and `sig=` patterns, so redacting one param no longer swallows the rest of the request line, and a second credential in the same query string is now redacted on its own instead of surviving once the first one stops the span. `key_name` is echoed into the budget error only when it still has the masked `sk-...abcd` shape `abbreviate_api_key` writes, so a value put there by a direct DB write or a migration falls back to the key alias. Also point the `/key/info` and spend-report examples at the sha256 hash both endpoints already accept, so callers stop putting raw keys in URLs that third-party access logs record. Resolves LIT-5909 * test(logging): assert on emitted access lines instead of filter registration The two registration tests checked that an AccessLogRedactionFilter instance sits in uvicorn.access.filters, which is the shape of the code rather than its behavior. Handing the logger a real access record and reading what a handler wrote covers the same wiring and still fails when the registration is removed. * fix(logging): redact percent-encoded credentials from access logs ?k%65y=sk%2D... is a working credential once the request parser decodes it, but the redaction patterns match literal text and never see it. Decode the request target as a detector and drop the query when decoding reveals a secret. The decoded text is never logged back, so a %0A cannot forge a following log line Also accept any four non-space characters in the masked key_name check, since abbreviate_api_key copies the last four characters of a custom key verbatim and those can be punctuation or non-ASCII * fix(auth): keep control codes out of the masked key label /key/generate accepts a custom key ending in an escape sequence, and abbreviate_api_key copies those four characters into key_name verbatim, so the over-budget message carried them to a terminal and a log viewer. Bar whitespace and C0/C1 control codes from the four, and keep everything else |
||
|
yucheng-berri
|
3fadcd7155
|
fix(auth): quiet malformed virtual key rejections to stdout (#38838)
* fix(auth): quiet malformed virtual key rejections to stdout Reduce noisy invalid-api-key error logs by classifying malformed virtual keys and routing their rejections to stdout as WARNING instead of stderr as ERROR. Suppressible via LITELLM_LOG=ERROR or log_client_error_tracebacks=true. Changes: - auth_utils: is_invalid_virtual_key_error() classifier and marker functions - auth_exception_handler: log invalid keys as WARNING to child logger before identity seeding and callbacks, escalate non-401 transforms to ERROR - user_api_key_auth: websocket early-raise WebSocketException(1008) to avoid double-logging at HTTP layer - _logging: child logger verbose_proxy_stdout_logger with no handler/level; LevelRoutingStreamHandler routes its WARNING records to stdout; handler setLevel in _turn_on_json() closes JSON config handler level leak - test_auth_exception_handler: new test case verifying malformed-key logs at WARNING with marker retention through transformations Fixes LIT-5362 * fix(auth): classify malformed-key 401 by raise-site marker, not message text Review round 1 (Greptile P2, veria Low): - Move the marker attribute name to litellm/constants.py per the shared sentinel convention - Stamp the marker on the malformed-key 401 where it is raised and classify only by it. Message text is caller-influenceable on other 401s (vector store ids, organization ids are interpolated into their messages), so a phrase match would let a request body demote an authorization failure to the quiet log path - Regression test: a 401 carrying the phrase but not the marker stays at ERROR on stderr |
||
|
yucheng-berri
|
22a349ee70
|
fix(logging): stop stream-based log collectors classifying INFO logs as errors (#38476)
Route records below WARNING to stdout (WARNING and above stay on stderr), emit ANSI color codes only when both streams are a TTY (honoring NO_COLOR), and parse JSON_LOGS strictly so JSON_LOGS=false no longer enables JSON logs. |
||
|
mubashir1osmani
|
66b930b540
|
fix(logging): preserve uvicorn color_message args during secret redaction (#37122)
* fix(logging): preserve uvicorn color_message args during secret redaction SecretRedactionFilter clears record.args after substituting record.msg, but uvicorn's colorized formatter re-renders the separate color_message extra field against record.args at emit time. With args cleared, uvicorn prints the raw "%s://%s:%d" template instead of the actual startup URL whenever output goes to a TTY (colors on). * fix(logging): narrow color_message fallback to TypeError Bare except-Exception-pass on the color_message substitution pushed the BLE001 and S110 strict-rule budgets over their ceiling. Narrow to the one exception the %-format can actually raise and give it a real fallback instead of silently swallowing it. * refactor(logging): move color_message substitution into a helper The two record.color_message stores put LIT011 over its ceiling. Building the value in a pure helper leaves one store, marked rebind-ok since scrubbing a record in place is the logging.Filter contract. Also pins the ordering that makes the substitution safe: it has to run before args are cleared, which puts it before the extra-field loop that redacts the result, so a secret arriving through record.args is still scrubbed out of color_message. |
||
|
devin-ai-integration[bot]
|
a9744645ee
|
fix(logging): bound oversized error payloads written to stdout (#37684)
Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
ryan-crabbe-berri
|
4493c826e7
|
fix(logging): close three secret-leak paths in verbose logging (#37391)
* fix(logging): close three secret-leak paths in verbose logging
The AWS credential pattern was the only key-name matcher in secret_redaction
that skipped optional quotes, so quoted dict-repr values leaked. Fold the three
AWS key names into the shared key-name alternation instead.
SecretRedactionFilter only scrubs str record attributes, so a dict/list/set
passed through extra={...} reached the formatter unredacted. Redact at the
formatter boundary so no value shape can bypass it.
log_raw_request_response wrote the request curl command to metadata["raw_request"]
unredacted, returned an unmasked raw_request_api_base, and fell back to dumping
model_call_details whenever api_base was empty.
* Update litellm/_logging.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(logging): redact JSON log values without breaking the document
JsonFormatter redacted the serialized JSON, so a secret-named member
collapsed from '"api_key": "sk-..."' to a bare REDACTED token and the
line stopped parsing as JSON.
Redact before serialization instead: safe_dumps takes an optional
value_transform hook (default None, so all other callers are unchanged)
and redact_structured_value collapses only the value, leaving the key
and surrounding structure intact.
JsonFormatter now emits "api_key": "REDACTED" where the formatter unit
test expected the already-masked "sk**********". That test bypasses
SecretRedactionFilter, which in production collapses the pair before any
formatter runs, so the assertion is updated to match real behavior.
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
|
||
|
Deepanshu Lulla
|
9ce96c2d34
|
feat(logging): add opt-in session_id and trace_id correlation to JSON log records via contextvars (#34418)
* feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars
Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and
two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both
setters after assigning litellm_trace_id so every JSON log record emitted within the
async request context carries trace_id and, when provided, session_id — enabling log
correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without
any changes to individual log call sites.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields
* fix(logging): always reset session_id_var to empty string when no session_id provided
* feat: gate request correlation IDs in logs behind request_correlation_in_logs flag
* refactor: move correlation ID injection into CorrelationContextFilter
* feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload
Plaintext log lines (json_logs off) now get the same trace_id/session_id
suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a
visible effect regardless of log format.
StandardLoggingPayload gets a new independent session_id field, populated
from litellm_session_id. trace_id's existing session_id-first fallback is
preserved when request_correlation_in_logs is off; with the flag on, an
explicit litellm_trace_id now takes priority over litellm_session_id so
the two fields carry genuinely independent values.
* fix(logging): restore correlation context after nested calls; sanitize correlation ids
Addresses two review findings on this PR.
CorrelationContextFilter's trace_id/session_id contextvars were set on every
Logging.__init__ but never reset, so a nested LiteLLM call sharing the same
asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call,
an MCP sampling call) would leave the outer request's subsequent log lines
stamped with the nested call's ids instead of its own. set_trace_id/
set_session_id now return their contextvars.Token, and Logging stores them
and resets both once its own success/failure handler actually completes,
via a new idempotent _restore_correlation_context() called from all four
terminal handlers.
set_trace_id/set_session_id also now strip control characters and bound
length before storing a caller-controlled trace_id/session_id, since these
values can originate from request input (litellm_session_id, x-litellm-
trace-id) and get interpolated into plain-text log lines - without this, a
caller could embed \r/\n or escape sequences to forge fake log entries.
* fix(logging): restore correlation context after nested calls, not before
The previous commit called _restore_correlation_context() as the first
line of each terminal handler, before that handler's own callback
dispatch loop runs. That's backwards: a nested LiteLLM call triggered
from within a callback (e.g. a guardrail's own LLM-as-judge call) would
then capture the *already-reset* value as its own pre-call baseline,
and its own reset would restore to that instead of the true outer
value - verified live to still leak.
success_handler/async_success_handler/failure_handler/async_failure_handler
are now thin wrappers: the original bodies move to
_success_handler_body/etc, called inside a try/finally that restores
context only once the full body - including any nested calls its own
callback dispatch triggers - has actually finished, mirroring proper
stack-scoped nesting semantics.
* test(logging): cover async_failure_handler's correlation-context restore
Codecov flagged the new async_failure_handler wrapper (try/finally around
_async_failure_handler_body) as uncovered - the method had no direct test
at all before this PR's refactor split it into a wrapper. Adds a test that
awaits it directly and asserts both that async_log_failure_event still
fires and that _restore_correlation_context() puts the pre-call
trace_id/session_id back.
* fix(logging): restore correlation context by value, not by contextvars.Token
veria-ai correctly flagged that contextvars.Token.reset() only works in the
exact Context it was created in, and litellm's async success path (and
streaming failure path) dispatch async_success_handler/async_failure_handler
via asyncio.create_task and the global logging worker - a different Context
than Logging.__init__ ran in. reset_trace_id/reset_session_id silently
swallowed the resulting ValueError, so the restore was a no-op for exactly
those paths. Verified independently: reproduced the raw contextvars
behavior, then confirmed litellm's async success dispatch really does go
through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py).
Logging now captures the pre-call *value* (not a Token) and restores via a
plain set_trace_id()/set_session_id() call, which works regardless of which
Task/Context calls it. reset_trace_id/reset_session_id are removed as
dead/unreliable code. Added a regression test that spawns __init__ and the
restore in different asyncio Tasks - confirmed it fails against the prior
Token-based commit and passes here.
* fix(logging): restore correlation context in the originating task too
Greptile's re-review correctly identified a remaining gap: for a
successful acompletion(), async_success_handler is dispatched via
asyncio.create_task + the global logging worker into a *different* Task
than the one wrapper_async/Logging.__init__ ran in. The prior fix (43c164a)
only restored the handler's own (detached, throwaway) Task - it never
touched the originating request Task, which keeps this call's trace_id/
session_id set for the rest of its own execution (e.g. nested calls made
via the same Task).
wrapper()/wrapper_async() in litellm/utils.py now restore the originating
Task's correlation context in a finally block once the whole call is done,
regardless of what detached logging tasks it spawned along the way. Since
the wrapped body rebinds its own `kwargs` local via function_setup(),
sharing the dict object doesn't work here; a small mutable holder carries
the constructed Logging instance back out to the outer wrapper instead.
_restore_correlation_context() is no longer guarded against repeat calls:
with value-based (not Token-based) restoration, each distinct Task that
calls it needs its own restore to take effect in that Task's own view of
the contextvars, so multiple calls (once per Task involved in an attempt)
are required, not just tolerated.
Added a regression test using mock_response to exercise the real success
dispatch path (asyncio.create_task + GLOBAL_LOGGING_WORKER) without a live
provider call, asserting the *test's own* (originating) task context is
restored after the call - this is exactly the case Greptile flagged and
the prior commit didn't cover.
* fix(logging): restore correlation context when function_setup itself fails
Greptile's 4th finding: if function_setup() constructs Logging() (whose
__init__ already mutates trace_id_var/session_id_var) and then raises
before returning - e.g. update_environment_variables() throws - the
caller's wrapper()/wrapper_async() never receives a logging_obj reference,
so its own restore-on-finally never fires. The correlation ids leak into
every subsequent log line on that thread/task with no way to clear them.
function_setup()'s own except block now restores the context itself in
that case, using whatever logging_obj it managed to construct before
failing (locals().get(), safe against the earlier failure modes where
logging_obj was never assigned at all).
Added a regression test that monkeypatches Logging.update_environment_variables
to raise after construction, confirmed it fails without this fix (the
leaked ids show up directly in the raised exception's own log line) and
passes with it. Broader sweep (test_utils.py, test_router.py,
test_main_module_header.py, streaming handler tests, plus all
logging-specific tests): 722 passed.
* fix(logging): don't assume every litellm_logging_obj is a real Logging instance
CI caught a real regression from the last commit: tests/test_litellm/llms/xai/test_xai_key_fallback.py
injects a minimal FakeLogging stand-in (only implementing
update_from_kwargs) as litellm_logging_obj for a narrow realtime-config
unit test, bypassing the real Logging class entirely. wrapper()/
wrapper_async()'s finally block and function_setup()'s except block both
unconditionally called _restore_correlation_context() on whatever ended up
in the holder, which doesn't exist on that stand-in.
_restore_correlation_context is new plumbing specific to this PR's
feature, not part of any pre-existing stand-in's expected interface, so
callers of it can't assume every object playing the litellm_logging_obj
role implements it. Added _restore_correlation_context_if_supported(),
a small getattr-guarded helper, and used it at all three call sites.
* fix(logging): don't restore context too early on setup failure or streaming
Two more findings from Greptile's 5th review round.
1. function_setup()'s except block restored correlation context *after*
logging the "Error in function_setup" exception, so that diagnostic log
line itself was stamped with the doomed call's ids instead of the outer
ids - misleading, since the failed call never produces anything else to
attribute those ids to. Restore now happens before the log call.
2. wrapper()/wrapper_async() restored the originating task's context as
soon as a streaming call returned, before the caller ever starts
iterating the CustomStreamWrapper it just got back. Any log lines
emitted while iterating (in the same thread/task) incorrectly showed
the pre-call ids instead of this call's own ones. The wrapper finally
block now skips the restore when the return value is a stream wrapper,
deferring to the terminal handler that already fires once the stream is
actually assembled/exhausted.
Both verified with tests that fail against the prior commit and pass
against this one. Broader sweep unchanged at 829 passing.
* fix(logging): best-effort correlation cleanup on abandoned streams
Greptile's 7th finding: if a caller returns a streaming response and never
fully consumes it - stops iterating early, drops the reference, cancels
it - the terminal handler that normally restores the originating task's
trace_id/session_id never fires, since it only runs once the stream is
actually assembled/exhausted. The ids leak into every subsequent log line
in that thread/task with no bound.
There's no reliable Python hook for "this was abandoned without being
closed" - CustomStreamWrapper has no close()/__aexit__/context-manager
convention today, and the only automatic option is __del__, whose timing
is inherently unpredictable (delayed by cyclic GC, not guaranteed at
interpreter shutdown, can run on a different thread). This is a best-effort
safety net, not a guarantee, and is documented as such in the docstring.
Testing this via real garbage collection proved unreliable in practice:
per-chunk logging submits work to a thread pool executor whose worker
thread transiently holds its own bound-method reference to the wrapper
until that task completes, so refcount doesn't hit zero on a
deterministic schedule even with polling. Tests call __del__ directly
instead - a plain method, safe to invoke early - which exercises exactly
the restore logic real garbage collection would eventually trigger,
plus a case confirming a broken logging_obj can never make __del__ raise.
* fix(logging): restore consumer's context at every real stream exit point
Two more findings from this round.
Veria AI: even a *fully consumed* stream never restored the actual
consuming thread/task's correlation context. The terminal success dispatch
(dispatch_success_handlers via asyncio.create_task for async, or
success_handler via the shared executor for sync) only restores whatever
detached context it runs in - never the caller's own thread/task that's
running the for/async for loop. Same root cause as the wrapper-level fix
two rounds ago, just missed for the streaming-completion path.
Greptile: explicit aclose() (client disconnect, router fallback aborting
a partial stream) closed the underlying stream without restoring
correlation context either, since request wrappers intentionally skip
restoration for returned streams and no terminal handler runs on this
path.
Added CustomStreamWrapper._restore_consumer_correlation_context(), called
from every point control genuinely returns to the consumer: the final
raise StopIteration/StopAsyncIteration on natural exhaustion (both sync
branches, both async branches), _handle_stream_fallback_error (the shared
choke point for all three failure-raising call sites), and aclose(). __del__
now delegates to the same helper instead of duplicating it.
Verified with tests extending the existing streaming-exhaustion cases to
assert the consuming context is restored after the loop completes (fails
against the prior commit, passes now), plus a dedicated aclose() test.
Broader sweep: 832 passing.
* fix(logging): don't let a delayed __del__ finalizer clobber a newer active call
If an abandoned stream's __del__ fires late (after cyclic GC delay), a
different call may have already taken over the correlation contextvars in
the same Task/thread. Restoring unconditionally would stomp that active
call's trace_id/session_id with the abandoned stream's stale pre-call
snapshot. __del__ now only restores when the contextvars still hold the
ids this call itself set.
* fix(logging): compare sanitized ids in the __del__ ownership guard
set_trace_id()/set_session_id() sanitize (strip control chars, bound length)
before storing, so the contextvar's value can differ from the raw
litellm_trace_id/litellm_session_id. The __del__ ownership guard was
comparing against the raw values, so a caller-supplied id containing control
characters or exceeding 256 chars would never match, permanently skipping
cleanup. Capture what set_trace_id()/set_session_id() actually stored and
compare against that instead.
* fix(logging): restore consumer context on the synthesized finish_reason chunk
Both __next__ and _finalize_completed_stream() have a branch that fires when
the underlying stream ends without ever emitting an explicit finish_reason
chunk: they synthesize one via finish_reason_handler() and return it. A
consumer that stops as soon as it sees finish_reason - a common pattern -
never calls __next__()/__anext__() again, so the existing restore in the
sent_last_chunk-is-True StopIteration branch never runs for them. The
underlying stream is already exhausted at this point regardless of whether
the caller keeps iterating, so restoring here is safe.
* fix(logging): don't restore correlation context before the caller receives the final chunk
The previous fix (5147c69186) restored context immediately before returning
the synthesized finish_reason chunk from __next__/_finalize_completed_stream,
reasoning that completion_stream was already exhausted. But that chunk is
still this call's own data, and the caller's own application-level log
statements processing it run in the same synchronous frame right after the
return - restoring first made those lines carry the wrong (outer) ids,
exactly what wrapper()/wrapper_async() deliberately avoid by not restoring
while a stream is being iterated.
Revert to not restoring there. A caller that keeps iterating still gets a
correct, deterministic restore on its very next __next__()/__anext__() call
(completion_stream is exhausted, so that immediately re-raises
StopIteration/StopAsyncIteration through the already-restoring branch). A
caller that stops right after finish_reason relies on aclose() or the
best-effort __del__ guard, same as any other stream the caller doesn't fully
exhaust.
* refactor(logging): hoist a safely-hoistable function-body import to module top
CorrelationContextFilter.filter()'s `import litellm` was a function-body
import; verified it can move to module top without a circular-import failure
(litellm/__init__.py already imports from litellm._logging before setting
request_correlation_in_logs, but a bare `import litellm` only binds the
already-in-sys.modules module object - the attribute itself isn't read until
filter() actually runs, by which point litellm is fully initialized).
* test(logging): move correlation tests into their conventionally-mapped files
tests/test_litellm/ mirrors litellm/ in a parallel path. Correlation tests
for the Logging class (litellm_logging.py), function_setup/wrapper_async
(utils.py), and CustomStreamWrapper (streaming_handler.py) had all landed in
test_logging.py, which only maps to litellm/_logging.py itself. Moving each
group to its correctly-mapped file: test_litellm_logging.py (Logging class
init/restore), test_utils.py (function_setup, wrapper_async), and
test_streaming_handler.py (CustomStreamWrapper) in the next commit.
test_logging.py keeps only what actually exercises _logging.py's own
contextvars/filters/formatters/sanitization. No behavior change - same
assertions, same coverage, just relocated.
* fix(logging): restore correlation context unconditionally in wrapper()'s sync path
Blocking finding from review: a caller-visible correlation feature was
silently misattributing one request's logs to a different, unrelated one on
the sync/threaded path. wrapper()/wrapper_async() both left trace_id/session_id
"open" across a stream's entire iteration so the caller's own log lines while
consuming it would carry the right ids. That's safe for wrapper_async(): each
async call gets its own asyncio Task with its own copy of the contextvars,
and Tasks are never recycled across requests, so a leftover value can only
ever affect that one already-abandoned Task.
It is not safe for wrapper() (sync): a plain OS thread has no such per-call
isolation, and a thread pool's worker threads *are* recycled across unrelated
requests. If a sync stream was abandoned (client disconnect, early break, an
uncaught exception) without ever being exhausted or closed, nothing restored
its contextvars, and a pool could later hand that same thread to a completely
different call, which would inherit the abandoned request's ids as its own
"pre-call" baseline and then restore back to that poison when it finished -
permanently misattributing every subsequent log line on that thread,
including its own, to the abandoned request. Strengthening the __del__
finalizer already added for this can't fix it: finalizer timing is exactly
what a permanently-reused thread can't rely on.
wrapper() now restores unconditionally in its own finally, before a sync
stream is ever handed back to the caller. The trade-off: a sync stream
consumer's own application-level log statements while iterating no longer
automatically carry this call's ids (litellm's own internal per-chunk
logging is unaffected, since it's dispatched separately). That's an
acceptable cost for eliminating a silent cross-request misattribution bug.
wrapper_async() keeps the existing conditional (skip-if-streaming) behavior,
justified by the Task-isolation argument above; CustomStreamWrapper's
__del__/aclose()/next-iteration restore machinery remains meaningful and
necessary there.
This also simplifies wrapper()/wrapper_async() back toward their original
shape: both previously used a mutable-dict-holder split into a separate
_body function to smuggle logging_obj/result out to an outer finally,
working around function_setup() rebinding its own local `kwargs`. That
restructuring is no longer needed - `logging_obj` (and, for wrapper_async(),
`result`) were already function-level locals in scope for a plain
try/finally; three of wrapper_async()'s retry-return statements now assign
through `result` first so it accurately reflects what's actually returned
even on a retry path.
Regression test: test_abandoned_sync_stream_does_not_contaminate_a_later_call_on_the_same_thread
in test_streaming_handler.py reproduces the exact reported scenario with a
real single-worker ThreadPoolExecutor - confirmed it fails with the prior
(skip-restore-on-stream) wrapper() and passes with this fix.
* refactor(logging): use Mapping instead of bare dict for read-only params
_get_standard_logging_payload_trace_id/_session_id only read litellm_params
(.get() calls, no mutation) - annotate it as Mapping[str, Any] rather than a
bare mutable dict, per the repo's no-mutable-collection-in-annotation rule.
* fix(logging): scope request_correlation_in_logs to the async/proxy path only
Blocking review finding: wrapper() (the sync entry point) used the same
skip-restore-on-stream design as wrapper_async(), but a plain OS thread has
no per-call context isolation the way an asyncio Task does, and a thread
pool's worker threads are recycled across unrelated requests - an abandoned
sync stream could leave its ids stuck on a thread a pool later hands to a
completely different request, misattributing that request's logs. A fix
existed and was tested (restore unconditionally in wrapper()'s own finally),
but it doesn't benefit this feature's primary consumer - the proxy only ever
calls the async entry point - and carries sync-specific complexity this PR
doesn't need.
Scope the feature to async only instead: Logging.__init__() takes a new
supports_correlation_logging parameter (default True), threaded down from a
new function_setup(..., is_async_call: bool = True) parameter. wrapper() is
the one caller that passes is_async_call=False; every other function_setup()
call site (wrapper_async(), the router, and proxy/MCP-internal call sites)
is already async and keeps the default. With
supports_correlation_logging=False, Logging.__init__() never calls
set_trace_id()/set_session_id() at all, so a sync call has nothing to leak
in the first place. wrapper() reverts to its pre-review shape with no
correlation-specific code at all.
StandardLoggingPayload's own trace_id/session_id fields are unaffected
either way - they're a deterministic per-call read of
self.litellm_trace_id/self.litellm_session_id, not ambient contextvar state,
so they were never exposed to the cross-request bug.
Full sync/direct-SDK support (stamping + its own safe-restore mechanism) is
deferred to a follow-up PR; the fix and its regression test already exist in
this branch's history at commit 9f3a20f4b2 and can be resurrected there.
Tests: replaced the two wrapper()-level tests with ones proving the new
invariant (sync calls, streaming and non-streaming, never touch
trace_id_var/session_id_var even when the caller explicitly passes
litellm_trace_id/litellm_session_id), and added a direct unit test for the
supports_correlation_logging=False gate on Logging.__init__ itself. Verified
live: a real proxy (Postgres-backed, real OpenAI calls) shows clean
trace_id/session_id isolation across two concurrent sessions with no
cross-contamination; a standalone script confirms real sync SDK calls
against a real model never touch the correlation contextvars.
* feat(logging): fall back to W3C traceparent/baggage for trace_id/session_id
request_correlation_in_logs previously only resolved trace_id/session_id from
litellm-specific sources: x-litellm-trace-id/x-litellm-session-id headers, a
generic x-<vendor>-session-id header, or Anthropic-style metadata.user_id. If
none were present, trace_id fell back to an auto-generated UUID unrelated to
anything else, and session_id stayed empty - even when the caller already had
real distributed-tracing instrumentation sending the actual industry-standard
headers for this.
Add a fallback to the W3C Trace Context traceparent header (trace-id
component) and W3C Baggage header (session.id entry), so a request already
carrying real OpenTelemetry trace context correlates litellm's own logs with
the same trace in the caller's observability backend (Datadog, Honeycomb,
Tempo, etc.) instead of getting an unrelated generated id. Precedence is
unchanged for existing sources: explicit litellm headers and the Anthropic
metadata path both still win over this new fallback, which only fires when
neither found anything. trace_id and session_id are resolved independently
here (unlike the existing chain_id mechanism, which uses one shared value for
both), since traceparent and baggage are semantically distinct W3C concepts.
New helpers _trace_id_from_traceparent/_session_id_from_baggage in
litellm_pre_call_utils.py parse the header formats directly (no new
dependency - both are simple fixed-width/delimited strings), wired into
LiteLLMProxyRequestSetup.add_litellm_metadata_from_request_headers() only
when the corresponding litellm_trace_id/litellm_session_id key isn't already
set by the existing paths.
Verified live against a real proxy: a bare traceparent header produces a log
trace_id exactly matching its trace-id component; a traceparent alongside an
explicit x-litellm-trace-id header (different value) produces a log showing
the explicit header's value, proving precedence.
* fix(logging): reserve trace_id/session_id in JsonFormatter against message-content spoofing
JsonFormatter merges keys parsed from the message body before applying extra
record attributes, and the extra-attributes loop skips a key that's already
present. A caller-controlled log message that happens to parse as JSON/dict
with a "trace_id"/"session_id" key (e.g. the proxy logging a raw request-header
dict) could therefore make the JSON record carry the attacker-supplied value
instead of the real correlation context set via CorrelationContextFilter.
trace_id/session_id are now applied from the LogRecord's own attributes after
message-content parsing, unconditionally overwriting anything the message body
claimed for those two keys.
* style(logging): fix import order (ruff I001) in _logging.py and litellm_logging.py
- _logging.py: import litellm belongs after the stdlib from-imports, grouped
with the other litellm.* imports, not before them.
- litellm_logging.py: the refactor to Mapping introduced a second, separate
`from collections.abc import Mapping` instead of merging it into the
existing `from collections.abc import Callable` import.
Caught by the strict-rule budget gate (ruff-strict-budget.json caps I001 at
0 new violations); both auto-fixed with `ruff check --fix --select I001`.
* style(logging): freeze mutable-collection constructions flagged by LIT002
Five sites in this PR's diff built a mutable list/dict literal instead of a
frozen value: a plain list of optional strings in CorrelationPlainFormatter,
a `kwargs or {}` fallback, a `metadata or {}` fallback, two `[...]` candidate
orderings, and a `dict(headers)` copy feeding a dict comprehension. Each is
build-once/read-only, so this rewrites them as tuples, MappingProxyType, or a
plain conditional `.get()` instead of seeding then reading a fresh mutable
collection - no behavior change, confirmed by the existing test suite.
Caught by the type-discipline budget gate (LIT002 capped at 0 new
violations).
* fix(logging): reserve trace_id/session_id even when no correlation context is active
Live-proxy verification surfaced a gap in the earlier message-content-spoofing
fix (7f390a57fc): that fix only overwrites trace_id/session_id from the
LogRecord's own attribute, so it does nothing for a log line emitted before
CorrelationContextFilter has stamped anything on this record (e.g. the
"Request Headers" debug line, which fires before Logging.__init__() runs for
the request). On such a record, a caller-supplied header literally named
trace_id/session_id still got promoted into the JSON output via the embedded
JSON/dict-repr parser, since there was no genuine value to protect.
Fixed at the source: trace_id/session_id are now excluded unconditionally from
the message-content-parsing promotion step, not just superseded afterward.
Verified live against a real proxy - the exact adversarial request (headers
literally named trace_id/session_id) no longer leaks into any JSON log record.
Added a regression test for this no-active-context variant specifically,
confirmed it fails against the prior commit and passes now.
Also fixes an unrelated basedpyright regression from an earlier rebase's
conflict resolution: litellm/utils.py's `logging_obj` was incorrectly
re-annotated `Final` at its second assignment in function_setup() (it's first
declared `None` a few lines earlier), which basedpyright correctly rejects.
* fix(proxy): stop logging the raw W3C baggage session_id value
_session_id_from_baggage() extracts the caller-controlled session.id entry
verbatim - it isn't sanitized until set_session_id() runs later in
Logging.__init__(). The debug log line for this extraction interpolated the
raw value directly, so a caller could embed terminal control characters or
ANSI escape sequences that forge/alter plaintext log output for anyone
tailing the proxy's logs.
Verified live: a baggage header with an embedded ANSI escape reached the
terminal as a real, unescaped control sequence before this fix. Drops the
value from the log line entirely (the extraction succeeding is enough signal
on its own) rather than sanitizing-then-logging, matching veria-ai's
suggestion. Added a regression test using caplog that fails against the prior
commit and passes now.
* fix(logging): restore consumer context only after stream-failure exception mapping
_map_anthropic_exception/_map_aleph_alpha_exception synchronously log a debug
diagnostic (the raw status code) as part of exception_type()'s mapping.
_handle_stream_fallback_error restored the consumer's outer correlation
context before calling exception_type(), so that diagnostic log line carried
the outer (or empty) trace_id/session_id instead of the failing stream's own -
flagged by Greptile.
Moved the restore to run after mapping completes, matching the same
restore-after-not-before pattern already applied elsewhere in this file for
success/finish_reason handling. Added a regression test that captures the
correlation context live during a mocked exception_type() call; fails against
the prior commit, passes now.
* fix(logging): restore consumer context only after aclose()'s stream close completes
aclose() restored the consumer's outer correlation context as its first
statement, before awaiting the underlying provider stream's own aclose()/
close(). If that close attempt raises, the except branch's debug diagnostic
ran under the already-restored outer context instead of the closing stream's
own trace_id/session_id - flagged by Greptile, same restore-too-early pattern
as the stream-failure fix in f1cf9589d6.
Moved the restore to the end of aclose(), after the close attempt (and its
diagnostic logging) completes. Added a regression test with a fake stream
whose aclose() raises, capturing the correlation context live during the
diagnostic log call; fails against the prior commit, passes now.
* style(logging): satisfy new strict-lint budgets introduced upstream (Final, ANN401, S110, TRY300, kwargs typing)
Rebasing onto litellm_internal_staging pulled in 116 upstream commits that
introduced/tightened several lint gates this PR's own code now trips:
- LIT010 (every local/module-level variable must be Final): added Final
annotations across _logging.py, litellm_logging.py, streaming_handler.py,
litellm_pre_call_utils.py, and utils.py. Where a name is genuinely
reassigned (logging_obj: starts None, later set to the real object) or
branch-assigned, either restructured into a single ternary expression
(ordered_candidates) or suppressed with `# rebind-ok: <reason>` matching
this repo's documented escape hatch.
- LIT011 (parameter mutation): suppressed the two new `data[key] = value`
writes in litellm_pre_call_utils.py with `# rebind-ok`, matching the
unsuppressed precedent already used for every other `data[...]` write in
the same function - `data` is an intentional out-param there.
- ANN001/ANN003/ANN202 (missing parameter/return type annotations): fully
typed success_handler/_success_handler_body, their async twins, and
failure_handler/_failure_handler_body/async variants in litellm_logging.py,
plus function_setup in utils.py (added Rules to its existing TYPE_CHECKING
block for the rules_obj: Rules annotation).
- ANN401 (explicit Any disallowed): suppressed with `# noqa: ANN401` on the
handful of genuinely-heterogeneous result/*args/**kwargs parameters, since
ordinary suppression is this repo's documented path.
- S110 (try/except/pass): added to the existing BLE001 noqa on the one
best-effort correlation-cleanup try/except this PR added.
- TRY300 (return inside try): moved two `return result` statements into
`else:` blocks in the retry-fallback paths this PR's own diff touched.
- reportPrivateUsage (basedpyright): renamed the two new
StandardLoggingPayloadSetup static methods (get_standard_logging_payload_
trace_id/session_id) to drop their leading underscore, since they're
genuinely called from a sibling module-level function in the same file.
No behavior change - confirmed by the full existing test suite (819 passed)
plus all four lint gates (ruff format, ruff-strict, type-discipline,
basedpyright) passing clean.
* fix(lint): stop RUF100 flagging noqa suppressions the strict gate needs
CI's plain "ruff check" job uses the default ruff.toml, a narrower config
than ruff-strict.toml (used only by the strict-rule budget gate). ANN401 and
S110 aren't enabled in the default config, so RUF100 (unused-noqa) flagged
the `# noqa: ANN401`/`# noqa: ...,S110` suppressions this PR added as pointless
under that config, even though they're genuinely needed under ruff-strict.toml.
- ANN401: added to ruff.toml's existing `lint.external` list (same mechanism
already used for C901/TID251, enforced by the strict gate but not by this
config) - these Any usages are genuinely dynamic/forwarded, so the
suppression itself is correct and just needed registering.
- S110: fixed the underlying code instead of registering another external
code - the try/except/pass in
CustomStreamWrapper._restore_consumer_correlation_context now logs at
debug level on failure (matching the existing best-effort-cleanup pattern
in _record_partial_usage_for_failure elsewhere in this file), which
satisfies S110's own suggestion directly and needs no suppression at all.
Verified against both ruff.toml and ruff-strict.toml directly, plus all
three other gates (ruff format, type-discipline, basedpyright) and the full
test suite (821 passed).
* fix(lint): scope the ANN401 exemption to file level instead of a repo-wide noqa
Ruff has no per-line-scoped way to register a noqa code across configs (that
requires the default ruff.toml's lint.external list, which is repo-wide in
scope even though the noqa itself is per-line). Since ruff does support
file-level exemptions via per-file-ignores, and ANN401 only needed exempting
in exactly two files, moved the exemption there instead:
- ruff-strict.toml: added [lint.per-file-ignores] disabling ANN401 for
litellm_logging.py and utils.py specifically, with a comment explaining
why (heterogeneous response/forwarded-args parameters with no fitting
concrete type - already verified by trying CostResponseTypes and hitting
a real basedpyright mismatch).
- ruff.toml: reverted the ANN401 entry from lint.external - no longer
needed, since there's no `# noqa: ANN401` left anywhere for RUF100 to
second-guess.
- Removed the now-redundant `# noqa: ANN401` from the 10 affected
parameters in both files, keeping the existing kwargs-ok reasons and
adding a short inline comment on the `result`/`*args` lines pointing at
the ruff-strict.toml exemption for context.
Verified against both configs directly (ANN401 clean under ruff-strict.toml
for these files, RUF100 clean under the default config), all four gates
(ruff format, ruff-strict, type-discipline, basedpyright), and the full
test suite (821 passed).
* fix(logging): redact credential-shaped trace_id/session_id before stamping log records
CorrelationContextFilter stamps trace_id/session_id onto a LogRecord after
SecretRedactionFilter has already run, so a caller-controlled value (e.g. via
x-litellm-trace-id or a W3C baggage header) that happens to look like a real
credential reached JSON and plaintext logs unredacted. Apply the same
credential redaction already used elsewhere in this module at
_sanitize_correlation_id(), the single choke point both set_trace_id() and
set_session_id() route through, so every caller-facing entry point is covered
without depending on filter ordering.
* fix(logging): restore correlation context when a stream's max-duration timeout fires
CustomStreamWrapper.__anext__() called _check_max_streaming_duration() before
entering its try block, so the litellm.Timeout it raises bypassed the except
Exception -> _handle_stream_fallback_error path entirely, leaking the timed-out
stream's own trace_id/session_id into whatever the consumer's task logs next.
Move the check inside the try so it flows through the same restoration path
every other stream failure already uses.
* test(streaming): make dispatch_failure_handlers mock awaitable for the async max-duration test
Moving _check_max_streaming_duration() inside __anext__()'s try block (prior
commit) means a max-duration Timeout now dispatches failure handlers through
the same path every other stream failure already uses, instead of bypassing
it entirely. dispatch_failure_handlers is async on the real Logging class;
the test's plain MagicMock logging_obj made asyncio.create_task() choke on a
non-coroutine return value once that path actually got exercised.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
|
||
|
Yassin Kortam
|
7984f4fa64
|
fix(logging): extend secret redaction to records litellm does not emit directly (#35977)
The redaction filter was attached to the handler shared by litellm's own loggers, so it only covered records litellm emits. A litellm value can also reach a log record through a dependency logging on its own logger, and those records never pass through a litellm handler. Attach the filter to each dependency logger that can carry one. The filter goes on the emitting logger rather than on the root logger or a root handler, since Logger.handle applies the emitting logger's filters before any handler runs, so every downstream handler is covered regardless of who owns it. |
||
|
mateo-berri
|
2708620d6a | feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) | ||
|
mateo-berri
|
b604e2b20c |
refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007 modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and friends), removal of the 1,296 typing imports the rewrite orphaned, and hand fixes for what the fixers could not see: five star-import freeloaders of typing names, two F823 late-import annotations, the /get/config/list introspection crash on types.UnionType, redundant function-local RoleMappings imports in ui_sso.py that shadowed the module-level name once the annotation lost its quotes, and one FURB168 tautology. B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms basedpyright then rejects (283 new errors measured), so their budgets stay at base values. ruff-strict-budget.json drops by 39,579 this commit (39,968 across the branch) with 28 rules at an actual 0 and 9 more sharply down. type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001 moves to the now-honest total: the checker matches the spelling `set` but not the alias `Set`, so the 160 typing.Set annotations rewritten to set[...] were always mutable-set annotations and only now count. |
||
|
Mateo Wang
|
48b5a5a0cc
|
style: unify ruff format width on 120 (#31518)
The repo linted at 120 (E501, isort) but ran ruff format at 88 via a --line-length 88 override in the Makefile and CI, leaving the formatter and the linter disagreeing on wrap width. Drop the override so ruff.toml's line-length = 120 is the single source of truth and reformat the tree to match. |
||
|
Mateo Wang
|
9c775e7304
|
ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516)
* ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default), RUF009 (function-call-in-dataclass-default-argument), and RUF100 (unused-noqa) in ruff.toml, and clean up every resulting violation. RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or blanket noqas: blanket `# noqa` are now scoped to the rule they actually suppress (mostly T201), dead directives are removed, and inapplicable codes are trimmed (e.g. F401 dropped from `import *`). lint.external lists rules enforced outside this config (the strict-rule gate via ruff-strict.toml and upstream litellm's own ruff config) so RUF100 keeps the noqa directives that protect them instead of stripping coverage this config can't see. * ci(lint): trim RUF100 external list to load-bearing codes only Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006, PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in the gated source. Keep only the 11 codes with live suppressions so RUF100 doesn't flag them as unused. Future strict-gate suppressions can re-add codes here (or fix the underlying issue) as needed. |
||
|
ishaan-berri
|
f9ba70d357
|
fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpo… (#27976)
* fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint (#27943) * docs: add one-line docstring to _disable_debugging (#27894) Squash-merged by litellm-agent from oss-agent-shin's PR. * Add jp. Bedrock cross-region inference profile for claude-sonnet-4-6 (#27831) Squash-merged by litellm-agent from Cyberfilo's PR. * Sanitize empty text content blocks on /v1/messages (#27832) Squash-merged by litellm-agent from Cyberfilo's PR. * fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpoint The bedrock-mantle gateway (Claude Mythos Preview) serves the Anthropic Messages API at /anthropic/v1/messages; /v1/messages returns 404 Not Found. Both AmazonMantleConfig (chat/completions caller route) and AmazonMantleMessagesConfig (anthropic-messages caller route) hardcoded the wrong path, so every Mantle request 404'd before reaching the model. Per the Anthropic docs: "[Claude in Amazon Bedrock] uses the Messages API at /anthropic/v1/messages with SSE streaming." https://platform.claude.com/docs/en/api/claude-on-amazon-bedrock Confirmed independently against the live endpoint: /v1/chat/completions -> 200 OK /v1/messages -> 404 Not Found (what litellm used) /anthropic/v1/messages -> 200 OK (Claude only) Adds a regression test asserting both Mantle configs build the /anthropic/v1/messages path, and updates the existing assertions that encoded the wrong path. --------- Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> * fix: sanitize empty text blocks in sync anthropic_messages_handler path Co-authored-by: Yassin Kortam <yassin@berri.ai> --------- Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: oss-agent-shin <ext-agent-shin@berri.ai> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Yassin Kortam <yassin@berri.ai> |
||
|
Cursor Agent
|
97cfb4f6fd
|
Unify secret redaction patterns | ||
|
Sameer Kankute
|
3eccededcc
|
Fix black | ||
|
Sameer Kankute
|
b3fdb5cc69
|
Fix import error | ||
|
Sameer Kankute
|
b0ac74c556
|
Fix import error | ||
|
Vedanshu Joshi
|
f46074664e
|
fix(llm translation): redact Gemini API key from URL query params in error traces (#24943)
* fix(proxy): use actual request start_time for failed spend logs async_post_call_failure_hook was calling datetime.now() for both start_time and end_time, making every failed request show Duration: 0.000s. litellm_logging_obj (already fetched in the same method for trace ID propagation) carries the real request start_time — use it as actual_start_time with a datetime.now() fallback when absent. Add two regression tests covering the fix and the fallback path. Fixes #24888 * fix(llm translation): redact Gemini API key from URL query params in error traces Gemini API requests authenticate via a ?key=<api_key> URL query param. When a provider call fails, httpx.Response.raise_for_status() embeds the full URL in the error message, leaking the key in exception traces and logs. Changes: - Extract secret-redaction logic from litellm/_logging.py into a new public utility module litellm/litellm_core_utils/secret_redaction.py, exposing redact_string() as a proper public API instead of a private helper - Add (?<=[?&])key=[^\s&'"]{8,} pattern to _SECRET_RE so ?key=VALUE and &key=VALUE fragments are caught by the existing SecretRedactionFilter - Apply redact_string() to error_str in exception_mapping_utils.py so the key is also stripped from the mapped exception message surfaced to callers - Add 5 regression tests covering: ?key=, &key=, short-value no-op, httpx raise_for_status path, and end-to-end logger output - Keep _redact_string = redact_string alias in _logging.py for backward compat Fixes #24902 * revert: undo start_time fix for failed spend logs * fix: gate exception redaction on _ENABLE_SECRET_REDACTION opt-out flag - Apply redact_string() conditionally in exception_mapping_utils.py, matching the same _ENABLE_SECRET_REDACTION guard used by SecretRedactionFilter so that LITELLM_DISABLE_REDACT_SECRETS=true is honoured for exception messages - Rewrite test_redact_string_applied_to_httpx_error_message to use pytest.raises so assertions cannot be silently skipped if raise_for_status() doesn't raise - Add test_exception_mapping_respects_redaction_opt_out to verify the flag is respected end-to-end through exception_type() |
||
|
user
|
25f93bed91
|
security: prevent API key leaks in error tracebacks, logs, and alerts
Gemini API keys embedded in URLs as ?key= query parameters leak through
httpx error tracebacks, which are then captured by traceback.format_exc()
and forwarded to logging callbacks, Slack/Teams alerts, and HTTP client
responses.
Short-term: all httpx.HTTPStatusError handlers now raise
MaskedHTTPStatusError(...) from None, which masks the URL and breaks
exception chaining so the original error never appears in tracebacks.
Long-term: moved all Gemini/Vertex URL constructions from ?key={api_key}
to x-goog-api-key header (Google's documented auth method), so the key
is never in the URL at all. WebSocket realtime is the only exception
since WS clients cannot use custom headers.
Additionally hardened all outbound credential paths:
- WebSocket close reasons now pass through _redact_string()
- Callback pipeline (failure_handler) redacts traceback_exception and
error_str before forwarding to integrations (Langfuse, Datadog, etc.)
- Slack/Teams alert messages redacted in send_llm_exception_alert,
ProxyLogging.failure_handler, and post_call_failure_hook
- HTTP error responses in proxy SSE and health endpoints redacted
- Exception messages in exception_mapping_utils redacted
- print_verbose() stdout output redacted when set_verbose=True
- HTTPHandler.put() now has MaskedHTTPStatusError (was missing)
|
||
|
Krrish Dholakia
|
f42ffed2bd
|
Litellm oss staging 04 02 2026 p1 (#25055)
* fix(vertex_ai): support pluggable (executable) credential_source for WIF auth (#24700) The WIF credential dispatch in load_auth() only handled identity_pool and aws credential types. When credential_source.executable was present (used for Azure Managed Identity via Workload Identity Federation), it fell through to identity_pool.Credentials which rejected it with MalformedError. Add dispatch to google.auth.pluggable.Credentials for executable-type credential sources, following the same pattern as the existing identity_pool and aws helpers. Fixes authentication for Azure Container Apps → GCP Vertex AI via WIF with executable credential sources. * feat(logging): add component and logger fields to JSON logs for 3rd p… (#24447) * feat(logging): add component and logger fields to JSON logs for 3rd party filtering * Let user-supplied extra fields win over auto-generated component/logger, tighten test assertions * Feat - Add organization into the metrics metadata for org_id & org_alias (#24440) * Add org_id and org_alias label names to Prometheus metric definitions * Add user_api_key_org_alias to StandardLoggingUserAPIKeyMetadata * Populate user_api_key_org_alias in pre-call metadata * Pass org_id and org_alias into per-request Prometheus metric labels * Add test for org labels on per-request Prometheus metrics * chore: resolve test mockdata * Address review: populate org_alias from DB view, add feature flag, use .get() for org metadata * Add org labels to failure path and verify flag behavior in test * Fix test: build flag-off enum_values without org fields * Gate org labels behind feature flag in get_labels() instead of static metric lists * Scope org label injection to metrics that carry team context, remove orphaned budget label defs, add test teardown * Use explicit metric allowlist for org label injection instead of team heuristic * Fix duplicate org label guard, move _org_label_metrics to class constant * Reset custom_prometheus_metadata_labels after duplicate label assertion * fix: emit org labels by default, remove flag, fix missing org_alias in all metadata paths * fix: emit org labels by default, no opt-in flag required * fix: write org_alias to metadata unconditionally in proxy_server.py * fix: 429s from batch creation being converted to 500 (#24703) * add us gov models (#24660) * add us gov models * added max tokens * Litellm dev 04 02 2026 p1 (#25052) * fix: replace hardcoded url * fix: Anthropic web search cost not tracked for Chat Completions The ModelResponse branch in response_object_includes_web_search_call() only checked url_citation annotations and prompt_tokens_details, missing Anthropic's server_tool_use.web_search_requests field. This caused _handle_web_search_cost() to never fire for Anthropic Claude models. Also routes vertex_ai/claude-* models to the Anthropic cost calculator instead of the Gemini one, since Claude on Vertex uses the same server_tool_use billing structure as the direct Anthropic API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * fix(anthropic): pass logging_obj to client.post for litellm_overhead_time_ms (#24071) When LITELLM_DETAILED_TIMING=true, litellm_overhead_time_ms was null for Anthropic because the handler did not pass logging_obj to client.post(), so track_llm_api_timing could not set llm_api_duration_ms. Pass logging_obj=logging_obj at all four post() call sites (make_call, make_sync_call, acompletion, completion). Add test to ensure make_call passes logging_obj to client.post. Made-with: Cursor * sap - add additional parameters for grounding - additional parameter for grounding added for the sap provider * sap - fix models * (sap) add filtering, masking, translation SAP GEN AI Hub modules * (sap) add tests and docs for new SAP modules * (sap) add support of multiple modules config * (sap) code refactoring * (sap) rename file * test(): add safeguard tests * (sap) update tests * (sap) update docs, solve merge conflict in transformation.py * (sap) linter fix * (sap) Align embedding request transformation with current API * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) mock commit * (sap) run black formater * (sap) add literals to models, add negative tests, fix test for tool transformation * (sap) fix formating * (sap) fix models * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) commit for rerun bot review * (sap) minor improve * (sap) fix after bot review * (sap) lint fix * docs(sap): update documentation * fix(sap): change creds priority * fix(sap): change creds priority * fix(sap): fix sap creds unit test * fix(sap): linter fix * fix(sap): linter fix * linter fix * (sap) update logic of fetching creds, add additional tests * (sap) clean up code * (sap) fix after review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) add a possibility to put the service key by both variants * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) update test * (sap) update service key resolve function * (sap) run black formater * (sap) fix validate credentials, add negative tests for credential fetching * (sap) fix validate credentials, add negative tests for credential fetching * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) fix after bot review * (sap) lint fix * (sap) lint fix * feat: support service_tier in gemini * chore: add a service_tier field mapping from openai to gemini * fix: use x-gemini-service-tier header in response * docs: add service_tier to gemini docs * chore: add defaut/standard mapping, and some tests * chore: tidying up some case insensitivity * chore: remove unnecessary guard * fix: remove redundant test file * fix: handle 'auto' case-insensitively * fix: return service_tier on final steamed chunk * chore: black * feat: enable supports_service_tier to gemini models * Fix get_standard_logging_metadata tests * Fix test_get_model_info_bedrock_models * Fix test_get_model_info_bedrock_models * Fix remaining tests * Fix mypy issues * Fix tests * Fix merge conflicts * Fix code qa * Fix code qa * Fix code qa * Fix greptile review --------- Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: Josh <36064836+J-Byron@users.noreply.github.com> Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: milan-berri <milan@berri.ai> Co-authored-by: Alperen Kömürcü <alperen.koemuercue@sap.com> Co-authored-by: Vasilisa Parshikova <vasilisa.parshikova@sap.com> Co-authored-by: Lin Xu <lin.xu03@sap.com> Co-authored-by: Mark McDonald <macd@google.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> |
||
|
ishaan-berri
|
b53cfe729a
|
Litellm ishaan march30 (#24887) (#25151)
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry Missing unversioned entry causes cost tracking to return $0.00 for all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI Claude models have both versioned and unversioned entries. * fix(router): skip misleading tags error when no candidates (e.g. cooldown) Return early from get_deployments_for_tag when healthy_deployments is empty so tag-based routing does not raise no_deployments_with_tag_routing after cooldown filters all deployments. Adds regression test. Made-with: Cursor * feat(oci): add embedding support and update model catalog - Add OCIEmbeddingConfig for OCI GenAI embedding models - Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini) - Add 8 embedding models (Cohere embed v3.0, v4.0) - Update documentation with embedding examples - Update pricing for all new models * test(oci): add unit tests for OCI embedding support - 17 unit tests covering OCIEmbeddingConfig - Tests for URL generation, param mapping, request/response transform - Tests for model pricing JSON completeness * style(oci): format with black and ruff * fix(oci): correct embedding request body format OCI embedText API expects inputs, truncate, and inputType at the top level of the request body, not nested under embedTextDetails. Fixed transformation and updated tests accordingly. Verified with real OCI API: 3/3 embedding models working. * docs: clarify tag routing early return and test intent Made-with: Cursor * fix(oci): address code review findings from Greptile - P1: Fix signing URL mismatch with custom api_base by accepting api_base parameter in transform_embedding_request - P2: Remove encoding_format from supported params (OCI does not support it, was silently dropped) - P2: Raise ValueError for token-array inputs instead of silently converting to string representation - Add test for token-list rejection * fix(mcp): add STS AssumeRole support for MCP SigV4 authentication MCPSigV4Auth only supported static AWS credentials or the boto3 default credential chain. Production Kubernetes environments typically authenticate via IAM role assumption (sts:AssumeRole), which was not possible. Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole to obtain temporary credentials before signing requests. Explicit keys, if also provided, are used as the source identity for the STS call; otherwise ambient credentials (pod role, instance profile) are used. * fix: stop logging credential values and add missing redaction patterns Replaces raw credential values in debug/error log messages with boolean presence checks or type names. Adds PEM block, GCP token, JWT, SAS token, and service-account blob patterns to the redaction filter. Fixes private_key pattern to capture full PEM blocks instead of stopping at the first whitespace. Addresses: Vertex AI credential JSON (including RSA private key) being logged to stderr on health check failures. * fix: log only field names for UserAPIKeyAuth, not full object * style: apply black formatting to experimental_mcp_client/client.py * style: fix black/isort formatting and mypy error in proxy_server.py - Fix black formatting in experimental_mcp_client/client.py (done in prev commit) - Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py - Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py * fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue When test_proxy_cli.py tests run before test_check_migration.py in the same xdist worker, litellm.proxy.db.check_migration is already in sys.modules. Patching litellm._logging.verbose_logger has no effect on the already-bound reference. Patch the correct target (check_migration.verbose_logger) and import the module before patching so the order doesn't matter. * fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature --------- Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com> Co-authored-by: Milan <milan@berri.ai> Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: michelligabriele <gabriele.michelli@icloud.com> Co-authored-by: user <70670632+stuxf@users.noreply.github.com> Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com> |
||
|
Ryan Crabbe
|
031b3d8198 |
fix: add key-name-based secret redaction to catch secrets in config dict dumps
Add key-name-based regex patterns (master_key, database_url, auth_token, etc.) to SecretRedactionFilter so secrets embedded in dict/config dumps are redacted by key name, regardless of value format. Fixes a leak where general_settings containing master_key and database_url was logged in full because the secret values didn't match any existing value-format regex pattern. |
||
|
Ishaan Jaff
|
8e61b32b8e
|
[Staging] - Ishaan March 17th (#23903)
* feat(xai): add grok-4.20 beta 2 models with pricing (#23900)
Add three grok-4.20 beta 2 model variants from xAI:
- grok-4.20-multi-agent-beta-0309 (reasoning + multi-agent)
- grok-4.20-beta-0309-reasoning (reasoning)
- grok-4.20-beta-0309-non-reasoning
Pricing (from https://docs.x.ai/docs/models):
- Input: $2.00/1M tokens ($0.20/1M cached)
- Output: $6.00/1M tokens
- Context: 2M tokens
All variants support vision, function calling, tool choice, and web search.
Closes LIT-2171
* docs: add Quick Install section for litellm --setup wizard (#23905)
* docs: add Quick Install section for litellm --setup wizard
* docs: clarify setup wizard is for local/beginner use
* feat(setup): interactive setup wizard + install.sh (#23644)
* feat(setup): add interactive setup wizard + install.sh
Adds `litellm --setup` — a Claude Code-style TUI onboarding wizard that
guides users through provider selection, API key entry, and proxy config
generation, then optionally starts the proxy immediately.
- litellm/setup_wizard.py: wizard with ASCII art, numbered provider menu
(OpenAI, Anthropic, Azure, Gemini, Bedrock, Ollama), API key prompts,
port/master-key config, and litellm_config.yaml generation
- litellm/proxy/proxy_cli.py: adds --setup flag that invokes the wizard
- scripts/install.sh: curl-installable script (detect OS/Python, pip
install litellm[proxy], launch wizard)
Usage:
curl -fsSL https://raw.githubusercontent.com/BerriAI/litellm/main/scripts/install.sh | sh
litellm --setup
* fix(install.sh): remove orange color, add LITELLM_BRANCH env var for branch installs
* fix(install.sh): install from git branch so --setup is available for QA
* fix(install.sh): remove stale LITELLM_BRANCH reference that caused unbound variable error
* fix(install.sh): force-reinstall from git to bypass cached PyPI version
* fix(install.sh): show pip progress bar during install
* fix(install.sh): always launch wizard via $PYTHON_BIN -m litellm, not PATH binary
* fix(install.sh): use litellm.proxy.proxy_cli module (no __main__.py exists)
* fix(install.sh): suppress RuntimeWarning from module invocation
* fix(install.sh): use Python bin-dir litellm binary to avoid CWD sys.path shadowing
* fix(install.sh): use sysconfig.get_path('scripts') to find pip-installed litellm binary
* fix(install.sh): redirect stdin from /dev/tty on exec so wizard gets terminal, not exhausted pipe
* fix(install.sh): warn about git clone duration, drop --no-cache-dir so re-runs are faster
* feat(setup_wizard): arrow-key selector, updated model names
* fix(setup_wizard): use sysconfig binary to start proxy, not python -m litellm
* feat(setup_wizard): credential validation after key entry + clear next-steps after proxy start
* style(install.sh): show git clone warning in blue
* refactor(setup_wizard): class with static methods, use check_valid_key from litellm.utils
* address greptile review: fix yaml escaping, port validation, display name collisions, tests
- setup_wizard.py: add _yaml_escape() for safe YAML embedding of API keys
- setup_wizard.py: add _styled_input() with readline ANSI ignore markers
- setup_wizard.py: change DIVIDER to _divider() fn to avoid import-time color capture
- setup_wizard.py: validate port range 1-65535, initialize before loop
- setup_wizard.py: qualify azure display names (azure-gpt-4o) to avoid collision with openai
- setup_wizard.py: work on env_copy in _build_config to avoid mutating caller's dict
- setup_wizard.py: skip model_list entries for providers with no credentials
- setup_wizard.py: prompt for azure deployment name
- setup_wizard.py: wrap os.execlp in try/except with friendly fallback
- setup_wizard.py: wrap config write in try/except OSError
- setup_wizard.py: fix _validate_and_report to use two print lines (no \r overwrite)
- setup_wizard.py: add .gitignore tip next to key storage notice
- setup_wizard.py: fix run_setup_wizard() return type annotation to None
- scripts/install.sh: drop pipefail (not supported by dash on Ubuntu when invoked as sh)
- scripts/install.sh: use litellm[proxy] from PyPI (not hardcoded dev branch)
- scripts/install.sh: guard /dev/tty read with -r check for Docker/CI compat
- scripts/install.sh: remove --force-reinstall to avoid downgrading dependencies
- tests/test_litellm/test_setup_wizard.py: 13 unit tests for _build_config and _yaml_escape
* style: black format setup_wizard.py
* fix: address remaining greptile issues - Windows compat, YAML quoting, credential flow
- guard termios/tty imports with try/except ImportError for Windows compat
- quote master_key as YAML double-quoted scalar (same as env vars)
- remove unused port param from _build_config signature
- _validate_and_report now returns the final key so re-entered creds are stored
- add test for master_key YAML quoting
* fix: add --port to suggested command, guard /dev/tty exec in install.sh
* fix: quote api_base in YAML, skip azure if no deployment, only redraw on state change
* fix: address greptile review comments
- _yaml_escape: add control character escaping (\n, \r, \t)
- test: fix tautological assertion in test_build_config_azure_no_deployment_skipped
- test: add tests for control character escaping in _yaml_escape
* feat(ui): remove Chat UI page link and banner from sidebar and playground (#23908)
* feat(guardrails): MCPJWTSigner - built-in guardrail for zero trust MCP auth (#23897)
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* Allow pre_mcp_call guardrail hooks to mutate outbound MCP headers
* Enhance MCPServerManager to support hook-modified arguments and extra headers. Update tests to validate argument mutation and header injection behavior, including warnings for OpenAPI-backed servers when headers are present.
* Refactor MCPServerManager to raise HTTPException for extra headers in OpenAPI-backed servers. Update tests to reflect this change, ensuring proper exception handling instead of logging warnings.
* feat(guardrails): add MCPJWTSigner built-in guardrail for zero trust MCP auth
Signs outbound MCP tool calls with a LiteLLM-issued RS256 JWT so MCP servers
can trust a single signing authority instead of every upstream IdP.
Enable in config.yaml:
guardrails:
- guardrail_name: mcp-jwt-signer
litellm_params:
guardrail: mcp_jwt_signer
mode: pre_mcp_call
default_on: true
JWT carries sub (user_id), act.sub (team_id, RFC 8693), tool-level scope, iss,
aud, iat/exp/nbf. RSA-2048 keypair auto-generated at startup unless
MCP_JWT_SIGNING_KEY env var is set.
Adds /.well-known/jwks.json endpoint and jwks_uri to /.well-known/openid-configuration
so MCP servers can verify LiteLLM-issued tokens via OIDC discovery.
* Update MCPServerManager to raise HTTPException with status code 400 for extra headers in OpenAPI-backed servers. Adjust tests to verify the correct status code and exception message.
* fix: address P1 issues in MCPJWTSigner
- OpenAPI servers: warn + skip header injection instead of 500
- JWKS Cache-Control: 5min for auto-generated keys, 1h for persistent
- sub claim: fallback to apikey:{token_hash} for anonymous callers
- ttl_seconds: validate > 0 at init time
* docs: add MCP zero trust auth guide with architecture diagram
* docs: add FastMCP JWT verification guide to zero trust doc
* fix: address remaining Greptile review issues (round 2)
- mcp_server_manager: warn when hook Authorization overwrites existing header
- __init__: remove _mcp_jwt_signer_instance from __all__ (private internal)
- discoverable_endpoints: copy dict instead of mutating in-place on OIDC augmentation
- test docstring: reflect warn-and-continue behavior for OpenAPI servers
- test: update scope assertions for least-privilege (no mcp:tools/list on tool-call JWTs)
* fix: address Greptile round 3 feedback
- initialize_guardrail: validate mode='pre_mcp_call' at init time — misconfigured
mode silently bypasses JWT injection, which is a zero-trust bypass
- _build_claims: remove duplicate inline 'import re' (module-level import already present)
- _types.py: add TODO comment explaining jwt_claims is forward-compat plumbing
for a follow-up PR that will forward upstream IdP claims into outbound MCP JWTs
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes
Addresses all missing pieces from the scoping doc review:
FR-5 (Verify + re-sign): MCPJWTSigner now accepts access_token_discovery_uri
and token_introspection_endpoint. When set, the incoming Bearer token is
extracted from raw_headers (threaded through pre_call_tool_check), verified
against the IdP's JWKS (JWT) or introspected (opaque), and only re-signed if
valid. Falls back to user_api_key_dict.jwt_claims for LiteLLM JWT-auth mode.
FR-12 (Configurable end-user identity mapping): end_user_claim_sources
ordered list drives sub resolution — sources: token:<claim>, litellm:user_id,
litellm:email, litellm:end_user_id, litellm:team_id.
FR-13 (Claim operations): add_claims (insert-if-absent), set_claims (always
override), remove_claims (delete) applied in that order.
FR-14 (Two-token model): channel_token_audience + channel_token_ttl issue a
second JWT injected as x-mcp-channel-token: Bearer <token>.
FR-15 (Incoming claim validation): required_claims raises HTTP 403 when any
listed claim is absent; optional_claims passes listed claims from verified
token into the outbound JWT.
FR-9 (Debug headers): debug_headers: true emits x-litellm-mcp-debug with kid,
sub, iss, exp, scope.
FR-10 (Configurable scopes): allowed_scopes replaces auto-generation. Also
fixed: tool-call JWTs no longer grant mcp:tools/list (overpermission).
P1 fixes:
- proxy/utils.py: _convert_mcp_hook_response_to_kwargs merges rather than
replaces extra_headers, preserving headers from prior guardrails.
- mcp_server_manager.py: warns when hook injects Authorization alongside a
server-configured authentication_token (previously silent).
- mcp_server_manager.py: pre_call_tool_check now accepts raw_headers and
extracts incoming_bearer_token so FR-5 verification has the raw token.
- proxy/utils.py: remove stray inline import inspect inside loop (pre-existing
lint error, now cleaned up).
Tests: 43 passing (28 new tests covering all FR flags + P1 fixes).
* feat(mcp_jwt_signer): add verify+re-sign, claim ops, two-token model, configurable scopes (core)
Remaining files from the FR implementation:
mcp_jwt_signer.py — full rewrite with all new params:
FR-5: access_token_discovery_uri, token_introspection_endpoint,
verify_issuer, verify_audience + _verify_incoming_jwt(),
_introspect_opaque_token()
FR-12: end_user_claim_sources ordered resolution chain
FR-13: add_claims, set_claims, remove_claims
FR-14: channel_token_audience, channel_token_ttl → x-mcp-channel-token
FR-15: required_claims (raises 403), optional_claims (passthrough)
FR-9: debug_headers → x-litellm-mcp-debug
FR-10: allowed_scopes; tool-call JWTs no longer over-grant tools/list
mcp_server_manager.py:
- pre_call_tool_check gains raw_headers param to extract incoming_bearer_token
- Silent Authorization override warning fixed: now fires when server has
authentication_token AND hook injects Authorization
tests/test_mcp_jwt_signer.py:
28 new tests covering all FR flags + P1 fixes (43 total, all passing)
* fix(mcp_jwt_signer): address pre-landing review issues
- Remove stale TODO comment on UserAPIKeyAuth.jwt_claims — the field is
already populated and consumed by MCPJWTSigner in the same PR
- Fix _get_oidc_discovery to only cache the OIDC discovery doc when
jwks_uri is present; a malformed/empty doc now retries on the next
request instead of being permanently cached until proxy restart
- Add FR-5 test coverage for _fetch_jwks (cache hit/miss),
_get_oidc_discovery (cache/no-cache on bad doc), _verify_incoming_jwt
(valid token, expired token), _introspect_opaque_token (active,
inactive, no endpoint), and the end-to-end 401 hook path — 53 tests
total, all passing
* docs(mcp_zero_trust): rewrite as use-case guide covering all new JWT signer features
Add scenario-driven sections for each new config area:
- Verify+re-sign with Okta/Azure AD (access_token_discovery_uri,
end_user_claim_sources, token_introspection_endpoint)
- Enforcing caller attributes with required_claims / optional_claims
- Adding metadata via add_claims / set_claims / remove_claims
- Two-token model for AWS Bedrock AgentCore Gateway
(channel_token_audience / channel_token_ttl)
- Controlling scopes with allowed_scopes
- Debugging JWT rejections with debug_headers
Update JWT claims table to reflect configurable sub (end_user_claim_sources)
* fix(mcp_jwt_signer): wire all config.yaml params through initialize_guardrail
The factory was only passing issuer/audience/ttl_seconds to MCPJWTSigner.
All FR-5/9/10/12/13/14/15 params (access_token_discovery_uri,
end_user_claim_sources, add/set/remove_claims, channel_token_audience,
required/optional_claims, debug_headers, allowed_scopes, etc.) were
silently dropped, making every advertised advanced feature non-functional
when loaded from config.yaml.
Add regression test that asserts every param is wired through correctly.
* docs(mcp_zero_trust): add hero image
* docs(mcp_zero_trust): apply Linear-style edits
- Lead with the problem (unsigned direct calls bypass access controls)
- Shorter statement section headers instead of question-form headers
- Move diagram/OIDC discovery block after the reader is bought in
- Add 'read further only if you need to' callout after basic setup
- Two-token section now opens from the user problem not product jargon
- Add concrete 403 error response example in required_claims section
- Debug section opens from the symptom (MCP server returning 401)
- Lowercase claims reference header for consistency
* fix(mcp_jwt_signer): fix algorithm confusion attack + add OIDC discovery 24h TTL
- Remove alg from unverified JWT header; use signing_jwk.algorithm_name from JWKS key instead.
Reading alg from attacker-controlled headers enables alg:none / HS256 confusion attacks.
- Add _oidc_discovery_fetched_at timestamp and _OIDC_DISCOVERY_TTL = 86400 (24h).
Without a TTL the cached discovery doc never refreshes, so IdP key rotation is invisible.
---------
Co-authored-by: Noah Nistler <60981020+noahnistler@users.noreply.github.com>
* fix(ci): stabilize CI - formatting, type errors, test polling, security CVEs, router bug, batch resolution
Fix 1: Run Black formatter on 35 files
Fix 2: Fix MyPy type errors:
- setup_wizard.py: add type annotation for 'selected' set variable
- user_api_key_auth.py: remove redundant type annotation on jwt_claims reassignment
Fix 3: Fix spend accuracy test burst 2 polling to wait for expected total
spend instead of just 'any increase' from burst 2
Fix 4: Bump Next.js 16.1.6 -> 16.1.7 to fix CVE-2026-27978, CVE-2026-27979,
CVE-2026-27980, CVE-2026-29057
Fix 5: Fix router _pre_call_checks model variable being overwritten inside
loop, causing wrong model lookups on subsequent deployments. Use local
_deployment_model variable instead.
Fix 6: Add missing resolve_output_file_ids_to_unified call in batch retrieve
non-terminal-to-terminal path (matching the terminal path behavior)
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* chore: regenerate poetry.lock to sync with pyproject.toml
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: format merged files from main and regenerate poetry.lock
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(mypy): annotate jwt_claims as Optional[dict] to fix type incompatibility
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(ci): update router region test to use gpt-4.1-mini (fix flaky model lookup)
Replace deprecated gpt-3.5-turbo-1106 with gpt-4.1-mini + mock_response in
test_router_region_pre_call_check, following the same pattern used in commit
|
||
|
Ryan Crabbe
|
47ddd0d0bf |
fix: redact secrets from proxy log output
Add SecretRedactionFilter to scrub API keys, tokens, and credentials from all log records (messages, args, tracebacks, extra fields). - Enable redaction by default; opt out with LITELLM_DISABLE_REDACT_SECRETS=true - Redact patterns: sk-*, Bearer tokens, x-api-key values, base64 creds - Handle JSON formatter exception hooks and percent-style format args - Snapshot dict iteration to avoid RuntimeError during concurrent logging |
||
|
Harshit Jain
|
dfbc0e2156 |
fix:Parse embedded JSON in the message field of logs (#20366)
* fix:Parse embedded JSON in the message field of logs * Update litellm/_logging.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> |
||
|
Alexsander Hamir
|
9cdd7a8fd2
|
Fix: log duplication when json_logs is enabled (#19705) | ||
|
Graham Neubig
|
17f8916ce3
|
fix(logging): Include langfuse logger in JSON logging when langfuse callback is used (#19162)
When JSON_LOGS is enabled and langfuse is configured as a success/failure callback, the langfuse logger now receives the JSON formatter. This ensures langfuse SDK log messages (like 'Item exceeds size limit' warnings) are output as JSON with proper level information, instead of plain text that log aggregators may incorrectly classify as errors. Fixes issue where langfuse warnings appeared as errors in Datadog due to missing log level in unformatted output. Co-authored-by: openhands <openhands@all-hands.dev> |
||
|
Ishaan Jaff
|
04dc1a5351
|
[Feat] Add support for returning images with gemini/gemini-2.5-flash-image-preview with /chat/completions (#13983)
* add gemini-2.5-flash-image-preview * add gemini-2.5-flash-image-preview * add image in ChatCompletionResponseMessage * test_gemini_image_generation_async * Revert "Merge pull request #13394 from Deviad/feature/enhance_logging_for_containers" This reverts commit |
||
|
Krish Dholakia
|
900bd10905
|
Merge branch 'main' into feature/enhance_logging_for_containers | ||
|
codeflash-ai[bot]
|
5dba5822f2
|
⚡️ Speed up function _is_debugging_on by 45% (#13988)
The optimization eliminates unnecessary conditional branching by replacing the explicit `if-else` structure with a direct return of the boolean expression. Instead of evaluating the condition and then branching to return `True` or `False`, the optimized version directly returns the result of the boolean expression `verbose_logger.isEnabledFor(logging.DEBUG) or set_verbose is True`. This change removes Python's conditional jump overhead and reduces the number of executed bytecode instructions per function call. The line profiler shows the original version required 3 lines of execution (condition check, conditional return True, fallback return False) while the optimized version executes only 1 line. The 45% speedup is achieved by: - **Eliminating branching overhead**: No conditional jumps needed - **Reducing bytecode instructions**: From ~3 instructions to 1 instruction per call - **Leveraging Python's short-circuit evaluation**: The `or` operator still evaluates left-to-right and stops early when the first condition is True The optimization is particularly effective for this logging utility function which is likely called frequently throughout the application. All test cases show consistent 40-75% improvements across different scenarios (debug on/off, verbose flag variations, edge cases with different logging levels), demonstrating the optimization works well regardless of the boolean expression's outcome. Co-authored-by: codeflash-ai[bot] <148906541+codeflash-ai[bot]@users.noreply.github.com> |
||
|
Davide Pugliese
|
81f2563338 | Enhance logging for containers | ||
|
Jugal D. Bhatt
|
8b2478e7d6
|
[Proxy server] Add apscheduler log suppress (#13299)
* Add log suppress * Add log suppress |
||
|
Jugal D. Bhatt
|
c125ae453b
|
suppress httpx logging (#13217) | ||
|
Ishaan Jaff
|
39955129f5
|
fix mapped tests (#12320)
* fix - use flush llm client cache
* faster mapped tests
* test_async_multiple_response_ids_routing
* fix tests
* test_ateam_member_update_admin_requires_premium
* regular mapped tests
* Revert "Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)"
This reverts commit
|
||
|
Cole McIntosh
|
2c60c316ec
|
Fix: Initialize JSON logging for all loggers when JSON_LOGS=True (#12206)
When JSON_LOGS=True is set, error logs were not being formatted as JSON despite the configuration. This was because the logging initialization code configured individual loggers but failed to properly initialize all loggers with the JSON formatter. This fix ensures that when json_logs is enabled, the _initialize_loggers_with_handler() function is called to: - Configure all loggers (root, LiteLLM, Router, Proxy) with JSON formatter - Disable logger propagation to prevent duplicate entries - Set up exception handlers for JSON formatting Fixes LIT-267 |
||
|
Ishaan Jaff
|
01b9e6efb1
|
Revert "fix: set logger levels based on LITELLM_LOG environment variable" (#12122) | ||
|
Cole McIntosh
|
9d58d872ac |
fix: set logger levels based on LITELLM_LOG environment variable
- Initialize verbose_logger, verbose_proxy_logger, and verbose_router_logger with the correct log level - Ensures loggers respect the LITELLM_LOG environment variable setting - Fixes part of issue #9815 where logs were shown regardless of log level setting Part 1/2 of fix for #9815 |
||
|
Ishaan Jaff
|
ae2a9cf4de | fix debug logs | ||
|
Ishaan Jaff
|
cd48ffb6c4
|
[Bug fix] JSON logs - Ensure only 1 log is emitted (previously duplicate json logs were getting emitted) (#10580)
* fix logging with JSON logs * test: test_initialize_loggers_with_handler_sets_propagate_false |
||
|
Ishaan Jaff
|
ddc24da2a7 |
Revert "fix logging with JSON logs"
This reverts commit
|
||
|
Ishaan Jaff
|
1109a6ea6e | fix logging with JSON logs | ||
|
Krish Dholakia
|
f3ff7e2779
|
fix(logging): add json formatting for uncaught exceptions (#9615) (#9619)
* fix(logging): add json formatting for uncaught exceptions (#9615) * fix(_logging.py): cleanup logging to catch unhandled exceptions * fix(_logging.py): avoid using 'print' ' --------- Co-authored-by: Henrique Cavarsan <hcavarsan@yahoo.com.br> |
||
|
Ishaan Jaff
|
75fb37217a
|
(sdk perf fix) - only print args passed to litellm when debugging mode is on (#7708)
* use _is_debugging_on * fix unused imports |
||
|
Ishaan Jaff
|
c7f14e936a
|
(code quality) run ruff rule to ban unused imports (#7313)
* remove unused imports * fix AmazonConverseConfig * fix test * fix import * ruff check fixes * test fixes * fix testing * fix imports |
||
|
Krish Dholakia
|
f79365df6e
|
LiteLLM Minor Fixes & Improvements (10/30/2024) (#6519)
* refactor: move gemini translation logic inside the transformation.py file easier to isolate the gemini translation logic * fix(gemini-transformation): support multiple tool calls in message body Merges https://github.com/BerriAI/litellm/pull/6487/files * test(test_vertex.py): add remaining tests from https://github.com/BerriAI/litellm/pull/6487 * fix(gemini-transformation): return tool calls for multiple tool calls * fix: support passing logprobs param for vertex + gemini * feat(vertex_ai): add logprobs support for gemini calls * fix(anthropic/chat/transformation.py): fix disable parallel tool use flag * fix: fix linting error * fix(_logging.py): log stacktrace information in json logs Closes https://github.com/BerriAI/litellm/issues/6497 * fix(utils.py): fix mem leak for async stream + completion Uses a global executor pool instead of creating a new thread on each request Fixes https://github.com/BerriAI/litellm/issues/6404 * fix(factory.py): handle tool call + content in assistant message for bedrock * fix: fix import * fix(factory.py): maintain support for content as a str in assistant response * fix: fix import * test: cleanup test * fix(vertex_and_google_ai_studio/): return none for content if no str value * test: retry flaky tests * (UI) Fix viewing members, keys in a team + added testing (#6514) * fix listing teams on ui * LiteLLM Minor Fixes & Improvements (10/28/2024) (#6475) * fix(anthropic/chat/transformation.py): support anthropic disable_parallel_tool_use param Fixes https://github.com/BerriAI/litellm/issues/6456 * feat(anthropic/chat/transformation.py): support anthropic computer tool use Closes https://github.com/BerriAI/litellm/issues/6427 * fix(vertex_ai/common_utils.py): parse out '$schema' when calling vertex ai Fixes issue when trying to call vertex from vercel sdk * fix(main.py): add 'extra_headers' support for azure on all translation endpoints Fixes https://github.com/BerriAI/litellm/issues/6465 * fix: fix linting errors * fix(transformation.py): handle no beta headers for anthropic * test: cleanup test * fix: fix linting error * fix: fix linting errors * fix: fix linting errors * fix(transformation.py): handle dummy tool call * fix(main.py): fix linting error * fix(azure.py): pass required param * LiteLLM Minor Fixes & Improvements (10/24/2024) (#6441) * fix(azure.py): handle /openai/deployment in azure api base * fix(factory.py): fix faulty anthropic tool result translation check Fixes https://github.com/BerriAI/litellm/issues/6422 * fix(gpt_transformation.py): add support for parallel_tool_calls to azure Fixes https://github.com/BerriAI/litellm/issues/6440 * fix(factory.py): support anthropic prompt caching for tool results * fix(vertex_ai/common_utils): don't pop non-null required field Fixes https://github.com/BerriAI/litellm/issues/6426 * feat(vertex_ai.py): support code_execution tool call for vertex ai + gemini Closes https://github.com/BerriAI/litellm/issues/6434 * build(model_prices_and_context_window.json): Add 'supports_assistant_prefill' for bedrock claude-3-5-sonnet v2 models Closes https://github.com/BerriAI/litellm/issues/6437 * fix(types/utils.py): fix linting * test: update test to include required fields * test: fix test * test: handle flaky test * test: remove e2e test - hitting gemini rate limits * Litellm dev 10 26 2024 (#6472) * docs(exception_mapping.md): add missing exception types Fixes https://github.com/Aider-AI/aider/issues/2120#issuecomment-2438971183 * fix(main.py): register custom model pricing with specific key Ensure custom model pricing is registered to the specific model+provider key combination * test: make testing more robust for custom pricing * fix(redis_cache.py): instrument otel logging for sync redis calls ensures complete coverage for all redis cache calls * (Testing) Add unit testing for DualCache - ensure in memory cache is used when expected (#6471) * test test_dual_cache_get_set * unit testing for dual cache * fix async_set_cache_sadd * test_dual_cache_local_only * redis otel tracing + async support for latency routing (#6452) * docs(exception_mapping.md): add missing exception types Fixes https://github.com/Aider-AI/aider/issues/2120#issuecomment-2438971183 * fix(main.py): register custom model pricing with specific key Ensure custom model pricing is registered to the specific model+provider key combination * test: make testing more robust for custom pricing * fix(redis_cache.py): instrument otel logging for sync redis calls ensures complete coverage for all redis cache calls * refactor: pass parent_otel_span for redis caching calls in router allows for more observability into what calls are causing latency issues * test: update tests with new params * refactor: ensure e2e otel tracing for router * refactor(router.py): add more otel tracing acrosss router catch all latency issues for router requests * fix: fix linting error * fix(router.py): fix linting error * fix: fix test * test: fix tests * fix(dual_cache.py): pass ttl to redis cache * fix: fix param * fix(dual_cache.py): set default value for parent_otel_span * fix(transformation.py): support 'response_format' for anthropic calls * fix(transformation.py): check for cache_control inside 'function' block * fix: fix linting error * fix: fix linting errors --------- Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> --------- Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com> * ui new build * Add retry strat (#6520) Signed-off-by: dbczumar <corey.zumar@databricks.com> * (fix) slack alerting - don't spam the failed cost tracking alert for the same model (#6543) * fix use failing_model as cache key for failed_tracking_alert * fix use standard logging payload for getting response cost * fix kwargs.get("response_cost") * fix getting response cost * (feat) add XAI ChatCompletion Support (#6373) * init commit for XAI * add full logic for xai chat completion * test_completion_xai * docs xAI * add xai/grok-beta * test_xai_chat_config_get_openai_compatible_provider_info * test_xai_chat_config_map_openai_params * add xai streaming test --------- Signed-off-by: dbczumar <corey.zumar@databricks.com> Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com> Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com> |
||
|
Krish Dholakia
|
d57be47b0f
|
Litellm ruff linting enforcement (#5992)
* ci(config.yml): add a 'check_code_quality' step Addresses https://github.com/BerriAI/litellm/issues/5991 * ci(config.yml): check why circle ci doesn't pick up this test * ci(config.yml): fix to run 'check_code_quality' tests * fix(__init__.py): fix unprotected import * fix(__init__.py): don't remove unused imports * build(ruff.toml): update ruff.toml to ignore unused imports * fix: fix: ruff + pyright - fix linting + type-checking errors * fix: fix linting errors * fix(lago.py): fix module init error * fix: fix linting errors * ci(config.yml): cd into correct dir for checks * fix(proxy_server.py): fix linting error * fix(utils.py): fix bare except causes ruff linting errors * fix: ruff - fix remaining linting errors * fix(clickhouse.py): use standard logging object * fix(__init__.py): fix unprotected import * fix: ruff - fix linting errors * fix: fix linting errors * ci(config.yml): cleanup code qa step (formatting handled in local_testing) * fix(_health_endpoints.py): fix ruff linting errors * ci(config.yml): just use ruff in check_code_quality pipeline for now * build(custom_guardrail.py): include missing file * style(embedding_handler.py): fix ruff check |
||
|
Krrish Dholakia
|
14da2d5ade |
fix(_logging.py): fix timestamp format for json logs
Outputs timestamp in ISO 8601 format to work with fluentbit |
||
|
Krish Dholakia
|
63216f42b8
|
Merge branch 'main' into litellm_gemini_refactoring | ||
|
Krrish Dholakia
|
4cddd1eb31 |
fix(_logging.py): add loglevel and timestamp to json logs
Fixes https://github.com/BerriAI/litellm/issues/4248 |