Kimi K3 accepts exactly low, high and max, defaults to max, and always thinks.
The map could not say that: medium and high have no supports_*_reasoning_effort
flag because every other reasoning model takes them, so the ten kimi-k3 entries
carried supports_reasoning alone and resolved to unknown. The dashboard then fell
back to a capability-blind level list that deliberately omits max, which is why a
kimi-k3 tier cannot be set to max thinking today.
Add reasoning_effort_levels, an array key in the shape the map already uses for
supported_endpoints and supported_modalities. Where present it is read first and
wins whole; every other entry keeps answering through the per-level flags,
unchanged. It is deliberately a different name from the computed
ModelGroupInfo.supported_reasoning_efforts, which stays derived from a group's
deployments and is never seeded from one deployment's model_info.
The levels are per entry rather than per model, because the deployments differ:
Moonshot, Together, Fireworks and Azure Foundry all forward the level unchanged
and get the model's own low/high/max, while Perplexity documents a six-value
enum it maps down internally and gets that. The /v1/messages degradation chain
consults the same declaration, so the level the map advertises is the level that
path forwards.
The realtime health check always built the Azure websocket URL with the
default beta protocol, so GA-only transcription models such as
azure/gpt-realtime-whisper got probed at /openai/realtime and were
rejected with HTTP 400 on every /health run, while real calls through
the proxy resolved the GA path via intent=transcription and worked.
The probe now resolves the protocol the way the real call path does:
an explicit realtime_protocol (argument, deployment litellm_params, or
LITELLM_AZURE_REALTIME_PROTOCOL) wins, transcription-only models fall
back to GA with intent=transcription, and everything else keeps beta.
Transcription-only detection reads both mode and supported_endpoints
from get_model_info because a live proxy overwrites the catalog mode
with the operator's deployment model_info (mode: realtime) during
router registration, while supported_endpoints survives it.
get_model_info now propagates supported_endpoints from the cost map;
it declared the field but never populated it.
* feat(logging): add async_post_call_failure_deployment_hook
CustomLogger already has async_pre_call_deployment_hook and
async_post_call_success_deployment_hook, both firing once per real
deployment attempt from wrapper_async since the router re-enters that
wrapper fresh on every retry and fallback step. There was no failure-side
counterpart; the only failure signal, async_log_failure_event, fires once
per logical client request behind a dedup gate, so fallback chain attempts
2+ were invisible to callbacks needing per-deployment-attempt granularity.
Adds async_post_call_failure_deployment_hook(request_data, exception,
call_type) to CustomLogger and a matching dispatcher in utils.py, called
from wrapper_async's except block. It needs no dedup coordination since
each real attempt naturally re-enters the wrapper once. Unlike its two
siblings, the dispatcher wraps each callback call in its own try/except
since it runs on the wrapper's own exception path and a broken callback
must never mask the exception about to be re-raised to the caller.
* feat(logging): pass fallback_depth through to async_post_call_failure_deployment_hook
Router already tracks fallback_depth internally on each fallback hop
(litellm/router_utils/fallback_event_handlers.py), incrementing it once per
target tried, but nothing surfaced it to CustomLogger callbacks. Reads it
off request_data in the dispatcher and passes it through as a best-effort
int | None keyword: None on the first, pre-fallback attempt or a bare SDK
call with no router, 1 on the first fallback hop, 2 on the second, and so
on. Verified live against a real multi-hop Router fallback chain before
adding the regression tests.
* fix(logging): fire async_post_call_failure_deployment_hook on internal calls too
The failure hook was gated behind the same not _is_litellm_internal_call
check as the request-level dedup-gated failure logging, so a failed
internal sub-call (e.g. an emulated file-search step) never reached it,
even though its async_pre_call_deployment_hook and
async_post_call_success_deployment_hook siblings already fire
unconditionally for such calls.
* chore: retrigger CI (lint job hit a transient GitHub Actions infra outage on the prior push)
* chore: retrigger CI (lint job hit the same GitHub Actions infra outage again)
* fix(logging): scope async_post_call_failure_deployment_hook to the actual model call
The hook was dispatched from the wrapper's broad outer except, which also
catches BudgetExceededError (raised before any deployment attempt),
errors from async_pre_call_deployment_hook, and errors raised after a
successful model call (post_call_processing, async_post_call_success_deployment_hook,
caching). None of those are a deployment attempt failing, so the hook
misreported them as one.
Scoped the hook to a try/except around the model call itself, so it only
fires when that specific call raises, matching its own documented contract.
* test: assert the callback actually ran in the failure-hook error-isolation test
An upstream test-quality gate (TQ001) flagged this test for asserting
nothing, so it could only fail by raising. Track whether the exploding
callback actually ran and assert on it, so the test would catch a
dispatcher that silently skipped every callback instead of isolating a
raising one.
* fix(logging): harden async_post_call_failure_deployment_hook against 5 maintainer-verified issues
A maintainer's live-proxy A/B review against base found five real
problems with the failure hook, all reproduced and fixed:
- The dispatcher called overrides with fallback_depth as a required
keyword, so an override matching this PR's own earlier 3-arg
proof-of-fix example raised TypeError, swallowed at debug level, on
every call. Now checks the override's signature once per class and
omits the keyword when unsupported.
- A callback mutating the exception it receives (e.g. status_code)
changed what the real caller got back, since it was the same live
object about to be re-raised. Callbacks now receive a same-class
snapshot instead.
- request_data exposed attempted_targets, the router's own live
fallback-walk bookkeeping shared by reference across every hop, so a
callback calling .record() on it could make the router skip a
deployment it never actually tried. Now excluded from what the hook
receives.
- The hook's own await sat directly in the model-call except block, so
a caller-side cancellation landing mid-await (e.g. asyncio.wait_for)
replaced the real deployment exception with CancelledError/
TimeoutError. Now isolated so hook dispatch can never mask the real
failure.
- The timestamp used for the reported failure duration was captured
after the hook ran, so a slow callback inflated
async_log_failure_event's duration. Now captured before the hook
dispatches.
* fix(logging): preserve traceback/cause/context on the failure-hook exception snapshot
Bugbot found a real gap in the previous round's exception-mutation fix:
_snapshot_exception_for_hook only copied __dict__ and args, so a
callback formatting or inspecting the failure chain saw an empty
traceback and lost chained-exception context, even though the live
exception still has them. __traceback__/__cause__/__context__ aren't
stored in __dict__, so they need copying explicitly.
* fix(logging): preserve __suppress_context__ on the failure-hook exception snapshot
Setting __cause__ has a documented CPython side effect of implicitly
forcing __suppress_context__ to True. Since the previous round's
traceback fix set __cause__ before __suppress_context__, a normal
implicit-chaining exception (no `raise ... from`, __suppress_context__
naturally False) got its context wrongly suppressed on the snapshot.
Now __suppress_context__ is set explicitly, after __cause__, so it
always reflects the real exception.
* fix(logging): use MappingProxyType for the failure-hook's sanitized request_data
A LIT002 budget check (surfaced by rebasing onto a moved base) flagged
the dict comprehension building safe_request_data as mutable
construction. MappingProxyType is also a strictly better fit here: a
genuinely read-only view, not just an immutable-looking dict, matching
the intent that callbacks should never be able to mutate what they're
handed.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Follow-up to #38130. The function has no callers in the repo or the docs and is
not exported from `litellm/__init__.py`, and `token_counter` already does the same
job better, so keeping a second entry point only preserves a trap.
That trap is real: Greptile flagged on #38130 that `token_counter` picks the claude
tokenizer only for bare ids. `claude-sonnet-4-5` resolves to huggingface_tokenizer,
while `claude-3-opus-20240229` and `anthropic/claude-sonnet-4-5` fall back to the
OpenAI one, 24 tokens against 27 on the same string. Deleting the wrapper removes
the surface rather than papering over it; the selection gap in `token_counter`
itself is worth its own fix.
BREAKING CHANGE: `from litellm.utils import prompt_token_calculator` no longer
resolves. Use `litellm.token_counter(model=..., text=...)`.
The claude branch called the anthropic SDK's `Anthropic().count_tokens`, which the
SDK removed, so every claude call raised AttributeError. Counting now goes through
litellm's own token_counter, which handles anthropic models offline and drops the
SDK dependency entirely.
Hiding that was a swallowed error: `except Exception: Exception("Anthropic import
failed please run `pip install anthropic`")` built the exception without raising
it, so an environment missing the SDK fell through to the unguarded
`from anthropic import ...` on the next line and got a bare ModuleNotFoundError
instead of the install hint.
That was the codebase's last PLW0133, so the rule graduates from the ratcheted
budget into ruff.toml where it hard-fails, and editors get the diagnostic inline.
Regenerate model_prices_and_context_window.schema.json and add the flag to
the inline validator schema in test_utils.py so the new cost-map key passes
validate-model-prices-json and the JSON-valid test.
Six defects in the RunwayML video provider:
- transform_video_create_request hardcoded /image_to_video, so text-to-video 400'd and video-to-video was unreachable; the endpoint is now selected from the inputs present (promptVideo/videoUri, promptImage, or text only)
- get_error_class raised instead of returning, turning a provider 4xx into a proxy 500 APIConnectionError; it now returns a RunwayMLError
- VideoObject.progress was typed int while Runway sends a 0..1 float, 500'ing status polls while RUNNING; it is now scaled to a 0..100 percent
- custom per-deployment pricing stored under litellm_metadata was ignored for video; the deployment model_info lookup now checks both metadata keys
- stale cost-map entries (gen3a_turbo, gen4_aleph) were removed and current models added, with output_cost_per_second_480p/_4k tier keys plumbed through the model-info and router types
- video cost now falls back to Runway's estimatedCost from the create response when no custom pricing is configured, and custom pricing always wins over it
Fixes#36483
Tencent chat completions route through the OpenAI SDK's
chat.completions.create(), which raises TypeError on unknown kwargs -
so a top-level 'thinking' optional param crashed every reasoning
request with a 500 before any HTTP call was made.
Nest the resolved thinking object in extra_body instead: the SDK merges
extra_body into the top-level JSON payload, so TokenHub still receives
the documented thinking field (type/budget_tokens) in the request body.
Also align the param mapping with TokenHub's documented behavior:
- reasoning_effort="none" now maps to thinking={"type": "disabled"}
instead of being dropped (deepseek-v4-* default to thinking enabled,
so dropping it never actually disabled thinking)
- MiniMax models only accept thinking.type "adaptive"/"disabled",
so "enabled" is coerced to "adaptive" instead of returning a 400
Refs: https://www.tencentcloud.com/document/product/1300/82345
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
* test: use monkeypatch.setenv for env writes in tests/test_litellm
`os.environ["X"] = v` inside a test leaks the value into every test that runs
after it in the same worker, so ordering decides the result. 262 of those
writes across 40 files now go through pytest's `monkeypatch` fixture, which
restores the previous value at teardown.
The rewrite skips any test that a mock.patch-family decorator wraps, any test
with defaulted positional parameters, any test whose own name is called
directly elsewhere, and rebinds nothing inside nested defs, because in each of
those cases appending a fixture parameter changes what pytest or mock binds.
Ratchets the TQ004 ceiling from 768 to 506.
* fix(test): delete the key through monkeypatch instead of popping it first
Five tests popped a key straight out of `os.environ`, ran, then restored it with
`monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone,
so it recorded "absent" as the value to go back to and deleted the key at
teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`,
`UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first
one ran without it.
`monkeypatch.delenv(..., raising=False)` removes the key and restores whatever
was there, so the try/finally the manual restore needed goes with it.
* chore(test): leave the two cost-calc files to the PR that rewrites them fully
Both files are also in #37815, which converts the module-global writes as well
as the env writes and folds them into one fixture. Two PRs rewriting the same
lines differently is a conflict nobody benefits from resolving, so this one
drops back to staging on those two and keeps the other 39.
TQ004 clears 200 here instead of 275; the rest moves with #37815.
Four more ruff rules for code the test suite runs but never checks. F601 is the
one that paid: the duplicate key it flagged in a get_form_data fixture was the
mock reproducing the production bug fixed in the previous commit.
B025 removed two unreachable handlers, one of them a pytest.skip shadowed by an
earlier `pass`, so an upstream Vertex flake reported green having asserted
nothing. F632 turned an `is ""` identity check, which passes only on CPython
interning, into the `== ""` it meant. B023 fixed three closures over loop
variables, all latent today but one iteration-order change away from checking the
last case N times.
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
Perplexity's Agent API model ids already start with perplexity/, so a litellm
model like perplexity/perplexity/glm-5.2 normalizes to model perplexity/glm-5.2
with custom_llm_provider perplexity. Every existing candidate in the cost map
lookup ladder reads that leading perplexity/ as the litellm prefix and strips
it, so the four new rows never matched and supports_reasoning, get_model_info,
and completion_cost all failed on them.
Add the doubled form as a last-resort candidate, tried after every candidate
tried today and still ahead of the capability generalization rules, so only
lookups that already raised can newly succeed. Sweeping all 3062 cost map keys
across the three _get_potential_model_names branches (8433 probes) shows 7
lookups change, all of them errors that now resolve: the 4 new perplexity rows
plus openrouter/openrouter/auto, /free, and /bodybuilder. Nothing regresses.
The cache control hook also runs on litellm.responses() input. On a
GPT-5.6 deployment it wrapped a string-content item into a chat-shaped
{"type": "text"} part, which the Responses API rejects, and it never
marked input_text, input_image or input_file parts, so no breakpoint and
no prompt_cache_options reached the provider. Add the Responses part
types to the eligible block set and translate chat-shaped text parts on
non-assistant items to input_text in
ResponsesAPIRequestUtils.merge_prompt_management_input, which both the
async and the sync prompt management sites go through.
The dialect also fired for any GPT-5.6 name that resolved to provider
openai, including deployments pointed at a custom api_base that does not
understand prompt_cache_breakpoint. Decide it once per request from the
provider, the model map and the resolved api_base (request, then
litellm.api_base, then OPENAI_BASE_URL / OPENAI_API_BASE): only
api.openai.com and *.api.openai.com hosts speak the dialect, a top-level
prompt_cache_options opts a custom target in, and litellm_proxy/ targets
never get it. maybe_seed_default_injection_points takes api_base and
stamps the finished decision on the points as _litellm_openai_dialect so
the sync completion() path, whose hook params do not carry api_base,
honors it; maybe_inject_cache_control takes api_base from the
/v1/messages handler.
Eligibility now comes from a supports_prompt_cache_breakpoint model map
flag on the OpenAI gpt-5.6 entries, exposed through
litellm.utils.supports_prompt_cache_breakpoint, with the GPT version rule
kept only for models the map does not know. The OpenAI dialect no longer
reserves a slot for tool_config points, which OpenAI has no cache block
for, and with_prompt_cache_breakpoint plus the chat bridge helper return
a new block instead of mutating their input.
The proxy pre-creates the logging object before the router picks a deployment,
and the native /v1/messages handler never copied the deployment's
vertex_location into the logging params it updates, so cost resolution fell
back to the environment or the default region and priced every call on this
surface with the regional endpoint uplift. Copy the explicitly configured
location from the request's litellm params, the same source dispatch builds
the request URL from, and register the new regional_endpoint_uplift_multiplier
field in the cost map schema test.
litellm.completion(thinking=True) crashed pre-network in is_thinking_enabled
with a retryable APIConnectionError ('bool' object has no attribute 'get'),
so the router burned retries on a deterministic failure and proxy clients got
a traceback instead of a usable response.
validate_and_fix_thinking_param now coerces thinking=True to the enabled dict
with the default medium budget and drops thinking=False, and the remaining
dict-assuming thinking accessors (base config, bedrock converse, deepseek)
guard with isinstance so raw bools can never crash a transform.
A blocked chunk now logs the summed usage and cost of every ApplyGuardrail
call AWS billed for the logical request, not just the blocking chunk.
Client-supplied metadata.standard_logging_guardrail_information is stripped
at the proxy boundary so callers cannot forge (even negative) guardrail
cost into spend, and guardrail_information_cost ignores negative or
non-finite entry costs as defense in depth. The cost map schema test now
allows guardrail_cost_per_unit and the guardrail mode.
Registering the five managed-batch fields in all_litellm_params stops them leaking into
extra_body, but two of them never reached the transformation that reads them.
CredentialLiteLLMParams is a whitelist, so get_deployment_credentials_with_provider
round-tripped the deployment and silently dropped s3_output_bucket_name and bedrock_tags
before the files/batch/passthrough callers saw them. s3_bucket_name, s3_region_name and
aws_batch_role_arn were added to that model for #25104; these two are the remainder of
the same deployment config
bedrock_tags is typed as a plain list rather than a stricter shape so a malformed value
still reaches _validate_bedrock_tags and gets its own error message instead of a Pydantic
one
The preservation assertion previously round-tripped through GenericLiteLLMParams, which is
extra="allow" and would hold even for a field nothing declares. It now also reproduces the
CredentialLiteLLMParams normalization the proxy actually performs, and fails naming
exactly the dropped fields without this change
A Bedrock managed-batch deployment carries aws_batch_role_arn, s3_bucket_name,
s3_region_name, s3_output_bucket_name and bedrock_tags in its litellm_params,
and the batch and files transformations read all five from there. None was
registered in all_litellm_params, so the param builder swept them into
extra_body on every other route that deployment serves: Bedrock answers
"aws_batch_role_arn: Extra inputs are not permitted" on Anthropic models and
"extraneous key [aws_batch_role_arn] is not permitted" on Nova, Llama and
Titan, so configuring batch turns every chat and embedding request to that
model into a 400.
Register them alongside the agentic-loop and callback-credential fields, which
are listed for exactly this reason. The batch path is unaffected because
GenericLiteLLMParams is extra="allow" and preserves them into litellm_params
for the transformations that consume them.
Before this, batch could only be configured on a deployment dedicated to
batch; the same model group could not serve both.
Register Nano Banana 2 Lite on the unprefixed, gemini/, and vertex_ai/ keys
so completion_cost and pass-through spend tracking no longer treat the
model as unmapped
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
The web-search interception hooks stamp
_websearch_interception_emit_native_blocks and
_websearch_interception_converted_stream onto kwargs to carry state across
the agentic loop, but neither was registered in all_litellm_params. The
param builder sweeps anything it does not recognize into the outbound
request, so a provider that validates its body rejects the whole call:
Bedrock Converse answers "_websearch_interception_emit_native_blocks:
Extra inputs are not permitted" with a 400, which breaks every request
interception touches on that route.
Register both alongside their code-interpreter counterparts, which were
already listed for exactly this reason.
Resolves LIT-5391
* feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars
Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and
two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both
setters after assigning litellm_trace_id so every JSON log record emitted within the
async request context carries trace_id and, when provided, session_id — enabling log
correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without
any changes to individual log call sites.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields
* fix(logging): always reset session_id_var to empty string when no session_id provided
* feat: gate request correlation IDs in logs behind request_correlation_in_logs flag
* refactor: move correlation ID injection into CorrelationContextFilter
* feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload
Plaintext log lines (json_logs off) now get the same trace_id/session_id
suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a
visible effect regardless of log format.
StandardLoggingPayload gets a new independent session_id field, populated
from litellm_session_id. trace_id's existing session_id-first fallback is
preserved when request_correlation_in_logs is off; with the flag on, an
explicit litellm_trace_id now takes priority over litellm_session_id so
the two fields carry genuinely independent values.
* fix(logging): restore correlation context after nested calls; sanitize correlation ids
Addresses two review findings on this PR.
CorrelationContextFilter's trace_id/session_id contextvars were set on every
Logging.__init__ but never reset, so a nested LiteLLM call sharing the same
asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call,
an MCP sampling call) would leave the outer request's subsequent log lines
stamped with the nested call's ids instead of its own. set_trace_id/
set_session_id now return their contextvars.Token, and Logging stores them
and resets both once its own success/failure handler actually completes,
via a new idempotent _restore_correlation_context() called from all four
terminal handlers.
set_trace_id/set_session_id also now strip control characters and bound
length before storing a caller-controlled trace_id/session_id, since these
values can originate from request input (litellm_session_id, x-litellm-
trace-id) and get interpolated into plain-text log lines - without this, a
caller could embed \r/\n or escape sequences to forge fake log entries.
* fix(logging): restore correlation context after nested calls, not before
The previous commit called _restore_correlation_context() as the first
line of each terminal handler, before that handler's own callback
dispatch loop runs. That's backwards: a nested LiteLLM call triggered
from within a callback (e.g. a guardrail's own LLM-as-judge call) would
then capture the *already-reset* value as its own pre-call baseline,
and its own reset would restore to that instead of the true outer
value - verified live to still leak.
success_handler/async_success_handler/failure_handler/async_failure_handler
are now thin wrappers: the original bodies move to
_success_handler_body/etc, called inside a try/finally that restores
context only once the full body - including any nested calls its own
callback dispatch triggers - has actually finished, mirroring proper
stack-scoped nesting semantics.
* test(logging): cover async_failure_handler's correlation-context restore
Codecov flagged the new async_failure_handler wrapper (try/finally around
_async_failure_handler_body) as uncovered - the method had no direct test
at all before this PR's refactor split it into a wrapper. Adds a test that
awaits it directly and asserts both that async_log_failure_event still
fires and that _restore_correlation_context() puts the pre-call
trace_id/session_id back.
* fix(logging): restore correlation context by value, not by contextvars.Token
veria-ai correctly flagged that contextvars.Token.reset() only works in the
exact Context it was created in, and litellm's async success path (and
streaming failure path) dispatch async_success_handler/async_failure_handler
via asyncio.create_task and the global logging worker - a different Context
than Logging.__init__ ran in. reset_trace_id/reset_session_id silently
swallowed the resulting ValueError, so the restore was a no-op for exactly
those paths. Verified independently: reproduced the raw contextvars
behavior, then confirmed litellm's async success dispatch really does go
through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py).
Logging now captures the pre-call *value* (not a Token) and restores via a
plain set_trace_id()/set_session_id() call, which works regardless of which
Task/Context calls it. reset_trace_id/reset_session_id are removed as
dead/unreliable code. Added a regression test that spawns __init__ and the
restore in different asyncio Tasks - confirmed it fails against the prior
Token-based commit and passes here.
* fix(logging): restore correlation context in the originating task too
Greptile's re-review correctly identified a remaining gap: for a
successful acompletion(), async_success_handler is dispatched via
asyncio.create_task + the global logging worker into a *different* Task
than the one wrapper_async/Logging.__init__ ran in. The prior fix (43c164a)
only restored the handler's own (detached, throwaway) Task - it never
touched the originating request Task, which keeps this call's trace_id/
session_id set for the rest of its own execution (e.g. nested calls made
via the same Task).
wrapper()/wrapper_async() in litellm/utils.py now restore the originating
Task's correlation context in a finally block once the whole call is done,
regardless of what detached logging tasks it spawned along the way. Since
the wrapped body rebinds its own `kwargs` local via function_setup(),
sharing the dict object doesn't work here; a small mutable holder carries
the constructed Logging instance back out to the outer wrapper instead.
_restore_correlation_context() is no longer guarded against repeat calls:
with value-based (not Token-based) restoration, each distinct Task that
calls it needs its own restore to take effect in that Task's own view of
the contextvars, so multiple calls (once per Task involved in an attempt)
are required, not just tolerated.
Added a regression test using mock_response to exercise the real success
dispatch path (asyncio.create_task + GLOBAL_LOGGING_WORKER) without a live
provider call, asserting the *test's own* (originating) task context is
restored after the call - this is exactly the case Greptile flagged and
the prior commit didn't cover.
* fix(logging): restore correlation context when function_setup itself fails
Greptile's 4th finding: if function_setup() constructs Logging() (whose
__init__ already mutates trace_id_var/session_id_var) and then raises
before returning - e.g. update_environment_variables() throws - the
caller's wrapper()/wrapper_async() never receives a logging_obj reference,
so its own restore-on-finally never fires. The correlation ids leak into
every subsequent log line on that thread/task with no way to clear them.
function_setup()'s own except block now restores the context itself in
that case, using whatever logging_obj it managed to construct before
failing (locals().get(), safe against the earlier failure modes where
logging_obj was never assigned at all).
Added a regression test that monkeypatches Logging.update_environment_variables
to raise after construction, confirmed it fails without this fix (the
leaked ids show up directly in the raised exception's own log line) and
passes with it. Broader sweep (test_utils.py, test_router.py,
test_main_module_header.py, streaming handler tests, plus all
logging-specific tests): 722 passed.
* fix(logging): don't assume every litellm_logging_obj is a real Logging instance
CI caught a real regression from the last commit: tests/test_litellm/llms/xai/test_xai_key_fallback.py
injects a minimal FakeLogging stand-in (only implementing
update_from_kwargs) as litellm_logging_obj for a narrow realtime-config
unit test, bypassing the real Logging class entirely. wrapper()/
wrapper_async()'s finally block and function_setup()'s except block both
unconditionally called _restore_correlation_context() on whatever ended up
in the holder, which doesn't exist on that stand-in.
_restore_correlation_context is new plumbing specific to this PR's
feature, not part of any pre-existing stand-in's expected interface, so
callers of it can't assume every object playing the litellm_logging_obj
role implements it. Added _restore_correlation_context_if_supported(),
a small getattr-guarded helper, and used it at all three call sites.
* fix(logging): don't restore context too early on setup failure or streaming
Two more findings from Greptile's 5th review round.
1. function_setup()'s except block restored correlation context *after*
logging the "Error in function_setup" exception, so that diagnostic log
line itself was stamped with the doomed call's ids instead of the outer
ids - misleading, since the failed call never produces anything else to
attribute those ids to. Restore now happens before the log call.
2. wrapper()/wrapper_async() restored the originating task's context as
soon as a streaming call returned, before the caller ever starts
iterating the CustomStreamWrapper it just got back. Any log lines
emitted while iterating (in the same thread/task) incorrectly showed
the pre-call ids instead of this call's own ones. The wrapper finally
block now skips the restore when the return value is a stream wrapper,
deferring to the terminal handler that already fires once the stream is
actually assembled/exhausted.
Both verified with tests that fail against the prior commit and pass
against this one. Broader sweep unchanged at 829 passing.
* fix(logging): best-effort correlation cleanup on abandoned streams
Greptile's 7th finding: if a caller returns a streaming response and never
fully consumes it - stops iterating early, drops the reference, cancels
it - the terminal handler that normally restores the originating task's
trace_id/session_id never fires, since it only runs once the stream is
actually assembled/exhausted. The ids leak into every subsequent log line
in that thread/task with no bound.
There's no reliable Python hook for "this was abandoned without being
closed" - CustomStreamWrapper has no close()/__aexit__/context-manager
convention today, and the only automatic option is __del__, whose timing
is inherently unpredictable (delayed by cyclic GC, not guaranteed at
interpreter shutdown, can run on a different thread). This is a best-effort
safety net, not a guarantee, and is documented as such in the docstring.
Testing this via real garbage collection proved unreliable in practice:
per-chunk logging submits work to a thread pool executor whose worker
thread transiently holds its own bound-method reference to the wrapper
until that task completes, so refcount doesn't hit zero on a
deterministic schedule even with polling. Tests call __del__ directly
instead - a plain method, safe to invoke early - which exercises exactly
the restore logic real garbage collection would eventually trigger,
plus a case confirming a broken logging_obj can never make __del__ raise.
* fix(logging): restore consumer's context at every real stream exit point
Two more findings from this round.
Veria AI: even a *fully consumed* stream never restored the actual
consuming thread/task's correlation context. The terminal success dispatch
(dispatch_success_handlers via asyncio.create_task for async, or
success_handler via the shared executor for sync) only restores whatever
detached context it runs in - never the caller's own thread/task that's
running the for/async for loop. Same root cause as the wrapper-level fix
two rounds ago, just missed for the streaming-completion path.
Greptile: explicit aclose() (client disconnect, router fallback aborting
a partial stream) closed the underlying stream without restoring
correlation context either, since request wrappers intentionally skip
restoration for returned streams and no terminal handler runs on this
path.
Added CustomStreamWrapper._restore_consumer_correlation_context(), called
from every point control genuinely returns to the consumer: the final
raise StopIteration/StopAsyncIteration on natural exhaustion (both sync
branches, both async branches), _handle_stream_fallback_error (the shared
choke point for all three failure-raising call sites), and aclose(). __del__
now delegates to the same helper instead of duplicating it.
Verified with tests extending the existing streaming-exhaustion cases to
assert the consuming context is restored after the loop completes (fails
against the prior commit, passes now), plus a dedicated aclose() test.
Broader sweep: 832 passing.
* fix(logging): don't let a delayed __del__ finalizer clobber a newer active call
If an abandoned stream's __del__ fires late (after cyclic GC delay), a
different call may have already taken over the correlation contextvars in
the same Task/thread. Restoring unconditionally would stomp that active
call's trace_id/session_id with the abandoned stream's stale pre-call
snapshot. __del__ now only restores when the contextvars still hold the
ids this call itself set.
* fix(logging): compare sanitized ids in the __del__ ownership guard
set_trace_id()/set_session_id() sanitize (strip control chars, bound length)
before storing, so the contextvar's value can differ from the raw
litellm_trace_id/litellm_session_id. The __del__ ownership guard was
comparing against the raw values, so a caller-supplied id containing control
characters or exceeding 256 chars would never match, permanently skipping
cleanup. Capture what set_trace_id()/set_session_id() actually stored and
compare against that instead.
* fix(logging): restore consumer context on the synthesized finish_reason chunk
Both __next__ and _finalize_completed_stream() have a branch that fires when
the underlying stream ends without ever emitting an explicit finish_reason
chunk: they synthesize one via finish_reason_handler() and return it. A
consumer that stops as soon as it sees finish_reason - a common pattern -
never calls __next__()/__anext__() again, so the existing restore in the
sent_last_chunk-is-True StopIteration branch never runs for them. The
underlying stream is already exhausted at this point regardless of whether
the caller keeps iterating, so restoring here is safe.
* fix(logging): don't restore correlation context before the caller receives the final chunk
The previous fix (5147c69186) restored context immediately before returning
the synthesized finish_reason chunk from __next__/_finalize_completed_stream,
reasoning that completion_stream was already exhausted. But that chunk is
still this call's own data, and the caller's own application-level log
statements processing it run in the same synchronous frame right after the
return - restoring first made those lines carry the wrong (outer) ids,
exactly what wrapper()/wrapper_async() deliberately avoid by not restoring
while a stream is being iterated.
Revert to not restoring there. A caller that keeps iterating still gets a
correct, deterministic restore on its very next __next__()/__anext__() call
(completion_stream is exhausted, so that immediately re-raises
StopIteration/StopAsyncIteration through the already-restoring branch). A
caller that stops right after finish_reason relies on aclose() or the
best-effort __del__ guard, same as any other stream the caller doesn't fully
exhaust.
* refactor(logging): hoist a safely-hoistable function-body import to module top
CorrelationContextFilter.filter()'s `import litellm` was a function-body
import; verified it can move to module top without a circular-import failure
(litellm/__init__.py already imports from litellm._logging before setting
request_correlation_in_logs, but a bare `import litellm` only binds the
already-in-sys.modules module object - the attribute itself isn't read until
filter() actually runs, by which point litellm is fully initialized).
* test(logging): move correlation tests into their conventionally-mapped files
tests/test_litellm/ mirrors litellm/ in a parallel path. Correlation tests
for the Logging class (litellm_logging.py), function_setup/wrapper_async
(utils.py), and CustomStreamWrapper (streaming_handler.py) had all landed in
test_logging.py, which only maps to litellm/_logging.py itself. Moving each
group to its correctly-mapped file: test_litellm_logging.py (Logging class
init/restore), test_utils.py (function_setup, wrapper_async), and
test_streaming_handler.py (CustomStreamWrapper) in the next commit.
test_logging.py keeps only what actually exercises _logging.py's own
contextvars/filters/formatters/sanitization. No behavior change - same
assertions, same coverage, just relocated.
* fix(logging): restore correlation context unconditionally in wrapper()'s sync path
Blocking finding from review: a caller-visible correlation feature was
silently misattributing one request's logs to a different, unrelated one on
the sync/threaded path. wrapper()/wrapper_async() both left trace_id/session_id
"open" across a stream's entire iteration so the caller's own log lines while
consuming it would carry the right ids. That's safe for wrapper_async(): each
async call gets its own asyncio Task with its own copy of the contextvars,
and Tasks are never recycled across requests, so a leftover value can only
ever affect that one already-abandoned Task.
It is not safe for wrapper() (sync): a plain OS thread has no such per-call
isolation, and a thread pool's worker threads *are* recycled across unrelated
requests. If a sync stream was abandoned (client disconnect, early break, an
uncaught exception) without ever being exhausted or closed, nothing restored
its contextvars, and a pool could later hand that same thread to a completely
different call, which would inherit the abandoned request's ids as its own
"pre-call" baseline and then restore back to that poison when it finished -
permanently misattributing every subsequent log line on that thread,
including its own, to the abandoned request. Strengthening the __del__
finalizer already added for this can't fix it: finalizer timing is exactly
what a permanently-reused thread can't rely on.
wrapper() now restores unconditionally in its own finally, before a sync
stream is ever handed back to the caller. The trade-off: a sync stream
consumer's own application-level log statements while iterating no longer
automatically carry this call's ids (litellm's own internal per-chunk
logging is unaffected, since it's dispatched separately). That's an
acceptable cost for eliminating a silent cross-request misattribution bug.
wrapper_async() keeps the existing conditional (skip-if-streaming) behavior,
justified by the Task-isolation argument above; CustomStreamWrapper's
__del__/aclose()/next-iteration restore machinery remains meaningful and
necessary there.
This also simplifies wrapper()/wrapper_async() back toward their original
shape: both previously used a mutable-dict-holder split into a separate
_body function to smuggle logging_obj/result out to an outer finally,
working around function_setup() rebinding its own local `kwargs`. That
restructuring is no longer needed - `logging_obj` (and, for wrapper_async(),
`result`) were already function-level locals in scope for a plain
try/finally; three of wrapper_async()'s retry-return statements now assign
through `result` first so it accurately reflects what's actually returned
even on a retry path.
Regression test: test_abandoned_sync_stream_does_not_contaminate_a_later_call_on_the_same_thread
in test_streaming_handler.py reproduces the exact reported scenario with a
real single-worker ThreadPoolExecutor - confirmed it fails with the prior
(skip-restore-on-stream) wrapper() and passes with this fix.
* refactor(logging): use Mapping instead of bare dict for read-only params
_get_standard_logging_payload_trace_id/_session_id only read litellm_params
(.get() calls, no mutation) - annotate it as Mapping[str, Any] rather than a
bare mutable dict, per the repo's no-mutable-collection-in-annotation rule.
* fix(logging): scope request_correlation_in_logs to the async/proxy path only
Blocking review finding: wrapper() (the sync entry point) used the same
skip-restore-on-stream design as wrapper_async(), but a plain OS thread has
no per-call context isolation the way an asyncio Task does, and a thread
pool's worker threads are recycled across unrelated requests - an abandoned
sync stream could leave its ids stuck on a thread a pool later hands to a
completely different request, misattributing that request's logs. A fix
existed and was tested (restore unconditionally in wrapper()'s own finally),
but it doesn't benefit this feature's primary consumer - the proxy only ever
calls the async entry point - and carries sync-specific complexity this PR
doesn't need.
Scope the feature to async only instead: Logging.__init__() takes a new
supports_correlation_logging parameter (default True), threaded down from a
new function_setup(..., is_async_call: bool = True) parameter. wrapper() is
the one caller that passes is_async_call=False; every other function_setup()
call site (wrapper_async(), the router, and proxy/MCP-internal call sites)
is already async and keeps the default. With
supports_correlation_logging=False, Logging.__init__() never calls
set_trace_id()/set_session_id() at all, so a sync call has nothing to leak
in the first place. wrapper() reverts to its pre-review shape with no
correlation-specific code at all.
StandardLoggingPayload's own trace_id/session_id fields are unaffected
either way - they're a deterministic per-call read of
self.litellm_trace_id/self.litellm_session_id, not ambient contextvar state,
so they were never exposed to the cross-request bug.
Full sync/direct-SDK support (stamping + its own safe-restore mechanism) is
deferred to a follow-up PR; the fix and its regression test already exist in
this branch's history at commit 9f3a20f4b2 and can be resurrected there.
Tests: replaced the two wrapper()-level tests with ones proving the new
invariant (sync calls, streaming and non-streaming, never touch
trace_id_var/session_id_var even when the caller explicitly passes
litellm_trace_id/litellm_session_id), and added a direct unit test for the
supports_correlation_logging=False gate on Logging.__init__ itself. Verified
live: a real proxy (Postgres-backed, real OpenAI calls) shows clean
trace_id/session_id isolation across two concurrent sessions with no
cross-contamination; a standalone script confirms real sync SDK calls
against a real model never touch the correlation contextvars.
* feat(logging): fall back to W3C traceparent/baggage for trace_id/session_id
request_correlation_in_logs previously only resolved trace_id/session_id from
litellm-specific sources: x-litellm-trace-id/x-litellm-session-id headers, a
generic x-<vendor>-session-id header, or Anthropic-style metadata.user_id. If
none were present, trace_id fell back to an auto-generated UUID unrelated to
anything else, and session_id stayed empty - even when the caller already had
real distributed-tracing instrumentation sending the actual industry-standard
headers for this.
Add a fallback to the W3C Trace Context traceparent header (trace-id
component) and W3C Baggage header (session.id entry), so a request already
carrying real OpenTelemetry trace context correlates litellm's own logs with
the same trace in the caller's observability backend (Datadog, Honeycomb,
Tempo, etc.) instead of getting an unrelated generated id. Precedence is
unchanged for existing sources: explicit litellm headers and the Anthropic
metadata path both still win over this new fallback, which only fires when
neither found anything. trace_id and session_id are resolved independently
here (unlike the existing chain_id mechanism, which uses one shared value for
both), since traceparent and baggage are semantically distinct W3C concepts.
New helpers _trace_id_from_traceparent/_session_id_from_baggage in
litellm_pre_call_utils.py parse the header formats directly (no new
dependency - both are simple fixed-width/delimited strings), wired into
LiteLLMProxyRequestSetup.add_litellm_metadata_from_request_headers() only
when the corresponding litellm_trace_id/litellm_session_id key isn't already
set by the existing paths.
Verified live against a real proxy: a bare traceparent header produces a log
trace_id exactly matching its trace-id component; a traceparent alongside an
explicit x-litellm-trace-id header (different value) produces a log showing
the explicit header's value, proving precedence.
* fix(logging): reserve trace_id/session_id in JsonFormatter against message-content spoofing
JsonFormatter merges keys parsed from the message body before applying extra
record attributes, and the extra-attributes loop skips a key that's already
present. A caller-controlled log message that happens to parse as JSON/dict
with a "trace_id"/"session_id" key (e.g. the proxy logging a raw request-header
dict) could therefore make the JSON record carry the attacker-supplied value
instead of the real correlation context set via CorrelationContextFilter.
trace_id/session_id are now applied from the LogRecord's own attributes after
message-content parsing, unconditionally overwriting anything the message body
claimed for those two keys.
* style(logging): fix import order (ruff I001) in _logging.py and litellm_logging.py
- _logging.py: import litellm belongs after the stdlib from-imports, grouped
with the other litellm.* imports, not before them.
- litellm_logging.py: the refactor to Mapping introduced a second, separate
`from collections.abc import Mapping` instead of merging it into the
existing `from collections.abc import Callable` import.
Caught by the strict-rule budget gate (ruff-strict-budget.json caps I001 at
0 new violations); both auto-fixed with `ruff check --fix --select I001`.
* style(logging): freeze mutable-collection constructions flagged by LIT002
Five sites in this PR's diff built a mutable list/dict literal instead of a
frozen value: a plain list of optional strings in CorrelationPlainFormatter,
a `kwargs or {}` fallback, a `metadata or {}` fallback, two `[...]` candidate
orderings, and a `dict(headers)` copy feeding a dict comprehension. Each is
build-once/read-only, so this rewrites them as tuples, MappingProxyType, or a
plain conditional `.get()` instead of seeding then reading a fresh mutable
collection - no behavior change, confirmed by the existing test suite.
Caught by the type-discipline budget gate (LIT002 capped at 0 new
violations).
* fix(logging): reserve trace_id/session_id even when no correlation context is active
Live-proxy verification surfaced a gap in the earlier message-content-spoofing
fix (7f390a57fc): that fix only overwrites trace_id/session_id from the
LogRecord's own attribute, so it does nothing for a log line emitted before
CorrelationContextFilter has stamped anything on this record (e.g. the
"Request Headers" debug line, which fires before Logging.__init__() runs for
the request). On such a record, a caller-supplied header literally named
trace_id/session_id still got promoted into the JSON output via the embedded
JSON/dict-repr parser, since there was no genuine value to protect.
Fixed at the source: trace_id/session_id are now excluded unconditionally from
the message-content-parsing promotion step, not just superseded afterward.
Verified live against a real proxy - the exact adversarial request (headers
literally named trace_id/session_id) no longer leaks into any JSON log record.
Added a regression test for this no-active-context variant specifically,
confirmed it fails against the prior commit and passes now.
Also fixes an unrelated basedpyright regression from an earlier rebase's
conflict resolution: litellm/utils.py's `logging_obj` was incorrectly
re-annotated `Final` at its second assignment in function_setup() (it's first
declared `None` a few lines earlier), which basedpyright correctly rejects.
* fix(proxy): stop logging the raw W3C baggage session_id value
_session_id_from_baggage() extracts the caller-controlled session.id entry
verbatim - it isn't sanitized until set_session_id() runs later in
Logging.__init__(). The debug log line for this extraction interpolated the
raw value directly, so a caller could embed terminal control characters or
ANSI escape sequences that forge/alter plaintext log output for anyone
tailing the proxy's logs.
Verified live: a baggage header with an embedded ANSI escape reached the
terminal as a real, unescaped control sequence before this fix. Drops the
value from the log line entirely (the extraction succeeding is enough signal
on its own) rather than sanitizing-then-logging, matching veria-ai's
suggestion. Added a regression test using caplog that fails against the prior
commit and passes now.
* fix(logging): restore consumer context only after stream-failure exception mapping
_map_anthropic_exception/_map_aleph_alpha_exception synchronously log a debug
diagnostic (the raw status code) as part of exception_type()'s mapping.
_handle_stream_fallback_error restored the consumer's outer correlation
context before calling exception_type(), so that diagnostic log line carried
the outer (or empty) trace_id/session_id instead of the failing stream's own -
flagged by Greptile.
Moved the restore to run after mapping completes, matching the same
restore-after-not-before pattern already applied elsewhere in this file for
success/finish_reason handling. Added a regression test that captures the
correlation context live during a mocked exception_type() call; fails against
the prior commit, passes now.
* fix(logging): restore consumer context only after aclose()'s stream close completes
aclose() restored the consumer's outer correlation context as its first
statement, before awaiting the underlying provider stream's own aclose()/
close(). If that close attempt raises, the except branch's debug diagnostic
ran under the already-restored outer context instead of the closing stream's
own trace_id/session_id - flagged by Greptile, same restore-too-early pattern
as the stream-failure fix in f1cf9589d6.
Moved the restore to the end of aclose(), after the close attempt (and its
diagnostic logging) completes. Added a regression test with a fake stream
whose aclose() raises, capturing the correlation context live during the
diagnostic log call; fails against the prior commit, passes now.
* style(logging): satisfy new strict-lint budgets introduced upstream (Final, ANN401, S110, TRY300, kwargs typing)
Rebasing onto litellm_internal_staging pulled in 116 upstream commits that
introduced/tightened several lint gates this PR's own code now trips:
- LIT010 (every local/module-level variable must be Final): added Final
annotations across _logging.py, litellm_logging.py, streaming_handler.py,
litellm_pre_call_utils.py, and utils.py. Where a name is genuinely
reassigned (logging_obj: starts None, later set to the real object) or
branch-assigned, either restructured into a single ternary expression
(ordered_candidates) or suppressed with `# rebind-ok: <reason>` matching
this repo's documented escape hatch.
- LIT011 (parameter mutation): suppressed the two new `data[key] = value`
writes in litellm_pre_call_utils.py with `# rebind-ok`, matching the
unsuppressed precedent already used for every other `data[...]` write in
the same function - `data` is an intentional out-param there.
- ANN001/ANN003/ANN202 (missing parameter/return type annotations): fully
typed success_handler/_success_handler_body, their async twins, and
failure_handler/_failure_handler_body/async variants in litellm_logging.py,
plus function_setup in utils.py (added Rules to its existing TYPE_CHECKING
block for the rules_obj: Rules annotation).
- ANN401 (explicit Any disallowed): suppressed with `# noqa: ANN401` on the
handful of genuinely-heterogeneous result/*args/**kwargs parameters, since
ordinary suppression is this repo's documented path.
- S110 (try/except/pass): added to the existing BLE001 noqa on the one
best-effort correlation-cleanup try/except this PR added.
- TRY300 (return inside try): moved two `return result` statements into
`else:` blocks in the retry-fallback paths this PR's own diff touched.
- reportPrivateUsage (basedpyright): renamed the two new
StandardLoggingPayloadSetup static methods (get_standard_logging_payload_
trace_id/session_id) to drop their leading underscore, since they're
genuinely called from a sibling module-level function in the same file.
No behavior change - confirmed by the full existing test suite (819 passed)
plus all four lint gates (ruff format, ruff-strict, type-discipline,
basedpyright) passing clean.
* fix(lint): stop RUF100 flagging noqa suppressions the strict gate needs
CI's plain "ruff check" job uses the default ruff.toml, a narrower config
than ruff-strict.toml (used only by the strict-rule budget gate). ANN401 and
S110 aren't enabled in the default config, so RUF100 (unused-noqa) flagged
the `# noqa: ANN401`/`# noqa: ...,S110` suppressions this PR added as pointless
under that config, even though they're genuinely needed under ruff-strict.toml.
- ANN401: added to ruff.toml's existing `lint.external` list (same mechanism
already used for C901/TID251, enforced by the strict gate but not by this
config) - these Any usages are genuinely dynamic/forwarded, so the
suppression itself is correct and just needed registering.
- S110: fixed the underlying code instead of registering another external
code - the try/except/pass in
CustomStreamWrapper._restore_consumer_correlation_context now logs at
debug level on failure (matching the existing best-effort-cleanup pattern
in _record_partial_usage_for_failure elsewhere in this file), which
satisfies S110's own suggestion directly and needs no suppression at all.
Verified against both ruff.toml and ruff-strict.toml directly, plus all
three other gates (ruff format, type-discipline, basedpyright) and the full
test suite (821 passed).
* fix(lint): scope the ANN401 exemption to file level instead of a repo-wide noqa
Ruff has no per-line-scoped way to register a noqa code across configs (that
requires the default ruff.toml's lint.external list, which is repo-wide in
scope even though the noqa itself is per-line). Since ruff does support
file-level exemptions via per-file-ignores, and ANN401 only needed exempting
in exactly two files, moved the exemption there instead:
- ruff-strict.toml: added [lint.per-file-ignores] disabling ANN401 for
litellm_logging.py and utils.py specifically, with a comment explaining
why (heterogeneous response/forwarded-args parameters with no fitting
concrete type - already verified by trying CostResponseTypes and hitting
a real basedpyright mismatch).
- ruff.toml: reverted the ANN401 entry from lint.external - no longer
needed, since there's no `# noqa: ANN401` left anywhere for RUF100 to
second-guess.
- Removed the now-redundant `# noqa: ANN401` from the 10 affected
parameters in both files, keeping the existing kwargs-ok reasons and
adding a short inline comment on the `result`/`*args` lines pointing at
the ruff-strict.toml exemption for context.
Verified against both configs directly (ANN401 clean under ruff-strict.toml
for these files, RUF100 clean under the default config), all four gates
(ruff format, ruff-strict, type-discipline, basedpyright), and the full
test suite (821 passed).
* fix(logging): redact credential-shaped trace_id/session_id before stamping log records
CorrelationContextFilter stamps trace_id/session_id onto a LogRecord after
SecretRedactionFilter has already run, so a caller-controlled value (e.g. via
x-litellm-trace-id or a W3C baggage header) that happens to look like a real
credential reached JSON and plaintext logs unredacted. Apply the same
credential redaction already used elsewhere in this module at
_sanitize_correlation_id(), the single choke point both set_trace_id() and
set_session_id() route through, so every caller-facing entry point is covered
without depending on filter ordering.
* fix(logging): restore correlation context when a stream's max-duration timeout fires
CustomStreamWrapper.__anext__() called _check_max_streaming_duration() before
entering its try block, so the litellm.Timeout it raises bypassed the except
Exception -> _handle_stream_fallback_error path entirely, leaking the timed-out
stream's own trace_id/session_id into whatever the consumer's task logs next.
Move the check inside the try so it flows through the same restoration path
every other stream failure already uses.
* test(streaming): make dispatch_failure_handlers mock awaitable for the async max-duration test
Moving _check_max_streaming_duration() inside __anext__()'s try block (prior
commit) means a max-duration Timeout now dispatches failure handlers through
the same path every other stream failure already uses, instead of bypassing
it entirely. dispatch_failure_handlers is async on the real Logging class;
the test's plain MagicMock logging_obj made asyncio.create_task() choke on a
non-coroutine return value once that path actually got exercised.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
