* test: point CircleCI-only suites at models still in the cost map
#42435 removed cost map entries past their deprecation date and #42437 added
litellm_uisettings to the config-synced tables, but both only updated
tests/test_litellm. The CircleCI-only suites (local_testing, llm_translation,
logging_callback_tests, litellm_utils_tests, unit) kept using the removed
models or the old table list and went red on main.
Each test keeps its assertions and swaps the removed model for a current one
with the same provider and capabilities. The fireworks tests pick a vision
model from the cost map because #34941 set supports_vision false on
minimax-m3, and the vertex image provider test injects the image model set
because #42435 removed every vertex_ai-image-models entry.
* test(vertex_ai): register the image model through add_known_models in the provider test
* fix(langsmith): json.dumps with default=str so non-serializable metadata does not crash batch flush
Serialize the runs/batch payload with json.dumps(default=str, allow_nan=False) and send it as content= with an explicit Content-Type, so datetime, Decimal and similar metadata values no longer raise TypeError and drop the batch. Forward content= on the AsyncHTTPHandler retry path so a retried batch re-sends the identical body
Replaces #39133, which was cut from the retired staging branch and conflicts with main
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): drop test docstrings and replace monkeypatch with a client-injecting handler
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): add live e2e for non-native metadata reaching LangSmith
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(langsmith): scope the e2e docstring to the values the test injects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(http_handler): close injected retry clients
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(e2e): deselect the LangSmith live e2e on the stage-mirror stack
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Damien Smrt <dsmrt@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
#41569 made SpendLogsMetadata always carry azure_spillover, null unless
Azure reported a spillover, and updated the unit tests that run on the PR
gate. The GCS pub/sub golden only runs on CircleCI's logging lane, so it
kept the old key set and test_async_gcs_pub_sub_v1 has failed on every
run since that merge with an extra metadata.azure_spillover key
_PROXY_MaxBudgetLimiter re-checked spend:user:{id} against user_max_budget in
async_pre_call_hook without the zero-cost model exemption that
_user_max_budget_check applies in auth, so free models were rejected with
"Max budget limit reached." once a user was over budget. Auth already owns
this check, so the hook is deleted rather than taught the exemption again
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The generic API logger batches whatever is queued when it flushes, so records
from tests in other files in the same job land in the same request. Two tests
assumed otherwise: one read actual_request[0], the other counted NDJSON lines,
and both broke whenever another file logged first. Select by the messages each
test sent instead, which keeps the format assertions and stops the order from
deciding the outcome.
Auth gate rejections are raised before add_litellm_data_to_request stamps the caller User-Agent and SpendLogsMetadata dropped the field, so failure spend logs and prometheus labels could not identify an abusive client. Stamp requester_ip_address and user_agent on the failure hook payload and carry user_agent through spend log metadata. Request scopes without a headers entry are tolerated.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Keeps the base's rule that a non-admin id lookup matching no spend-log row answers 403, so the detail route never consults cold storage without an owner row
Two branches independently added embedding_executor to the same fake
search handler in this file, #39472 in the middle of the signature and
#39474 at the end. Neither conflicted with the other, so both edits
merged and the function ended up declaring the parameter twice.
Python rejects that at compile time, so the whole module fails to
import and every test in the file is uncollectable, taking the
logging_testing job down on staging.
Keep the earlier of the two, which sits where the real handler declares
the parameter.
The Router executor only routed a query embedding when the vector store
carried extra embedding configuration, so a store registered with no
embedding model at all always went to the Router and 500'd on the
s3_vectors default text-embedding-3-small when no deployment served it.
Route on whether the Router serves the model, which is the rule the
executor had before, and keep the request metadata on the SDK fallback so
the embedding stays attributed either way.
The base added router_metadata to SpendLogsMetadata in #39001 without
updating this fixture, and its CI run never executed logging_testing,
so the job now fails on every branch merged with current staging.
Two unrelated causes, both leaving staging red with tests that no longer describe
anything true.
#38264 gave LangFuseLogger a langfuse_environment argument and started carrying it
in the credentials dict. The handler test's fake logger did not accept the new
keyword, so constructing it raised TypeError, and four cases in
test_langfuse_unit_tests rebuilt the cache key by hand from three fields and missed
on the four-field key production now writes. Caching itself was never broken: the
handler sets and gets with the same dict. The fake now takes the argument and
asserts it is forwarded, and the cache assertion issues a second identical request
and expects the same logger back, which is the behaviour that matters and cannot
rot the next time a credential field is added.
OpenAI has retired the Assistants API. /v1/assistants and /v1/threads both answer
404 with a valid key, where every live route answers 401, so nothing calling them
can pass again. test_custom_logger_passthrough covered generic passthrough logging
and only used assistants because it is a route with no provider-specific handler;
it moves to /v1/moderations, which is still unclaimed by
_is_supported_openai_endpoint, so the same generic branch is exercised. The two
tests there asserted the same thing against different dead routes, so they collapse
into one. The Ruby suite existed solely to drive assistants, threads, messages and
runs, so it goes along with the RVM and bundler steps that were installed only to
run it, and the two dead OpenAI assistants cases leave
test_openai_assistants_passthrough.
The Azure assistants case in that file stays. Azure runs its own lifecycle and I
could not reach the CI deployment to check whether that API is still there.
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
Four more ruff rules for code the test suite runs but never checks. F601 is the
one that paid: the duplicate key it flagged in a get_form_data fixture was the
mock reproducing the production bug fixed in the previous commit.
B025 removed two unreachable handlers, one of them a pytest.skip shadowed by an
earlier `pass`, so an upstream Vertex flake reported green having asserted
nothing. F632 turned an `is ""` identity check, which passes only on CPython
interning, into the `== ""` it meant. B023 fixed three closures over loop
variables, all latent today but one iteration-order change away from checking the
last case N times.
The auto-router savings figure was computed only inside the spend-update
writer, downstream of where logging callbacks consume the standard logging
payload, so Datadog-style callbacks never received it. Compute it once in
the payload builder, stamp it as a top-level payload field beside
cost_breakdown, thread it into the spend log metadata, and have both
spend-writer call sites read the recorded value with recomputation as the
fallback for rows written before the field shipped. Internal sub-calls
(classifier, shadow eval) are never stamped, and a caller-forged metadata
value is discarded by the unconditional overwrite.
Resolves LIT-5973
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
`pytest.raises(Exception)` with no `match=` passes on any error that broad. A
TypeError from a refactor, a botched fixture, an import that moved: all of them
read as the rejection the test claims to police, so the test goes green for the
wrong reason and stays green after the behaviour it guards is gone.
PT011 closes that gap for the 317 sites B017 could not reach, because B017 only
fires on a single-statement body with no `as e` binding. Each pattern here is the
message the code actually raised, recorded by running the sites under a plugin
that logged the concrete type and text per call site, so the assertions describe
observed behaviour rather than a guess. Where a site raises more than one message
across its parametrize cases, the pattern is an alternation of what was seen;
where the exception carries an empty `str()` and puts the text on `.message`, the
site keeps a narrow `noqa` with the reason.
PT014 removes four parametrize cases that were listed twice. The duplicate re-runs
an assertion that already passed, and it usually marks a case someone meant to
vary and forgot to edit.
* test: point the live web search, groq and vertex image suites at models that still exist
Three CircleCI jobs on the staging-to-main promotion are red because the models
their live suites call have been retired by the providers, not because anything
in litellm changed.
openai/gpt-4o-search-preview now answers "has been deprecated" (its dated id
gpt-4o-search-preview-2025-03-11 carries deprecation_date 2026-07-23), so the
two web search conformance tests and the web search cost tracking test move to
gpt-5-search-api, the current search model. It keeps mode chat,
supports_web_search and a search_context_cost_per_query map, so the cost
assertion still resolves.
groq/llama-3.1-8b-instant reached its deprecation_date of 2026-08-16 and Groq
answers "does not exist or you do not have access to it". It follows
groq/llama-3.3-70b-versatile to groq/openai/gpt-oss-120b, the same replacement
PR #37422 already picked. The proxy config that job boots routes on a */*
wildcard, so no config change is needed.
vertex_ai/imagen-3.0-fast-generate-001 404s with "was not found or your project
does not have access to it". Google retired the whole Imagen family across
Vertex and the Gemini API, so there is no Imagen id left to point at. The class
is removed rather than repointed: Vertex image generation is already covered
live by TestVertexAIGeminiImageGeneration on vertex_ai/gemini-2.5-flash-image,
and the Imagen request and response transformations keep their offline coverage
in tests/test_litellm/llms/vertex_ai/image_generation/.
Only live call sites move. Remaining references to the old ids sit in offline
cost-map and transformation tests, where the string is a lookup key and no
request leaves the process.
* chore(lint): ratchet the TQ005 ceiling down to the count this branch reached
Removing the retired TestVertexImageGeneration class cleared one TQ005
violation, so the gate demands the limit come down with it.
make lint-budget-update only lowers a limit by the delta a branch cleared, and
this ceiling already sat 2 above the base count, so the tool landed on 2834
while the gate wants the limit at or below the 2832 this branch reached. The
remaining 2 are that stale headroom, which is exactly what the gate is asking
to reclaim.
`assert False` inside a `try:` raises AssertionError, which the `except
Exception` right below it catches, so several tests reported green no matter
what the code did. `pytest.fail` raises Failed, a BaseException, and escapes.
A bare `a == b` statement is evaluated and discarded. Nine of those sat in
tests, and one was comparing against a model name the router never produces.
Selects B011, B015, B018, PT015, PLR0133 and PLW0127 in ruff-tests.toml
alongside F821, with all 50 existing violations fixed, so no budget file or
ratchet is needed. CI already runs this config over tests/.
Python binds a name once per scope, so when a module or class defines the same
test twice only the last one exists. The earlier definitions are unreachable:
pytest never collects them, and nothing that references them can fail.
A sweep in August cleared nine of these. Five have appeared since, which is the
argument for a rule rather than another sweep.
Each survivor is the better version, so nothing is lost. The two SQS logger
twins additionally stub `asyncio.create_task`, which the shadowed copies did
not. The cost-calculator duplicate is a two-line stub that also takes a
`model_item` parameter no fixture supplies, so it could not have run even
unshadowed. The two `test_prompt_caching` bodies are both `pass`.
Collecting the four files reports 416 tests before and after.
`tests/proxy_unit_tests/conftest copy.py` goes with them. pytest only loads a
file named exactly `conftest.py`, nothing imports this one, and the space in the
name says what it was.
Two suites broke because they stood in for production objects with stand-ins
that no longer answer the same way.
The redaction test faked a ResponsesAPIResponse and then reassigned
builtins.isinstance so the fake would pass the type check. Redaction now gates
on a tuple of accepted types, and the patched isinstance only recognised the
bare class, so the fake fell through to the generic branch and the assertions
ran against a plain dict. Building a real ResponsesAPIResponse drops the
builtins patch entirely and exercises the same type gate production takes.
The batch rate limiter tests constructed _PROXY_BatchRateLimiter with
parallel_request_limiter=None even though the parameter is not optional. That
stayed harmless until the output-token estimate started reading the limiter,
which turned it into an AttributeError. Inject the limiter the proxy injects,
sharing one InternalUsageCache the way _add_proxy_hooks does.
Greptile flagged that the newly collected SQS tests construct SQSLogger without
mocking asyncio.create_task, so the constructor's periodic_flush task
(while True: sleep; flush_queue) is left running on the session-scoped event
loop. That is correct, and checking each test against the survivor that shadowed
it changes the answer for two of the three.
test_async_log_success_event_adds_to_queue and its failure variant assert exactly
what their survivors assert, that the payload lands in log_queue. The only
difference is whether create_task is mocked, and nothing asserts anything about
that, so restoring them added a leaked task for no coverage. Both renames are
reverted; those definitions stay shadowed and belong in a deletion set instead.
test_async_send_batch keeps its rename. Its assertion, that async_send_message is
not awaited inline, is only meaningful with a real create_task: under a MagicMock
the await count is trivially zero. So it now wraps the real create_task in a spy
that records the tasks and cancels them in a finally block, which covers both the
periodic_flush task and the dispatched send.
Verification against staging for tests/logging_callback_tests/test_sqs_logger.py:
17 passed and 2 "periodic_flush was never awaited" warnings before, 18 passed and
the same 2 after, so the restored test adds no leak. Those 2 warnings are
pre-existing and come from the survivors mocking create_task with MagicMock.
Across the seven touched files, collection goes from 401 to 409 with nothing
lost, and all 409 pass.
Python keeps only the last binding for a name, so when a file defines the same
test twice the earlier one is unreachable. pytest cannot collect a function that
no longer exists, so nothing reports it and the file still looks like it covers
the scenario.
These ten are cases where the two definitions have different bodies, meaning a
real test was replaced rather than duplicated. Each is renamed to say what it
actually covers, which makes it reachable again:
- test_gemini_frequency_penalty: the dead copy checks the parameter is listed in
get_supported_openai_params for vertex_ai; the survivor checks get_optional_params
maps a value for gemini. Different function and different provider.
- test_async_log_success_event_adds_to_queue and the failure variant: the dead
copies run without mocking asyncio.create_task, so they exercise the real task
path the survivors mock out.
- test_async_send_batch_triggers_tasks: the dead copy asserts send is not awaited
directly; the survivor asserts create_task was called.
- test_model_id_in_required_metrics: the dead copy checks the model_id label on
twelve further metrics the survivor dropped.
- test_anthropic_messages_pt_file_block_preserves_cache_control: the dead copy
passes model and llm_provider explicitly and uses real base64 PDF content.
- test_translate_streaming_openai_chunk_to_anthropic_with_thinking: the dead copy
covers thinking_delta; the survivor covers signature_delta.
- test_client_initialization and test_client_without_api_key: the dead copies
assert the resource clients are wired with the right base URL and key; the
survivors only construct the object.
- test_client_initialization_strips_trailing_slash: the dead copy constructs
ModelsManagementClient directly rather than going through Client.
Verification: collecting the seven touched files gives 401 node IDs before and
411 after, the ten new names and nothing else, with nothing lost. All ten pass.
Running the touched files in full gives 299 passed, and test_optional_params.py
goes from 111 passed to 112.
Two further shadowed definitions were left alone rather than renamed: the dead
copies of test_prompt_caching and test_cost_calculator_with_base_model_with_router
have no assertions at all, one being a bare pass and the other a lone import, so
restoring them would add tests that cannot fail.
* feat(logging): add opt-in session_id/trace_id correlation to JSON log records via contextvars
Adds two ContextVar instances (session_id_var, trace_id_var) to litellm/_logging.py and
two setter functions (set_session_id, set_trace_id). Logging.__init__() now calls both
setters after assigning litellm_trace_id so every JSON log record emitted within the
async request context carries trace_id and, when provided, session_id — enabling log
correlation in Loki, CloudWatch Logs Insights, and other structured-log sinks without
any changes to individual log call sites.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(logging): guard session_id/trace_id injection against overwriting caller-supplied extra fields
* fix(logging): always reset session_id_var to empty string when no session_id provided
* feat: gate request correlation IDs in logs behind request_correlation_in_logs flag
* refactor: move correlation ID injection into CorrelationContextFilter
* feat(logging): extend request_correlation_in_logs to plaintext logs and StandardLoggingPayload
Plaintext log lines (json_logs off) now get the same trace_id/session_id
suffix as JSON logs via a new CorrelationPlainFormatter, so the flag has a
visible effect regardless of log format.
StandardLoggingPayload gets a new independent session_id field, populated
from litellm_session_id. trace_id's existing session_id-first fallback is
preserved when request_correlation_in_logs is off; with the flag on, an
explicit litellm_trace_id now takes priority over litellm_session_id so
the two fields carry genuinely independent values.
* fix(logging): restore correlation context after nested calls; sanitize correlation ids
Addresses two review findings on this PR.
CorrelationContextFilter's trace_id/session_id contextvars were set on every
Logging.__init__ but never reset, so a nested LiteLLM call sharing the same
asyncio Task as an outer request (e.g. a guardrail's own LLM-as-judge call,
an MCP sampling call) would leave the outer request's subsequent log lines
stamped with the nested call's ids instead of its own. set_trace_id/
set_session_id now return their contextvars.Token, and Logging stores them
and resets both once its own success/failure handler actually completes,
via a new idempotent _restore_correlation_context() called from all four
terminal handlers.
set_trace_id/set_session_id also now strip control characters and bound
length before storing a caller-controlled trace_id/session_id, since these
values can originate from request input (litellm_session_id, x-litellm-
trace-id) and get interpolated into plain-text log lines - without this, a
caller could embed \r/\n or escape sequences to forge fake log entries.
* fix(logging): restore correlation context after nested calls, not before
The previous commit called _restore_correlation_context() as the first
line of each terminal handler, before that handler's own callback
dispatch loop runs. That's backwards: a nested LiteLLM call triggered
from within a callback (e.g. a guardrail's own LLM-as-judge call) would
then capture the *already-reset* value as its own pre-call baseline,
and its own reset would restore to that instead of the true outer
value - verified live to still leak.
success_handler/async_success_handler/failure_handler/async_failure_handler
are now thin wrappers: the original bodies move to
_success_handler_body/etc, called inside a try/finally that restores
context only once the full body - including any nested calls its own
callback dispatch triggers - has actually finished, mirroring proper
stack-scoped nesting semantics.
* test(logging): cover async_failure_handler's correlation-context restore
Codecov flagged the new async_failure_handler wrapper (try/finally around
_async_failure_handler_body) as uncovered - the method had no direct test
at all before this PR's refactor split it into a wrapper. Adds a test that
awaits it directly and asserts both that async_log_failure_event still
fires and that _restore_correlation_context() puts the pre-call
trace_id/session_id back.
* fix(logging): restore correlation context by value, not by contextvars.Token
veria-ai correctly flagged that contextvars.Token.reset() only works in the
exact Context it was created in, and litellm's async success path (and
streaming failure path) dispatch async_success_handler/async_failure_handler
via asyncio.create_task and the global logging worker - a different Context
than Logging.__init__ ran in. reset_trace_id/reset_session_id silently
swallowed the resulting ValueError, so the restore was a no-op for exactly
those paths. Verified independently: reproduced the raw contextvars
behavior, then confirmed litellm's async success dispatch really does go
through asyncio.create_task + GLOBAL_LOGGING_WORKER (litellm/utils.py).
Logging now captures the pre-call *value* (not a Token) and restores via a
plain set_trace_id()/set_session_id() call, which works regardless of which
Task/Context calls it. reset_trace_id/reset_session_id are removed as
dead/unreliable code. Added a regression test that spawns __init__ and the
restore in different asyncio Tasks - confirmed it fails against the prior
Token-based commit and passes here.
* fix(logging): restore correlation context in the originating task too
Greptile's re-review correctly identified a remaining gap: for a
successful acompletion(), async_success_handler is dispatched via
asyncio.create_task + the global logging worker into a *different* Task
than the one wrapper_async/Logging.__init__ ran in. The prior fix (43c164a)
only restored the handler's own (detached, throwaway) Task - it never
touched the originating request Task, which keeps this call's trace_id/
session_id set for the rest of its own execution (e.g. nested calls made
via the same Task).
wrapper()/wrapper_async() in litellm/utils.py now restore the originating
Task's correlation context in a finally block once the whole call is done,
regardless of what detached logging tasks it spawned along the way. Since
the wrapped body rebinds its own `kwargs` local via function_setup(),
sharing the dict object doesn't work here; a small mutable holder carries
the constructed Logging instance back out to the outer wrapper instead.
_restore_correlation_context() is no longer guarded against repeat calls:
with value-based (not Token-based) restoration, each distinct Task that
calls it needs its own restore to take effect in that Task's own view of
the contextvars, so multiple calls (once per Task involved in an attempt)
are required, not just tolerated.
Added a regression test using mock_response to exercise the real success
dispatch path (asyncio.create_task + GLOBAL_LOGGING_WORKER) without a live
provider call, asserting the *test's own* (originating) task context is
restored after the call - this is exactly the case Greptile flagged and
the prior commit didn't cover.
* fix(logging): restore correlation context when function_setup itself fails
Greptile's 4th finding: if function_setup() constructs Logging() (whose
__init__ already mutates trace_id_var/session_id_var) and then raises
before returning - e.g. update_environment_variables() throws - the
caller's wrapper()/wrapper_async() never receives a logging_obj reference,
so its own restore-on-finally never fires. The correlation ids leak into
every subsequent log line on that thread/task with no way to clear them.
function_setup()'s own except block now restores the context itself in
that case, using whatever logging_obj it managed to construct before
failing (locals().get(), safe against the earlier failure modes where
logging_obj was never assigned at all).
Added a regression test that monkeypatches Logging.update_environment_variables
to raise after construction, confirmed it fails without this fix (the
leaked ids show up directly in the raised exception's own log line) and
passes with it. Broader sweep (test_utils.py, test_router.py,
test_main_module_header.py, streaming handler tests, plus all
logging-specific tests): 722 passed.
* fix(logging): don't assume every litellm_logging_obj is a real Logging instance
CI caught a real regression from the last commit: tests/test_litellm/llms/xai/test_xai_key_fallback.py
injects a minimal FakeLogging stand-in (only implementing
update_from_kwargs) as litellm_logging_obj for a narrow realtime-config
unit test, bypassing the real Logging class entirely. wrapper()/
wrapper_async()'s finally block and function_setup()'s except block both
unconditionally called _restore_correlation_context() on whatever ended up
in the holder, which doesn't exist on that stand-in.
_restore_correlation_context is new plumbing specific to this PR's
feature, not part of any pre-existing stand-in's expected interface, so
callers of it can't assume every object playing the litellm_logging_obj
role implements it. Added _restore_correlation_context_if_supported(),
a small getattr-guarded helper, and used it at all three call sites.
* fix(logging): don't restore context too early on setup failure or streaming
Two more findings from Greptile's 5th review round.
1. function_setup()'s except block restored correlation context *after*
logging the "Error in function_setup" exception, so that diagnostic log
line itself was stamped with the doomed call's ids instead of the outer
ids - misleading, since the failed call never produces anything else to
attribute those ids to. Restore now happens before the log call.
2. wrapper()/wrapper_async() restored the originating task's context as
soon as a streaming call returned, before the caller ever starts
iterating the CustomStreamWrapper it just got back. Any log lines
emitted while iterating (in the same thread/task) incorrectly showed
the pre-call ids instead of this call's own ones. The wrapper finally
block now skips the restore when the return value is a stream wrapper,
deferring to the terminal handler that already fires once the stream is
actually assembled/exhausted.
Both verified with tests that fail against the prior commit and pass
against this one. Broader sweep unchanged at 829 passing.
* fix(logging): best-effort correlation cleanup on abandoned streams
Greptile's 7th finding: if a caller returns a streaming response and never
fully consumes it - stops iterating early, drops the reference, cancels
it - the terminal handler that normally restores the originating task's
trace_id/session_id never fires, since it only runs once the stream is
actually assembled/exhausted. The ids leak into every subsequent log line
in that thread/task with no bound.
There's no reliable Python hook for "this was abandoned without being
closed" - CustomStreamWrapper has no close()/__aexit__/context-manager
convention today, and the only automatic option is __del__, whose timing
is inherently unpredictable (delayed by cyclic GC, not guaranteed at
interpreter shutdown, can run on a different thread). This is a best-effort
safety net, not a guarantee, and is documented as such in the docstring.
Testing this via real garbage collection proved unreliable in practice:
per-chunk logging submits work to a thread pool executor whose worker
thread transiently holds its own bound-method reference to the wrapper
until that task completes, so refcount doesn't hit zero on a
deterministic schedule even with polling. Tests call __del__ directly
instead - a plain method, safe to invoke early - which exercises exactly
the restore logic real garbage collection would eventually trigger,
plus a case confirming a broken logging_obj can never make __del__ raise.
* fix(logging): restore consumer's context at every real stream exit point
Two more findings from this round.
Veria AI: even a *fully consumed* stream never restored the actual
consuming thread/task's correlation context. The terminal success dispatch
(dispatch_success_handlers via asyncio.create_task for async, or
success_handler via the shared executor for sync) only restores whatever
detached context it runs in - never the caller's own thread/task that's
running the for/async for loop. Same root cause as the wrapper-level fix
two rounds ago, just missed for the streaming-completion path.
Greptile: explicit aclose() (client disconnect, router fallback aborting
a partial stream) closed the underlying stream without restoring
correlation context either, since request wrappers intentionally skip
restoration for returned streams and no terminal handler runs on this
path.
Added CustomStreamWrapper._restore_consumer_correlation_context(), called
from every point control genuinely returns to the consumer: the final
raise StopIteration/StopAsyncIteration on natural exhaustion (both sync
branches, both async branches), _handle_stream_fallback_error (the shared
choke point for all three failure-raising call sites), and aclose(). __del__
now delegates to the same helper instead of duplicating it.
Verified with tests extending the existing streaming-exhaustion cases to
assert the consuming context is restored after the loop completes (fails
against the prior commit, passes now), plus a dedicated aclose() test.
Broader sweep: 832 passing.
* fix(logging): don't let a delayed __del__ finalizer clobber a newer active call
If an abandoned stream's __del__ fires late (after cyclic GC delay), a
different call may have already taken over the correlation contextvars in
the same Task/thread. Restoring unconditionally would stomp that active
call's trace_id/session_id with the abandoned stream's stale pre-call
snapshot. __del__ now only restores when the contextvars still hold the
ids this call itself set.
* fix(logging): compare sanitized ids in the __del__ ownership guard
set_trace_id()/set_session_id() sanitize (strip control chars, bound length)
before storing, so the contextvar's value can differ from the raw
litellm_trace_id/litellm_session_id. The __del__ ownership guard was
comparing against the raw values, so a caller-supplied id containing control
characters or exceeding 256 chars would never match, permanently skipping
cleanup. Capture what set_trace_id()/set_session_id() actually stored and
compare against that instead.
* fix(logging): restore consumer context on the synthesized finish_reason chunk
Both __next__ and _finalize_completed_stream() have a branch that fires when
the underlying stream ends without ever emitting an explicit finish_reason
chunk: they synthesize one via finish_reason_handler() and return it. A
consumer that stops as soon as it sees finish_reason - a common pattern -
never calls __next__()/__anext__() again, so the existing restore in the
sent_last_chunk-is-True StopIteration branch never runs for them. The
underlying stream is already exhausted at this point regardless of whether
the caller keeps iterating, so restoring here is safe.
* fix(logging): don't restore correlation context before the caller receives the final chunk
The previous fix (5147c69186) restored context immediately before returning
the synthesized finish_reason chunk from __next__/_finalize_completed_stream,
reasoning that completion_stream was already exhausted. But that chunk is
still this call's own data, and the caller's own application-level log
statements processing it run in the same synchronous frame right after the
return - restoring first made those lines carry the wrong (outer) ids,
exactly what wrapper()/wrapper_async() deliberately avoid by not restoring
while a stream is being iterated.
Revert to not restoring there. A caller that keeps iterating still gets a
correct, deterministic restore on its very next __next__()/__anext__() call
(completion_stream is exhausted, so that immediately re-raises
StopIteration/StopAsyncIteration through the already-restoring branch). A
caller that stops right after finish_reason relies on aclose() or the
best-effort __del__ guard, same as any other stream the caller doesn't fully
exhaust.
* refactor(logging): hoist a safely-hoistable function-body import to module top
CorrelationContextFilter.filter()'s `import litellm` was a function-body
import; verified it can move to module top without a circular-import failure
(litellm/__init__.py already imports from litellm._logging before setting
request_correlation_in_logs, but a bare `import litellm` only binds the
already-in-sys.modules module object - the attribute itself isn't read until
filter() actually runs, by which point litellm is fully initialized).
* test(logging): move correlation tests into their conventionally-mapped files
tests/test_litellm/ mirrors litellm/ in a parallel path. Correlation tests
for the Logging class (litellm_logging.py), function_setup/wrapper_async
(utils.py), and CustomStreamWrapper (streaming_handler.py) had all landed in
test_logging.py, which only maps to litellm/_logging.py itself. Moving each
group to its correctly-mapped file: test_litellm_logging.py (Logging class
init/restore), test_utils.py (function_setup, wrapper_async), and
test_streaming_handler.py (CustomStreamWrapper) in the next commit.
test_logging.py keeps only what actually exercises _logging.py's own
contextvars/filters/formatters/sanitization. No behavior change - same
assertions, same coverage, just relocated.
* fix(logging): restore correlation context unconditionally in wrapper()'s sync path
Blocking finding from review: a caller-visible correlation feature was
silently misattributing one request's logs to a different, unrelated one on
the sync/threaded path. wrapper()/wrapper_async() both left trace_id/session_id
"open" across a stream's entire iteration so the caller's own log lines while
consuming it would carry the right ids. That's safe for wrapper_async(): each
async call gets its own asyncio Task with its own copy of the contextvars,
and Tasks are never recycled across requests, so a leftover value can only
ever affect that one already-abandoned Task.
It is not safe for wrapper() (sync): a plain OS thread has no such per-call
isolation, and a thread pool's worker threads *are* recycled across unrelated
requests. If a sync stream was abandoned (client disconnect, early break, an
uncaught exception) without ever being exhausted or closed, nothing restored
its contextvars, and a pool could later hand that same thread to a completely
different call, which would inherit the abandoned request's ids as its own
"pre-call" baseline and then restore back to that poison when it finished -
permanently misattributing every subsequent log line on that thread,
including its own, to the abandoned request. Strengthening the __del__
finalizer already added for this can't fix it: finalizer timing is exactly
what a permanently-reused thread can't rely on.
wrapper() now restores unconditionally in its own finally, before a sync
stream is ever handed back to the caller. The trade-off: a sync stream
consumer's own application-level log statements while iterating no longer
automatically carry this call's ids (litellm's own internal per-chunk
logging is unaffected, since it's dispatched separately). That's an
acceptable cost for eliminating a silent cross-request misattribution bug.
wrapper_async() keeps the existing conditional (skip-if-streaming) behavior,
justified by the Task-isolation argument above; CustomStreamWrapper's
__del__/aclose()/next-iteration restore machinery remains meaningful and
necessary there.
This also simplifies wrapper()/wrapper_async() back toward their original
shape: both previously used a mutable-dict-holder split into a separate
_body function to smuggle logging_obj/result out to an outer finally,
working around function_setup() rebinding its own local `kwargs`. That
restructuring is no longer needed - `logging_obj` (and, for wrapper_async(),
`result`) were already function-level locals in scope for a plain
try/finally; three of wrapper_async()'s retry-return statements now assign
through `result` first so it accurately reflects what's actually returned
even on a retry path.
Regression test: test_abandoned_sync_stream_does_not_contaminate_a_later_call_on_the_same_thread
in test_streaming_handler.py reproduces the exact reported scenario with a
real single-worker ThreadPoolExecutor - confirmed it fails with the prior
(skip-restore-on-stream) wrapper() and passes with this fix.
* refactor(logging): use Mapping instead of bare dict for read-only params
_get_standard_logging_payload_trace_id/_session_id only read litellm_params
(.get() calls, no mutation) - annotate it as Mapping[str, Any] rather than a
bare mutable dict, per the repo's no-mutable-collection-in-annotation rule.
* fix(logging): scope request_correlation_in_logs to the async/proxy path only
Blocking review finding: wrapper() (the sync entry point) used the same
skip-restore-on-stream design as wrapper_async(), but a plain OS thread has
no per-call context isolation the way an asyncio Task does, and a thread
pool's worker threads are recycled across unrelated requests - an abandoned
sync stream could leave its ids stuck on a thread a pool later hands to a
completely different request, misattributing that request's logs. A fix
existed and was tested (restore unconditionally in wrapper()'s own finally),
but it doesn't benefit this feature's primary consumer - the proxy only ever
calls the async entry point - and carries sync-specific complexity this PR
doesn't need.
Scope the feature to async only instead: Logging.__init__() takes a new
supports_correlation_logging parameter (default True), threaded down from a
new function_setup(..., is_async_call: bool = True) parameter. wrapper() is
the one caller that passes is_async_call=False; every other function_setup()
call site (wrapper_async(), the router, and proxy/MCP-internal call sites)
is already async and keeps the default. With
supports_correlation_logging=False, Logging.__init__() never calls
set_trace_id()/set_session_id() at all, so a sync call has nothing to leak
in the first place. wrapper() reverts to its pre-review shape with no
correlation-specific code at all.
StandardLoggingPayload's own trace_id/session_id fields are unaffected
either way - they're a deterministic per-call read of
self.litellm_trace_id/self.litellm_session_id, not ambient contextvar state,
so they were never exposed to the cross-request bug.
Full sync/direct-SDK support (stamping + its own safe-restore mechanism) is
deferred to a follow-up PR; the fix and its regression test already exist in
this branch's history at commit 9f3a20f4b2 and can be resurrected there.
Tests: replaced the two wrapper()-level tests with ones proving the new
invariant (sync calls, streaming and non-streaming, never touch
trace_id_var/session_id_var even when the caller explicitly passes
litellm_trace_id/litellm_session_id), and added a direct unit test for the
supports_correlation_logging=False gate on Logging.__init__ itself. Verified
live: a real proxy (Postgres-backed, real OpenAI calls) shows clean
trace_id/session_id isolation across two concurrent sessions with no
cross-contamination; a standalone script confirms real sync SDK calls
against a real model never touch the correlation contextvars.
* feat(logging): fall back to W3C traceparent/baggage for trace_id/session_id
request_correlation_in_logs previously only resolved trace_id/session_id from
litellm-specific sources: x-litellm-trace-id/x-litellm-session-id headers, a
generic x-<vendor>-session-id header, or Anthropic-style metadata.user_id. If
none were present, trace_id fell back to an auto-generated UUID unrelated to
anything else, and session_id stayed empty - even when the caller already had
real distributed-tracing instrumentation sending the actual industry-standard
headers for this.
Add a fallback to the W3C Trace Context traceparent header (trace-id
component) and W3C Baggage header (session.id entry), so a request already
carrying real OpenTelemetry trace context correlates litellm's own logs with
the same trace in the caller's observability backend (Datadog, Honeycomb,
Tempo, etc.) instead of getting an unrelated generated id. Precedence is
unchanged for existing sources: explicit litellm headers and the Anthropic
metadata path both still win over this new fallback, which only fires when
neither found anything. trace_id and session_id are resolved independently
here (unlike the existing chain_id mechanism, which uses one shared value for
both), since traceparent and baggage are semantically distinct W3C concepts.
New helpers _trace_id_from_traceparent/_session_id_from_baggage in
litellm_pre_call_utils.py parse the header formats directly (no new
dependency - both are simple fixed-width/delimited strings), wired into
LiteLLMProxyRequestSetup.add_litellm_metadata_from_request_headers() only
when the corresponding litellm_trace_id/litellm_session_id key isn't already
set by the existing paths.
Verified live against a real proxy: a bare traceparent header produces a log
trace_id exactly matching its trace-id component; a traceparent alongside an
explicit x-litellm-trace-id header (different value) produces a log showing
the explicit header's value, proving precedence.
* fix(logging): reserve trace_id/session_id in JsonFormatter against message-content spoofing
JsonFormatter merges keys parsed from the message body before applying extra
record attributes, and the extra-attributes loop skips a key that's already
present. A caller-controlled log message that happens to parse as JSON/dict
with a "trace_id"/"session_id" key (e.g. the proxy logging a raw request-header
dict) could therefore make the JSON record carry the attacker-supplied value
instead of the real correlation context set via CorrelationContextFilter.
trace_id/session_id are now applied from the LogRecord's own attributes after
message-content parsing, unconditionally overwriting anything the message body
claimed for those two keys.
* style(logging): fix import order (ruff I001) in _logging.py and litellm_logging.py
- _logging.py: import litellm belongs after the stdlib from-imports, grouped
with the other litellm.* imports, not before them.
- litellm_logging.py: the refactor to Mapping introduced a second, separate
`from collections.abc import Mapping` instead of merging it into the
existing `from collections.abc import Callable` import.
Caught by the strict-rule budget gate (ruff-strict-budget.json caps I001 at
0 new violations); both auto-fixed with `ruff check --fix --select I001`.
* style(logging): freeze mutable-collection constructions flagged by LIT002
Five sites in this PR's diff built a mutable list/dict literal instead of a
frozen value: a plain list of optional strings in CorrelationPlainFormatter,
a `kwargs or {}` fallback, a `metadata or {}` fallback, two `[...]` candidate
orderings, and a `dict(headers)` copy feeding a dict comprehension. Each is
build-once/read-only, so this rewrites them as tuples, MappingProxyType, or a
plain conditional `.get()` instead of seeding then reading a fresh mutable
collection - no behavior change, confirmed by the existing test suite.
Caught by the type-discipline budget gate (LIT002 capped at 0 new
violations).
* fix(logging): reserve trace_id/session_id even when no correlation context is active
Live-proxy verification surfaced a gap in the earlier message-content-spoofing
fix (7f390a57fc): that fix only overwrites trace_id/session_id from the
LogRecord's own attribute, so it does nothing for a log line emitted before
CorrelationContextFilter has stamped anything on this record (e.g. the
"Request Headers" debug line, which fires before Logging.__init__() runs for
the request). On such a record, a caller-supplied header literally named
trace_id/session_id still got promoted into the JSON output via the embedded
JSON/dict-repr parser, since there was no genuine value to protect.
Fixed at the source: trace_id/session_id are now excluded unconditionally from
the message-content-parsing promotion step, not just superseded afterward.
Verified live against a real proxy - the exact adversarial request (headers
literally named trace_id/session_id) no longer leaks into any JSON log record.
Added a regression test for this no-active-context variant specifically,
confirmed it fails against the prior commit and passes now.
Also fixes an unrelated basedpyright regression from an earlier rebase's
conflict resolution: litellm/utils.py's `logging_obj` was incorrectly
re-annotated `Final` at its second assignment in function_setup() (it's first
declared `None` a few lines earlier), which basedpyright correctly rejects.
* fix(proxy): stop logging the raw W3C baggage session_id value
_session_id_from_baggage() extracts the caller-controlled session.id entry
verbatim - it isn't sanitized until set_session_id() runs later in
Logging.__init__(). The debug log line for this extraction interpolated the
raw value directly, so a caller could embed terminal control characters or
ANSI escape sequences that forge/alter plaintext log output for anyone
tailing the proxy's logs.
Verified live: a baggage header with an embedded ANSI escape reached the
terminal as a real, unescaped control sequence before this fix. Drops the
value from the log line entirely (the extraction succeeding is enough signal
on its own) rather than sanitizing-then-logging, matching veria-ai's
suggestion. Added a regression test using caplog that fails against the prior
commit and passes now.
* fix(logging): restore consumer context only after stream-failure exception mapping
_map_anthropic_exception/_map_aleph_alpha_exception synchronously log a debug
diagnostic (the raw status code) as part of exception_type()'s mapping.
_handle_stream_fallback_error restored the consumer's outer correlation
context before calling exception_type(), so that diagnostic log line carried
the outer (or empty) trace_id/session_id instead of the failing stream's own -
flagged by Greptile.
Moved the restore to run after mapping completes, matching the same
restore-after-not-before pattern already applied elsewhere in this file for
success/finish_reason handling. Added a regression test that captures the
correlation context live during a mocked exception_type() call; fails against
the prior commit, passes now.
* fix(logging): restore consumer context only after aclose()'s stream close completes
aclose() restored the consumer's outer correlation context as its first
statement, before awaiting the underlying provider stream's own aclose()/
close(). If that close attempt raises, the except branch's debug diagnostic
ran under the already-restored outer context instead of the closing stream's
own trace_id/session_id - flagged by Greptile, same restore-too-early pattern
as the stream-failure fix in f1cf9589d6.
Moved the restore to the end of aclose(), after the close attempt (and its
diagnostic logging) completes. Added a regression test with a fake stream
whose aclose() raises, capturing the correlation context live during the
diagnostic log call; fails against the prior commit, passes now.
* style(logging): satisfy new strict-lint budgets introduced upstream (Final, ANN401, S110, TRY300, kwargs typing)
Rebasing onto litellm_internal_staging pulled in 116 upstream commits that
introduced/tightened several lint gates this PR's own code now trips:
- LIT010 (every local/module-level variable must be Final): added Final
annotations across _logging.py, litellm_logging.py, streaming_handler.py,
litellm_pre_call_utils.py, and utils.py. Where a name is genuinely
reassigned (logging_obj: starts None, later set to the real object) or
branch-assigned, either restructured into a single ternary expression
(ordered_candidates) or suppressed with `# rebind-ok: <reason>` matching
this repo's documented escape hatch.
- LIT011 (parameter mutation): suppressed the two new `data[key] = value`
writes in litellm_pre_call_utils.py with `# rebind-ok`, matching the
unsuppressed precedent already used for every other `data[...]` write in
the same function - `data` is an intentional out-param there.
- ANN001/ANN003/ANN202 (missing parameter/return type annotations): fully
typed success_handler/_success_handler_body, their async twins, and
failure_handler/_failure_handler_body/async variants in litellm_logging.py,
plus function_setup in utils.py (added Rules to its existing TYPE_CHECKING
block for the rules_obj: Rules annotation).
- ANN401 (explicit Any disallowed): suppressed with `# noqa: ANN401` on the
handful of genuinely-heterogeneous result/*args/**kwargs parameters, since
ordinary suppression is this repo's documented path.
- S110 (try/except/pass): added to the existing BLE001 noqa on the one
best-effort correlation-cleanup try/except this PR added.
- TRY300 (return inside try): moved two `return result` statements into
`else:` blocks in the retry-fallback paths this PR's own diff touched.
- reportPrivateUsage (basedpyright): renamed the two new
StandardLoggingPayloadSetup static methods (get_standard_logging_payload_
trace_id/session_id) to drop their leading underscore, since they're
genuinely called from a sibling module-level function in the same file.
No behavior change - confirmed by the full existing test suite (819 passed)
plus all four lint gates (ruff format, ruff-strict, type-discipline,
basedpyright) passing clean.
* fix(lint): stop RUF100 flagging noqa suppressions the strict gate needs
CI's plain "ruff check" job uses the default ruff.toml, a narrower config
than ruff-strict.toml (used only by the strict-rule budget gate). ANN401 and
S110 aren't enabled in the default config, so RUF100 (unused-noqa) flagged
the `# noqa: ANN401`/`# noqa: ...,S110` suppressions this PR added as pointless
under that config, even though they're genuinely needed under ruff-strict.toml.
- ANN401: added to ruff.toml's existing `lint.external` list (same mechanism
already used for C901/TID251, enforced by the strict gate but not by this
config) - these Any usages are genuinely dynamic/forwarded, so the
suppression itself is correct and just needed registering.
- S110: fixed the underlying code instead of registering another external
code - the try/except/pass in
CustomStreamWrapper._restore_consumer_correlation_context now logs at
debug level on failure (matching the existing best-effort-cleanup pattern
in _record_partial_usage_for_failure elsewhere in this file), which
satisfies S110's own suggestion directly and needs no suppression at all.
Verified against both ruff.toml and ruff-strict.toml directly, plus all
three other gates (ruff format, type-discipline, basedpyright) and the full
test suite (821 passed).
* fix(lint): scope the ANN401 exemption to file level instead of a repo-wide noqa
Ruff has no per-line-scoped way to register a noqa code across configs (that
requires the default ruff.toml's lint.external list, which is repo-wide in
scope even though the noqa itself is per-line). Since ruff does support
file-level exemptions via per-file-ignores, and ANN401 only needed exempting
in exactly two files, moved the exemption there instead:
- ruff-strict.toml: added [lint.per-file-ignores] disabling ANN401 for
litellm_logging.py and utils.py specifically, with a comment explaining
why (heterogeneous response/forwarded-args parameters with no fitting
concrete type - already verified by trying CostResponseTypes and hitting
a real basedpyright mismatch).
- ruff.toml: reverted the ANN401 entry from lint.external - no longer
needed, since there's no `# noqa: ANN401` left anywhere for RUF100 to
second-guess.
- Removed the now-redundant `# noqa: ANN401` from the 10 affected
parameters in both files, keeping the existing kwargs-ok reasons and
adding a short inline comment on the `result`/`*args` lines pointing at
the ruff-strict.toml exemption for context.
Verified against both configs directly (ANN401 clean under ruff-strict.toml
for these files, RUF100 clean under the default config), all four gates
(ruff format, ruff-strict, type-discipline, basedpyright), and the full
test suite (821 passed).
* fix(logging): redact credential-shaped trace_id/session_id before stamping log records
CorrelationContextFilter stamps trace_id/session_id onto a LogRecord after
SecretRedactionFilter has already run, so a caller-controlled value (e.g. via
x-litellm-trace-id or a W3C baggage header) that happens to look like a real
credential reached JSON and plaintext logs unredacted. Apply the same
credential redaction already used elsewhere in this module at
_sanitize_correlation_id(), the single choke point both set_trace_id() and
set_session_id() route through, so every caller-facing entry point is covered
without depending on filter ordering.
* fix(logging): restore correlation context when a stream's max-duration timeout fires
CustomStreamWrapper.__anext__() called _check_max_streaming_duration() before
entering its try block, so the litellm.Timeout it raises bypassed the except
Exception -> _handle_stream_fallback_error path entirely, leaking the timed-out
stream's own trace_id/session_id into whatever the consumer's task logs next.
Move the check inside the try so it flows through the same restoration path
every other stream failure already uses.
* test(streaming): make dispatch_failure_handlers mock awaitable for the async max-duration test
Moving _check_max_streaming_duration() inside __anext__()'s try block (prior
commit) means a max-duration Timeout now dispatches failure handlers through
the same path every other stream failure already uses, instead of bypassing
it entirely. dispatch_failure_handlers is async on the real Logging class;
the test's plain MagicMock logging_obj made asyncio.create_task() choke on a
non-coroutine return value once that path actually got exercised.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The spend-log metadata schema gained a compression_savings key, so the
gcs pubsub v1 payload now carries it. The golden fixture was never
updated, and the comparator flags any key present in the payload but
absent from the fixture, so test_async_gcs_pub_sub_v1 failed on every
run. Pin the key as null rather than adding it to ignored_keys; the
value is deterministic on this path, so ignoring it would leave the
assertion blind to the field entirely.
* fix response not being redacted for custom callbacks with streaming enabled
* reduce code duplication
* add unit test
* fix: resolve lint violations in adopted redaction fix
* fix: scope streaming response redaction to the opted-out custom logger
---------
Co-authored-by: Moritz Müller <moritz.mueller2@tu-dresden.de>
The security fix in 34e9be1ba7 removed turn_off_message_logging from
_supported_callback_params to stop callers bypassing global redaction via
the request body. That also killed the documented admin-only per-key or
per-team override because both flows resolve through the same allowlist
in initialize_standard_callback_dynamic_params.
Put turn_off_message_logging back in _supported_callback_params so an
admin-configured metadata.logging[].callback_vars.turn_off_message_logging
survives into StandardCallbackDynamicParams and can override the global
setting for that key or team, as documented at
docs/proxy/team_logging#disableenable-message-redaction.
Consolidate the metadata traversal so the extractor and the proxy strip
walk the same set of client-controllable slots. iter_client_callback_metadata_dicts
in litellm_core_utils/initialize_dynamic_callback_params.py is the single
source of truth for metadata, litellm_metadata, and litellm_params.metadata;
_strip_client_message_redaction_opt_out imports it so a future addition
to one side automatically reaches the other. The extractor iterates the
helper in reversed order so litellm_params.metadata keeps overriding
metadata, matching the pre-refactor merge precedence.
Client bypass stays blocked. Restoring the field re-enrolls it in the
auth layer's _BANNED_REQUEST_BODY_PARAMS (derived from
_supported_callback_params via _build_banned_observability_params), so
client submissions at the top level, inside metadata, or inside a
JSON-string litellm_metadata all 401 at ingress. is_request_body_safe
also now descends into litellm_params.metadata for the same 401 defense
against the nested-body attack vector, matching how the metadata and
litellm_metadata slots are handled. _strip_client_message_redaction_opt_out
runs after the litellm_metadata JSON parse and before the admin callback_vars
unpack, so admin values survive while any leftover client-supplied
opt-out is dropped when global redaction is on and the key or team
lacks allow_client_message_redaction_opt_out.
Flip the two dynamic-param e2e tests added by the security fix to
reflect the restored override behavior, keeping the invariant that
proxy client bypass is stopped by the auth layer 401 above.
Co-authored-by: yucheng <yucheng@yuchengs-MBP.attlocal.net>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* fix(logging): resolve model_map_value for proxy custom pricing
Use deployment model for standard logging cost-map lookup when the router overrides response.model to a group alias, and flush stdout when printing the payload.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(logging): add comment and test for deployment fallback in standard logging payload
Address review: explain why the metadata["deployment"] fallback is unconditional,
and add a test covering the get_standard_logging_object_payload code path.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(test): update model_map_key assertion for provider-prefixed keys
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(logging): scope base_model to model param only under custom_pricing
Passing model=base_model unconditionally caused _get_provider_for_cost_calc
to infer and prepend a provider prefix on all non-custom-pricing calls,
changing model_map_key for existing deployments. Scope it to custom_pricing=True
where the fix is actually needed.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
* feat(spend): store litellm_call_id on spend logs for DB-to-trace correlation
Successful spend logs keyed request_id to the provider response id while
tracing uses x-litellm-call-id, so a DB row could not be correlated with its
trace; this only worked for failures, where request_id already fell back to
the call id. Add a nullable litellm_call_id column to LiteLLM_SpendLogs,
populate it in get_logging_payload, and surface it in the spend logs read
endpoints so correlation works both directions for successful calls
Fixes LIT-3868
* chore: sync schema.prisma copies from root
* test(spend): cover cache-hit and missing-response-id paths for litellm_call_id
Lock the intended behavior surfaced in review: on a cache hit request_id gets
the uniqueness suffix while litellm_call_id stays the raw call id, and when the
provider returns no id request_id falls back to the call id so both columns
match. Both assertions fail when the populate line is reverted
* test(spend): ignore litellm_call_id in spend logs payload comparisons
get_logging_payload now always writes litellm_call_id, so the full-payload
comparisons in test_spend_management_endpoints.py saw an unexpected key and
failed. litellm_call_id is a per-request runtime uuid like request_id, which
is already ignored, so add it to ignored_keys
* test(logging): ignore litellm_call_id in gcs pubsub spend logs comparison
The gcs pubsub spend logs payload comparison flags any key present in the
actual payload but absent from the golden snapshot. get_logging_payload now
always emits litellm_call_id, a per-request runtime uuid like request_id which
is already ignored, so add it to ignored_keys
* refactor(spend): store litellm_call_id in spend log metadata, drop column
Switch DB-to-trace correlation off a dedicated column and onto the existing
metadata JSON, avoiding a schema migration entirely. litellm_call_id is now
written into spend log metadata (already selected and re-hydrated on the read
paths) instead of a new LiteLLM_SpendLogs column, so the three schema.prisma
copies and the migration are reverted and the read SELECTs go back to their
original form. Correlation is queryable via metadata->>'litellm_call_id'
Trade-off: an unindexed JSON lookup rather than an indexed column; acceptable
for this use case and removes all migration risk
* refactor(spend): thread litellm_call_id into _get_spend_logs_metadata
Set litellm_call_id beside the other computed metadata values inside
_get_spend_logs_metadata rather than mutating clean_metadata back in the
caller, matching how applied_guardrails, cost_breakdown and the rest are
threaded. No behavior change; the value still comes from kwargs with a
litellm_params fallback
---------
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
