The "Final returned optional params" line printed whatever the caller nested inside
extra_body, so a credential tucked in there reached stdout in plaintext one line after
the request line that already redacts it.
The call site now runs redact_credentials_in_payload behind a guard reading both of
print_verbose's consumers, litellm.set_verbose and the LiteLLM logger's DEBUG level, so
the line prints in exactly the cases it did before and the walk costs nothing when
nothing would read it.
redact_credentials_in_payload only recursed into mappings, so a
credential-named key one level inside a list or tuple, the shape
extra_body and metadata routinely carry, still reached stdout under
set_verbose. Rebuild sequences element by element too, keeping the
container's own type so the printed repr is unchanged apart from the
secret.
`litellm.set_verbose = True` printed the caller's kwargs verbatim to stdout, so
`api_key` and its siblings landed in terminals and container log drains in
plaintext while the same statement's logger emission was already redacted.
Mask the kwargs at the source with a shared helper in
`litellm_core_utils/sensitive_data_masker.py`, reusing the existing
`SensitiveDataMasker` key classification and the `REDACTED` marker
`secret_redaction.py` already owns, so both debug surfaces agree.
get_api_key had no callers. main.py imported it without using it, and
because main.py declares no __all__, the star import in __init__.py
published it as litellm.get_api_key. It duplicated key resolution that
get_llm_provider_logic already performs, which is how a misspelled env
var survived in it unnoticed until #35985. Drop the definition, the
unused import, the test that pinned the ai21 branch, and ratchet the
lint budgets down by the violations it carried.
Resolves LIT-5245
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Gemini 3.8 Flash launches today with the same promotional pricing, limits,
and thinking settings as Gemini 3.7 Flash, so the gemini/, vertex_ai/, and
bare cost map entries mirror the 3.7 Flash ones. Regression tests lock the
launch prices, the 4096-token cache minimum, and the gemini-3 thought
signature gate in for the new model.
Any registered guardrail made provider_specific_fields.search_results
vanish from /v1/chat/completions vector store responses, even when the
guardrail never ran. Two defects combined:
- CustomGuardrail.async_post_call_success_deployment_hook returned the
response instead of None when it did not run, claiming a modification
it never made
- the async_post_call_success_deployment_hook dispatcher in utils.py
returned at the first non-None callback result, so the lazily appended
VectorStorePreCallHook never got a chance to attach search_results
The hook now returns None when it does not run, and the dispatcher
chains non-None results through the remaining callbacks, matching the
pre-call dispatcher's behavior
The registry key was never copied into ModelInfo, so /v1/model/info reported
null for every model, /model_group/info reported false for every group, and
litellm.supports_parallel_function_calling() returned False for provider-prefixed
entries that declare true. Copy it like every other capability flag and pin the
three surfaces with regression tests.
Resolves LIT-6340
A gpt-5 model accepts a non-default temperature only while its effective reasoning
effort resolves to "none". litellm had no representation of the effort a model applies
when the request omits reasoning_effort, so it substituted supports_none_reasoning_effort,
which is a different fact. Every model that supports "none" without defaulting to it
therefore had temperature forwarded and rejected upstream, and because the carve-out
returned before the drop_params branch, drop_params: true could not save it.
Declare the fact instead. A new cost-map key, default_reasoning_effort, states the effort
the provider applies when the request omits one, and one shared predicate resolves the
effective effort from it: an explicit reasoning_effort wins, otherwise the declared
default, otherwise the catalogue decides.
That last step matters because the cost map is fetched from the published branch at import
time, so it can be OLDER than the code reading it. On such a map every model looks
undeclared, and reading that as "reasoning is active" would strip temperature from the 39
gpt-5.1/5.2/5.4 entries that accept it, a regression caused by data lag rather than by
anything about the model. So an absent declaration is only meaningful once the catalogue
carries the key at all; a map that predates the feature keeps the answer litellm gave
before it existed, and the conservative answer applies from the moment the data lands.
The top_p/logprobs/top_logprobs gate carried the same assumption spelled differently and
now shares the predicate, as does the Responses API, which reimplemented the rule and is
what the default /v1/messages bridge routes openai models through. Azure normalises its
routing names in one resolver that every capability lookup goes through, which replaces
its bespoke per-lookup rewrite.
Declared on the 37 gpt-5.1/5.2/5.4 entries measured to accept temperature=0 today, so
their behaviour is unchanged. The 23 gpt-5.5/5.6 entries that reject it stay undeclared
and are fixed once the catalogue carries the key.
Resolves LIT-3797
Resolves LIT-5028
Kimi K3 accepts exactly low, high and max, defaults to max, and always thinks.
The map could not say that: medium and high have no supports_*_reasoning_effort
flag because every other reasoning model takes them, so the ten kimi-k3 entries
carried supports_reasoning alone and resolved to unknown. The dashboard then fell
back to a capability-blind level list that deliberately omits max, which is why a
kimi-k3 tier cannot be set to max thinking today.
Add reasoning_effort_levels, an array key in the shape the map already uses for
supported_endpoints and supported_modalities. Where present it is read first and
wins whole; every other entry keeps answering through the per-level flags,
unchanged. It is deliberately a different name from the computed
ModelGroupInfo.supported_reasoning_efforts, which stays derived from a group's
deployments and is never seeded from one deployment's model_info.
The levels are per entry rather than per model, because the deployments differ:
Moonshot, Together, Fireworks and Azure Foundry all forward the level unchanged
and get the model's own low/high/max, while Perplexity documents a six-value
enum it maps down internally and gets that. The /v1/messages degradation chain
consults the same declaration, so the level the map advertises is the level that
path forwards.
The realtime health check always built the Azure websocket URL with the
default beta protocol, so GA-only transcription models such as
azure/gpt-realtime-whisper got probed at /openai/realtime and were
rejected with HTTP 400 on every /health run, while real calls through
the proxy resolved the GA path via intent=transcription and worked.
The probe now resolves the protocol the way the real call path does:
an explicit realtime_protocol (argument, deployment litellm_params, or
LITELLM_AZURE_REALTIME_PROTOCOL) wins, transcription-only models fall
back to GA with intent=transcription, and everything else keeps beta.
Transcription-only detection reads both mode and supported_endpoints
from get_model_info because a live proxy overwrites the catalog mode
with the operator's deployment model_info (mode: realtime) during
router registration, while supported_endpoints survives it.
get_model_info now propagates supported_endpoints from the cost map;
it declared the field but never populated it.
* feat(logging): add async_post_call_failure_deployment_hook
CustomLogger already has async_pre_call_deployment_hook and
async_post_call_success_deployment_hook, both firing once per real
deployment attempt from wrapper_async since the router re-enters that
wrapper fresh on every retry and fallback step. There was no failure-side
counterpart; the only failure signal, async_log_failure_event, fires once
per logical client request behind a dedup gate, so fallback chain attempts
2+ were invisible to callbacks needing per-deployment-attempt granularity.
Adds async_post_call_failure_deployment_hook(request_data, exception,
call_type) to CustomLogger and a matching dispatcher in utils.py, called
from wrapper_async's except block. It needs no dedup coordination since
each real attempt naturally re-enters the wrapper once. Unlike its two
siblings, the dispatcher wraps each callback call in its own try/except
since it runs on the wrapper's own exception path and a broken callback
must never mask the exception about to be re-raised to the caller.
* feat(logging): pass fallback_depth through to async_post_call_failure_deployment_hook
Router already tracks fallback_depth internally on each fallback hop
(litellm/router_utils/fallback_event_handlers.py), incrementing it once per
target tried, but nothing surfaced it to CustomLogger callbacks. Reads it
off request_data in the dispatcher and passes it through as a best-effort
int | None keyword: None on the first, pre-fallback attempt or a bare SDK
call with no router, 1 on the first fallback hop, 2 on the second, and so
on. Verified live against a real multi-hop Router fallback chain before
adding the regression tests.
* fix(logging): fire async_post_call_failure_deployment_hook on internal calls too
The failure hook was gated behind the same not _is_litellm_internal_call
check as the request-level dedup-gated failure logging, so a failed
internal sub-call (e.g. an emulated file-search step) never reached it,
even though its async_pre_call_deployment_hook and
async_post_call_success_deployment_hook siblings already fire
unconditionally for such calls.
* chore: retrigger CI (lint job hit a transient GitHub Actions infra outage on the prior push)
* chore: retrigger CI (lint job hit the same GitHub Actions infra outage again)
* fix(logging): scope async_post_call_failure_deployment_hook to the actual model call
The hook was dispatched from the wrapper's broad outer except, which also
catches BudgetExceededError (raised before any deployment attempt),
errors from async_pre_call_deployment_hook, and errors raised after a
successful model call (post_call_processing, async_post_call_success_deployment_hook,
caching). None of those are a deployment attempt failing, so the hook
misreported them as one.
Scoped the hook to a try/except around the model call itself, so it only
fires when that specific call raises, matching its own documented contract.
* test: assert the callback actually ran in the failure-hook error-isolation test
An upstream test-quality gate (TQ001) flagged this test for asserting
nothing, so it could only fail by raising. Track whether the exploding
callback actually ran and assert on it, so the test would catch a
dispatcher that silently skipped every callback instead of isolating a
raising one.
* fix(logging): harden async_post_call_failure_deployment_hook against 5 maintainer-verified issues
A maintainer's live-proxy A/B review against base found five real
problems with the failure hook, all reproduced and fixed:
- The dispatcher called overrides with fallback_depth as a required
keyword, so an override matching this PR's own earlier 3-arg
proof-of-fix example raised TypeError, swallowed at debug level, on
every call. Now checks the override's signature once per class and
omits the keyword when unsupported.
- A callback mutating the exception it receives (e.g. status_code)
changed what the real caller got back, since it was the same live
object about to be re-raised. Callbacks now receive a same-class
snapshot instead.
- request_data exposed attempted_targets, the router's own live
fallback-walk bookkeeping shared by reference across every hop, so a
callback calling .record() on it could make the router skip a
deployment it never actually tried. Now excluded from what the hook
receives.
- The hook's own await sat directly in the model-call except block, so
a caller-side cancellation landing mid-await (e.g. asyncio.wait_for)
replaced the real deployment exception with CancelledError/
TimeoutError. Now isolated so hook dispatch can never mask the real
failure.
- The timestamp used for the reported failure duration was captured
after the hook ran, so a slow callback inflated
async_log_failure_event's duration. Now captured before the hook
dispatches.
* fix(logging): preserve traceback/cause/context on the failure-hook exception snapshot
Bugbot found a real gap in the previous round's exception-mutation fix:
_snapshot_exception_for_hook only copied __dict__ and args, so a
callback formatting or inspecting the failure chain saw an empty
traceback and lost chained-exception context, even though the live
exception still has them. __traceback__/__cause__/__context__ aren't
stored in __dict__, so they need copying explicitly.
* fix(logging): preserve __suppress_context__ on the failure-hook exception snapshot
Setting __cause__ has a documented CPython side effect of implicitly
forcing __suppress_context__ to True. Since the previous round's
traceback fix set __cause__ before __suppress_context__, a normal
implicit-chaining exception (no `raise ... from`, __suppress_context__
naturally False) got its context wrongly suppressed on the snapshot.
Now __suppress_context__ is set explicitly, after __cause__, so it
always reflects the real exception.
* fix(logging): use MappingProxyType for the failure-hook's sanitized request_data
A LIT002 budget check (surfaced by rebasing onto a moved base) flagged
the dict comprehension building safe_request_data as mutable
construction. MappingProxyType is also a strictly better fit here: a
genuinely read-only view, not just an immutable-looking dict, matching
the intent that callbacks should never be able to mutate what they're
handed.
---------
Co-authored-by: Deepanshu <deepanshu.lulla@alpha-sense.com>
Follow-up to #38130. The function has no callers in the repo or the docs and is
not exported from `litellm/__init__.py`, and `token_counter` already does the same
job better, so keeping a second entry point only preserves a trap.
That trap is real: Greptile flagged on #38130 that `token_counter` picks the claude
tokenizer only for bare ids. `claude-sonnet-4-5` resolves to huggingface_tokenizer,
while `claude-3-opus-20240229` and `anthropic/claude-sonnet-4-5` fall back to the
OpenAI one, 24 tokens against 27 on the same string. Deleting the wrapper removes
the surface rather than papering over it; the selection gap in `token_counter`
itself is worth its own fix.
BREAKING CHANGE: `from litellm.utils import prompt_token_calculator` no longer
resolves. Use `litellm.token_counter(model=..., text=...)`.
The claude branch called the anthropic SDK's `Anthropic().count_tokens`, which the
SDK removed, so every claude call raised AttributeError. Counting now goes through
litellm's own token_counter, which handles anthropic models offline and drops the
SDK dependency entirely.
Hiding that was a swallowed error: `except Exception: Exception("Anthropic import
failed please run `pip install anthropic`")` built the exception without raising
it, so an environment missing the SDK fell through to the unguarded
`from anthropic import ...` on the next line and got a bare ModuleNotFoundError
instead of the install hint.
That was the codebase's last PLW0133, so the rule graduates from the ratcheted
budget into ruff.toml where it hard-fails, and editors get the diagnostic inline.
Regenerate model_prices_and_context_window.schema.json and add the flag to
the inline validator schema in test_utils.py so the new cost-map key passes
validate-model-prices-json and the JSON-valid test.
Six defects in the RunwayML video provider:
- transform_video_create_request hardcoded /image_to_video, so text-to-video 400'd and video-to-video was unreachable; the endpoint is now selected from the inputs present (promptVideo/videoUri, promptImage, or text only)
- get_error_class raised instead of returning, turning a provider 4xx into a proxy 500 APIConnectionError; it now returns a RunwayMLError
- VideoObject.progress was typed int while Runway sends a 0..1 float, 500'ing status polls while RUNNING; it is now scaled to a 0..100 percent
- custom per-deployment pricing stored under litellm_metadata was ignored for video; the deployment model_info lookup now checks both metadata keys
- stale cost-map entries (gen3a_turbo, gen4_aleph) were removed and current models added, with output_cost_per_second_480p/_4k tier keys plumbed through the model-info and router types
- video cost now falls back to Runway's estimatedCost from the create response when no custom pricing is configured, and custom pricing always wins over it
Fixes#36483
Tencent chat completions route through the OpenAI SDK's
chat.completions.create(), which raises TypeError on unknown kwargs -
so a top-level 'thinking' optional param crashed every reasoning
request with a 500 before any HTTP call was made.
Nest the resolved thinking object in extra_body instead: the SDK merges
extra_body into the top-level JSON payload, so TokenHub still receives
the documented thinking field (type/budget_tokens) in the request body.
Also align the param mapping with TokenHub's documented behavior:
- reasoning_effort="none" now maps to thinking={"type": "disabled"}
instead of being dropped (deepseek-v4-* default to thinking enabled,
so dropping it never actually disabled thinking)
- MiniMax models only accept thinking.type "adaptive"/"disabled",
so "enabled" is coerced to "adaptive" instead of returning a 400
Refs: https://www.tencentcloud.com/document/product/1300/82345
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
* test: use monkeypatch.setenv for env writes in tests/test_litellm
`os.environ["X"] = v` inside a test leaks the value into every test that runs
after it in the same worker, so ordering decides the result. 262 of those
writes across 40 files now go through pytest's `monkeypatch` fixture, which
restores the previous value at teardown.
The rewrite skips any test that a mock.patch-family decorator wraps, any test
with defaulted positional parameters, any test whose own name is called
directly elsewhere, and rebinds nothing inside nested defs, because in each of
those cases appending a fixture parameter changes what pytest or mock binds.
Ratchets the TQ004 ceiling from 768 to 506.
* fix(test): delete the key through monkeypatch instead of popping it first
Five tests popped a key straight out of `os.environ`, ran, then restored it with
`monkeypatch.setenv`. By the time monkeypatch saw the name it was already gone,
so it recorded "absent" as the value to go back to and deleted the key at
teardown. On a worker that inherited a real `RESEND_API_KEY`, `SENDGRID_API_KEY`,
`UI_PASSWORD`, `LITELLM_SALT_KEY` or `OPENAI_API_KEY`, every test after the first
one ran without it.
`monkeypatch.delenv(..., raising=False)` removes the key and restores whatever
was there, so the try/finally the manual restore needed goes with it.
* chore(test): leave the two cost-calc files to the PR that rewrites them fully
Both files are also in #37815, which converts the module-global writes as well
as the env writes and folds them into one fixture. Two PRs rewriting the same
lines differently is a conflict nobody benefits from resolving, so this one
drops back to staging on those two and keeps the other 39.
TQ004 clears 200 here instead of 275; the rest moves with #37815.
