litellm/tests
Nate Armstrong cd2e4b6b60
feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526)
* feat(custom-logger-v2): add types, dispatcher and route callback call sites (step 0)

* feat(custom-logger-v2): add the rust contract for native callbacks and bind the proxy request scope

* fix(custom-logger-v2): add field projections and keep the request scope across copies

* fix(custom-logger-v2): pass the request scope to the async failure attempt_ended line

* fix(custom-logger-v2): add v2 callbacks from litellm.callbacks to the success and failure lists

_safe_add_callback_to_list dropped CustomLoggerV2 instances, so a V2 callback registered through litellm.callbacks got provider events but never log_prepare or attempt_ended. The manager now adds each V2 instance once by identity, and proxy startup fans V2 callbacks out like CustomLogger ones.

* fix(custom-logger-v2): contain a failing v2 callback on logging hooks

A V2 log_prepare writer that raised, or returned Abort or Respond, ended the line and escaped async_success_handler, so attempt_ended delivery, sync-for-async scheduling and the Redis flush were skipped. On hooks off the request path a V2 entry's exception or outcome is now counted and logged like a legacy logger's exception, and the line goes on. ContractViolation and BaseException still propagate, and legacy entries are unchanged.

* fix(custom-logger-v2): run blocking and awaited v2 callbacks on synchronous logging lines

run_plan_sync raised ContractViolation for a V2 log_prepare writer or any awaited V2 subscriber, the logging plane swallowed it, and every legacy logger at that hook was skipped. Legacy entries now always run in order as before. A V2 writer or awaited subscriber runs inline in list order through asyncio.run when the thread has no running event loop, so sync SDK calls apply its patch and finish it before the handler returns. On a thread with a running loop it cannot block, so it is delivered off-path and its patch is not applied. Lines without such entries take the unchanged legacy path.

* fix(custom-logger-v2): map a v2 abort to its http error at every proxy hook point

Only pre_call turned a V2 Abort into an HTTP error. during_call, post_call_success, post_mcp_call, list_models and the deferred stream hook raised V2Outcome, which reached the client as a 500, and post_call_failure, which never raised before, raised it too. Abort now raises the exception it carries or an HTTPException with its status, message and headers at those points, and at post_call_failure it becomes the transformed error through keep_first. Respond is still not wired and keeps raising V2Outcome. The pre_call routing test is updated to the HTTPException raise_v2_abort already makes.

* fix(custom-logger-v2): apply no-log to v2 logging subscribers without a request scope

The proxy request scope already carries no-log and caller opt-outs, so a V2 callback that honors no-log is skipped on proxy requests. SDK calls have no scope, and the logging plane ran V2 subscribers with an empty selection, so a no-log call was still delivered to them. Without a bound scope the logging lines now open one with no_log set when the call's litellm_params carry no-log, unless global_disable_no_log_param is on. Callbacks that do not honor no-log, and so every writer or abortable, are unaffected.

* fix(custom-logger-v2): deliver a failed attempt to v2 subscribers once per logging object

Callers such as MCP tool errors and Responses streaming run both failure handlers without for_async_request, so a V2 attempt_ended subscriber, now present in both failure lists, got the failure twice. The first failure handler to include V2 subscribers claims the delivery on the Logging object and the other skips them. Legacy callbacks are unchanged.

* refactor(custom-logger-v2): settle a synchronous v2 run through the shared gather and use aware times in the new tests

Drops the blind except the sync line used to capture a V2 callback's exception, reusing _gather_started, and gives the new logging tests timezone-aware times, so the strict ruff counts do not grow.

* fix(custom_logger_v2): await a single-entry v2 stage in the caller's context

A native handler in its own stage ran inside an asyncio.gather child task, so a ContextVar it set was lost when the stage ended, while a legacy callback's stays visible to later callbacks and the rest of the request. A stage with one v2 entry is now awaited directly, with gather's exception handling kept. Multi-entry parallel stages still gather.

* fix(custom-logger-v2): raise a native callback panic as a runtime error

* fix(custom-logger-v2): run a synchronous v2 line on a private event loop

* fix(custom-logger-v2): deliver an mcp tool failure to v2 subscribers from the async failure handler

* fix(custom-logger-v2): raise an abort's carried exception at the deployment hooks and keep the original failure at post_deployment_failure

* fix(custom-logger-v2): apply caller opt-outs to v2 logging lines without a bound request scope

* fix(custom-logger-v2): dedupe v2 callbacks in the event lists by logger key so a native built twice is delivered once

* fix(custom-logger-v2): log an unregistered abort error kind and keep the carried traceback at post_call_failure

* fix(callbacks): reject a selection name or alias that is not lowercase

* fix(custom-logger-v2): run v2 deployment callbacks on rust bridge deployment hooks

* fix(router): type the fallback event kwargs handed to the v2 dispatch

* feat(custom_logger_v2): type the contract fields as records and mark derived-only fields

Fields that the host derives from a hook's arguments now carry pydantic records on the Python side and matching serde records in litellm-callbacks, with records.json as the shared sample both sides check. Twelve new fields take bits 30 to 41, a write to a derived-only field is a manifest error, and pre_call can write budget_reservation and rate_limit_statuses in place. The payload reference field and the PayloadHost trait give Rust a handle to the logging payload instead of a copy

* feat(custom_logger_v2): derive typed fields from hook arguments and always build call facts

A slot table maps each typed field to a function over the hook's arguments. A derived slot runs only when a subscriber reads it, once per hook invocation, and off-path readers settle their reads before the hook returns. Auth fields come from the auth object only, budget_plan is claimed once, and budget_reservation and rate_limit_statuses write through to their carriers so the next pre_call subscriber sees them. Call facts are now built on unscoped logging lines and at post_call_success, post_call_failure and response_headers

* feat(callbacks-python): hand the logging payload to native callbacks by reference

NativeCallback stashes the shallow payload copy it is given and passes Rust a PayloadRef. The Python host renders a handle with a safe_dumps compatible encoder, on the caller's event loop while the hook runs and on the blocking pool after it, and dropping the last reference releases the stash entry. The encoder and its JSON helpers move here from the S3 logging bridge so every port shares one copy

* feat(custom_logger_v2): accept every value python accepts for typed members and serve logged key identity

Typed members keep the bool, int, float or str python holds through a Scalar
type on both sides, so ids, limits and priorities no longer lose values that
lax pydantic refused. A member the contract cannot carry is named in rejected
so a port can reproduce what python does with it.

Logging hooks get logged_key_identity, built from the payload metadata the
python loggers read on every route, while key_identity stays on request hooks
and keeps reading the auth object.

* fix(custom_logger_v2): read the live budget reservation dict and carry non-finite floats both ways

budget_reservation read the dict budget_reservation.py builds with a KeyError
on callback_bound. Undeclared keys are now ignored, callback_bound is a member,
entries keep a non-mapping item as null so capacity's model access group check
stays exact, and costs read as the float budgets computes with float(x or 0.0).

Float members cross as the raw json marker when they are not finite, and a
native verdict returns through json_to_py, so an infinite or NaN cost survives
a read and a write instead of turning into null.

* feat(callbacks): carry a python value as its str() text and truthiness in scalar members

Scalar gains a Stringified variant with the str() text and bool() of a value
that is not a bool, int within i128, float or str, so a port can reproduce
the label, path or log text python formats and the falsiness of an empty
container without confusing it with a real string. Budget reservations say
whether python held the entries as a list.

* feat(custom_logger_v2): carry non-scalar values and any iterable tags or entries across the boundary

A Scalar member now carries a list, dict, set, bytes, object or an int past
i128 as its str() text and truthiness instead of naming it in rejected, which
stays for values whose str() or bool() raises. request_tags carries any
iterable as the items prometheus iterates, and budget reservation entries
carry what budgets iterates with entries_is_list for capacity's list check,
never written back to the shared dict.

* feat(custom_logger_v2): say whether a stringified value hashes and whether it is an int

A Stringified scalar now carries hashable, from trying hash() at the
boundary, so a port can raise the TypeError capacity's priority lookup and
the limiter's descriptor identities raise for a list, dict or set and not
for a tuple or object. It also carries integer, so an int past i128 is told
apart from any other value whose text is digits.

* fix(custom_logger_v2): wrap a non-list guardrail mode as one mode like prometheus does

Prometheus treats any guardrail_mode that is not a list as a single mode, so a
tuple mode holds no str mode and a missing mode is one None mode. The
derivation unpacked tuples and rejected other non-str modes instead.

* feat(callbacks): let a native handler await typed host calls the python caller serves

* test(callbacks-python): drive a native handler's host calls through the python dispatcher

* test(custom_logger_v2): route a native logger with a host through the execution driver

* fix(custom_logger_v2): type the wrapped guardrail modes for pyright

* feat(custom_logger_v2): carry the python type name and hash error text of a stringified value

* feat(custom_logger_v2): serve the model group the logging handlers compute and key remaining limits by it

* feat(custom_logger_v2): tell a budget reservation that is not a mapping apart from none

* feat(custom_logger_v2): name malformed team model preferences in rejected with the text capacity logs

* feat(custom_logger_v2): serve a logging payload that is not a mapping apart from a missing one

* fix(callbacks-python): render null keys like a python dict merge, adding a missing key last

* feat(custom_logger_v2): carry an explicit output cap past i64 as a scalar

* feat(custom_logger_v2): serve the custom metadata labels prometheus derives from the logged payload

* feat(custom_logger_v2): serve the input sequence length bucket prometheus picks at attempt end

* feat(custom_logger_v2): carry the start and end times prometheus reads from the call details

* feat(custom_logger_v2): serve the limiter's body tags, session id, resolved agent and agent policy limits at pre_call

* fix(callbacks): export the limiter request record

* test(custom_logger_v2): give the type name cases stable ids and sort the record exports

* fix(custom_logger_v2): type the new derivations and tests for pyright

* fix(custom_logger_v2): drop team model preferences and serve the fallback_on_saturation handoff main's capacity callback reads

* style(callbacks): rustfmt the record exports

* refactor(custom_logger_v2): rename native callbacks to rust callbacks

native.py becomes rust.py, with native_logger, NativeLogger, NativeHandle,
NativeState and HostedNative renamed to their rust forms. The pyo3 classes
NativeCallback and NativeRequestState become RustCallback and
RustRequestState, and the tests, fixtures and messages follow.

* refactor(custom_logger_v2): rename slots to field sources

slots.py becomes field_sources.py: Slot is FieldSource, SlotFields is
SourcedFields, and with_slots, NO_SLOTS and slot_table_problem follow. A
table that only derives is named DERIVED_*_FIELDS, and a table that can
write back keeps SOURCES (PRE_CALL_AUTH_SOURCES, PRE_CALL_SOURCES). The
include_v2 comment moves down to the functions that take it.

* refactor(litellm_logging): place the v2 members of Logging beside what they serve

v2_scope, v2_auth and v2_failure_delivered join the other per-request
class defaults, v2_logging_scope and its opt-out helper follow
should_run_callback, v2_call_facts sits by the standard logging payload
builder, and claim_v2_failure sits above failure_handler.
bind_v2_request_scope checks for a Logging object directly instead of two
runtime protocols.

* refactor(proxy): rename v2_fields to v2_proxy_fields and say why it is separate

* refactor(custom_logger_v2): drop the logged_payload_not_a_mapping field

A logged payload that is present but not a mapping now looks the same to a
rust callback as a missing one, a parity gap accepted for a case that does
not occur. metadata_labels, input_sequence_length and limiter_request move
down to bits 44-46, on both sides of the contract.

* feat(callbacks): type a rust callback's request state

Callback gains an associated State type: request_state returns it and
handle and handle_hosted receive Option<&State> beside the input, which no
longer carries the state. ErasedCallback is the object-safe form the python
wrapper stores; it carries the state type-erased and downcasts it once,
returning CallbackError::StateType rather than panicking on a mismatch.
handle_hosted's default names its unused host _host. The record sample
count follows the dropped logged_payload_not_a_mapping field.

* docs(callbacks): say what PayloadRef, PayloadHost and RenderOptions.null_keys are

* refactor(callbacks-python): render payloads with python-compat's json writers

pyjson.rs reimplemented py_encode_basestring_ascii and json's floatstr.
python-compat's json module now exposes write_string, generic over a String
or byte sink, and float_text, and the payload renderer uses them.

* docs(callbacks): note why Respond, Reroute, call_ended and flush are not wired yet

* docs(custom_logger_v2): add a README for the package

* fix(custom_logger_v2): import the field sources tests from field_sources

* test(callbacks-python): stub the litellm package in each payload test

* fix(custom_logger_v2): type the disabled callbacks lookup and the v2 dedupe check
2026-11-14 02:33:27 +00:00
..
_support feat(mcp): add an opt-in per-server setting to skip MCP tool output validation (#49961) 2026-11-03 20:42:10 -08:00
audio_tests feat(vertex-ai)!: support gemini 3.1 flash tts (#31915) 2026-10-21 04:18:42 +00:00
base_sdk_tests chore: move scripts and runtime sources under litellm (#51006) 2026-11-12 04:19:57 +00:00
basic_proxy_startup_tests feat(docker)!: one shipped image and one entrypoint beside the Dockerfile (#43332) 2026-11-10 18:32:16 -08:00
benchmarks fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079) 2026-11-14 02:29:16 +00:00
code_coverage_tests feat(proxy): customer budget rollup, reset on the budget row's window, named templates, team overrides, multi-window limits and pause_allow_increase (#51582) 2026-11-13 16:05:35 +00:00
documentation_tests chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
e2e refactor(redis): delete the Python RedisBatch now that every request op is native (#51383) 2026-11-13 03:22:28 +00:00
e2e_harness test(e2e): add a Codex row where a sub-agent must read the task its parent hands it (#50494) 2026-11-07 21:16:39 +00:00
fixtures test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
harness_e2e fix(harness): set PWD to the working directory of local sandbox commands (#47530) 2026-10-20 20:06:11 +00:00
integration fix(proxy): count tokens with the thinking, effort, and tool_choice that inference sends (#51079) 2026-11-14 02:29:16 +00:00
rust-python-harness chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
test_litellm_rust perf(rust): stop re-serializing the Messages request body in Python logging callbacks (#51440) 2026-11-12 17:13:55 -08:00
typing chore: move config and docs off the repo root (#50966) 2026-11-11 18:29:35 -08:00
unit feat(custom-logger-v2): add the contract for rust callbacks and bind the proxy request scope (#51526) 2026-11-14 02:33:27 +00:00
windows_tests feat(proxy): supervise the opt-in Rust forwarding server (#49606) 2026-11-11 19:40:00 +00:00
__init__.py refactor: expose core private helpers under public names (#44871) 2026-10-07 00:05:27 +00:00
_fake_openai_endpoint_server.py
_flush_vcr_cache.py
_live_test_helpers.py
_master_key.py fix(security): remove the publicly known master key from the repo (#44718) 2026-10-06 10:55:24 -07:00
_openai_record_replay_proxy.py
_process_helpers.py
_vcr_conftest_common.py ci: remove CircleCI (#50950) 2026-11-11 15:38:27 -08:00
_vcr_redis_persister.py
_wait_helpers.py
_ws_vcr.py
AGENTS.md docs(agents): ban semicolons and colons in human-facing text (#45684) 2026-10-09 16:36:32 -07:00
capturing_transport.py
eval_swe_bench.py
fake_openai_endpoint.py
gettysburg.wav
large_text.py
openai_batch_completions.jsonl
pyrightconfig.json
README.MD
test_logging.conf
test_rust_python_harness.py test: move 76 live top-level tests to offline unit and integration coverage (#45352) 2026-10-08 10:33:35 -07:00
VCR_TESTING.md test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
white_100x100.png

In total litellm runs 1000+ tests

[02/20/2025] Update:

To make it easier to contribute and map what behavior is tested,

we've started mapping the litellm directory in tests/unit

This folder can only run mock tests.