mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-27 06:13:07 +00:00
0e24bbb8a2
* test(responses): cover billing every round of a streamed MCP auto-execute * fix(responses): bill every round of a streamed MCP auto-execute on one spend row * fix(responses): bill every round of a streamed MCP auto-execute on bridged providers A streamed /v1/responses MCP auto-execute on a provider bridged through chat completions (for example Anthropic) billed only round 1, because each round's chat stream dispatched on the shared logging object and the first dispatch won. Bridged rounds now go through the same one-row billing as native rounds. A round is billed as soon as its chat stream ends, before its closing events reach the client. The row carries the final response, usage and cost summed over every round, and the fixed margin once. A round served from the response cache is billed at zero. A follow-up chat stream that breaks bills the paid rounds on its failure row. A single-round stream logs exactly as before. * fix(responses): charge one fixed margin and keep bridged usage fields on streamed MCP rounds A follow-up round that fails after earlier rounds were billed now charges the per-request fixed margin once, and its failure row carries no cost breakdown that would disagree with the summed cost. A bridged multi-round stream logs a plain cache_hit flag like a single-round stream does, and its usage keeps the summed provider usage fields of every round. A new test covers a stream dropped after its event loop closed, which must release its parked log without an unawaited coroutine warning. * fix(responses): price each streamed MCP round on its own and bill every round when a guardrail blocks Each round of a streamed MCP auto-execute is now priced from its own uncached cost, so a provider that states its cost (xAI) charges the stated cost of every round plus the fixed margin once, and the cost breakdown always agrees with the billed cost. Summed usage goes through a JSON-safe dump, so a nested provider field such as xAI's context_details no longer breaks a failed stream's response.failed event. A post-call guardrail that blocks a multi-round bridged stream now drops the parked success log and bills every round on the failure row, and a failed follow-up round's failure row carries the summed breakdown of both rounds. * fix(responses): keep earlier MCP rounds on their success row when a later round fails When a later round of a streamed MCP auto-execute fails, the rounds before it are now logged on their success row, and the failure row bills only the failed round with no second fixed margin. That failure log runs on its own copy of the request's logging state, so its usage and cost never leak into the earlier rounds' row. A post-call guardrail that blocks a native multi-round stream keeps the summed success row beside the $0 failure row, as at the merge base, while a bridged stream still bills every round on its failure row. * fix(responses): keep a failed later MCP round on its own failure row A streamed MCP auto-execute whose later round failed wrote only the earlier rounds' success row. The failure was logged on a copy of the logging state, while the proxy builds its failure row from the request's own state, which by then held the success payload, so the failure row took the success row's id and was dropped. Now the failure is logged on the request's own state as at the merge base, and once a later round starts, the success row for the rounds before it is logged from a copy of the state taken at that moment, so the two rows never mix. * fix(proxy): let an MCP tool call reuse its request's max_parallel_requests slot * test(proxy): create the other-request task in a way Python 3.10 supports and mark its bindings Final * fix(proxy): keep a reused max_parallel slot until the request's last round An MCP tool call that reuses its request's max_parallel_requests slot no longer loses it to the first round's late success callback, so the key stays counted for the rounds still to run. The reused call also stops replacing the request's rate limit headers. * fix(responses): keep an unpriced streamed MCP round unpriced instead of billing it as free * fix(responses): log a bridged MCP stream as a chat completion with every round's usage |
||
|---|---|---|
| .. | ||
| _support | ||
| base_sdk_tests | ||
| benchmarks | ||
| code_coverage_tests | ||
| documentation_tests | ||
| e2e | ||
| e2e_harness | ||
| fixtures | ||
| harness_e2e | ||
| integration | ||
| rust-python-harness | ||
| test_litellm_rust | ||
| typing | ||
| unit | ||
| windows_tests | ||
| __init__.py | ||
| _fake_openai_endpoint_server.py | ||
| _flush_vcr_cache.py | ||
| _live_test_helpers.py | ||
| _master_key.py | ||
| _openai_record_replay_proxy.py | ||
| _process_helpers.py | ||
| _vcr_conftest_common.py | ||
| _vcr_redis_persister.py | ||
| _wait_helpers.py | ||
| _ws_vcr.py | ||
| AGENTS.md | ||
| capturing_transport.py | ||
| eval_swe_bench.py | ||
| fake_openai_endpoint.py | ||
| gettysburg.wav | ||
| large_text.py | ||
| openai_batch_completions.jsonl | ||
| pyrightconfig.json | ||
| README.MD | ||
| test_logging.conf | ||
| test_rust_python_harness.py | ||
| VCR_TESTING.md | ||
| white_100x100.png | ||
In total litellm runs 1000+ tests
[02/20/2025] Update:
To make it easier to contribute and map what behavior is tested,
we've started mapping the litellm directory in tests/unit
This folder can only run mock tests.