mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
5ed7ec8511
* fix(lens): price agent traces by joining gen_ai.response.id to spend logs Trace spend now joins each model call to spend_logs on one key: the span's response id (gen_ai.response.id or the id the normalizers read from OpenInference/LangChain output) against spend_logs.response_id or the upstream id embedded in a managed resp_ id. The litellm.call_id and traceparent transport join paths and the per-row ownership gate are removed; the spend SQL still restricts rows to what the reader can see. A run with some unpriced calls now reports the sum of its priced calls plus priced_calls, instead of an unknown total. * test(lens): cover response id spend join and partial trace totals * chore(lens): regenerate trace types for priced_calls * feat(lens): show partial run cost as a lower bound with priced call count * fix(lens): treat litellm.call_id as the same assigned call id for spend joins The id LiteLLM assigned to a call is either the response id it returned (gen_ai.response.id -> spend_logs.response_id) or its gateway call id (litellm.call_id -> spend_logs.litellm_call_id). Both are exact ids the gateway mints and logs, so the join stays one rule. Transport span matching and the ownership gate stay removed. * test(lens): cover litellm.call_id spend joins and restore captured totals * feat(lens): link each priced model call to its spend log Spans gain spend_log_request_id, the spend_logs.request_id the call was priced from, and spend_match, which says whether a model call matched or why not (no assigned id on the span, no spend log with that id, or an ambiguous match). A model call span is priced from the same ids as the run total, so its cost and the total agree. * test(lens): cover spend log links on model call spans * chore(lens): regenerate trace types for spend log links * feat(lens): open the matched spend log from an LLM step An LLM step's header now shows a Spend log chip with the matched request id and cost; clicking it opens the request log drawer over the run, fetched by the exact spend_logs.request_id instead of the span's own response id. Unpriced steps say why (no assigned id on the span, or no spend log with it). Tree rows show each model call's cost, and a partial run cost shows its priced call count inline. * test(lens): cover the spend log link and unmatched cost reasons * feat(lens): show the spend log link as a bordered LiteLLM Spend Log button * feat(lens): add a back link from the spend log drawer to the agent trace * feat(lens): label the spend log back link Back to Lens trace with the Lens icon * fix(lens): ignore assigned ids that name no spend log when pricing a call An id that names no row no longer vetoes the call, so a span carrying both a response id and a call id still prices from a spend row logged before litellm_call_id existed. An id naming two or more rows makes the call ambiguous, and the match reason comes from the same per-id result, so a single matched row with no cost is reported as matched. * fix(lens): price a trace only from spend logs in its own team A reader with several teams could see the same assigned id in another team's spend log; only rows from the trace's team now price it. The user and key ownership gate stays removed. * perf(lens): resolve each model call's spend once per trace Model call matches are computed once when the trace is resolved and looked up by span index, instead of scanning the model call list for every span and walking the graph again for spans, agents and the run total. * fix(lens): hide a step's Cost fact only when its spend log link shows the cost * chore(lens): drop narrative doc comments from the spend join * fix(lens): price a model call only when its ids agree on one spend log per span * fix(lens): keep pricing spend logs written before litellm_call_id by their request id * fix(lens): price every attempt a model call's ids name when they agree |
||
|---|---|---|
| .. | ||
| fixtures | ||
| interactions | ||
| realtime | ||
| reasoning_effort_grid | ||
| test-skill | ||
| test_llm_response_utils | ||
| test_skills_data | ||
| base_audio_transcription_unit_tests.py | ||
| base_embedding_unit_tests.py | ||
| base_llm_unit_tests.py | ||
| base_rerank_unit_tests.py | ||
| conftest.py | ||
| databricks_config.template.txt | ||
| dog.wav | ||
| duck.png | ||
| gettysburg.wav | ||
| guinea.png | ||
| log.xt | ||
| Readme.md | ||
| test_a2a.py | ||
| test_anthropic_completion.py | ||
| test_aws_base_llm.py | ||
| test_azure_agents.py | ||
| test_azure_ai.py | ||
| test_azure_o_series.py | ||
| test_azure_openai.py | ||
| test_bedrock_agentcore.py | ||
| test_bedrock_agents.py | ||
| test_bedrock_anthropic_regression.py | ||
| test_bedrock_common_utils.py | ||
| test_bedrock_completion.py | ||
| test_bedrock_dynamic_auth_params_unit_tests.py | ||
| test_bedrock_embedding.py | ||
| test_bedrock_govcloud.py | ||
| test_bedrock_gpt_oss.py | ||
| test_bedrock_invoke_tests.py | ||
| test_bedrock_llama.py | ||
| test_bedrock_mantle.py | ||
| test_bedrock_moonshot.py | ||
| test_bedrock_nova_embedding.py | ||
| test_bedrock_nova_json.py | ||
| test_cloudflare.py | ||
| test_cohere.py | ||
| test_cometapi_chat_transformation.py | ||
| test_compression.py | ||
| test_convert_dict_to_image.py | ||
| test_crusoe.py | ||
| test_databricks.py | ||
| test_databricks_e2e.py | ||
| test_deepgram.py | ||
| test_deepseek_completion.py | ||
| test_elevenlabs.py | ||
| test_evals_api.py | ||
| test_fireworks_ai_translation.py | ||
| test_gemini.py | ||
| test_gemini_image_usage.py | ||
| test_gigachat.py | ||
| test_gpt4o_audio.py | ||
| test_groq.py | ||
| test_hosted_vllm_embedding_e2e.py | ||
| test_huggingface_chat_completion.py | ||
| test_hyperbolic.py | ||
| test_infinity.py | ||
| test_jina_ai.py | ||
| test_json_providers.py | ||
| test_lambda_ai.py | ||
| test_langgraph.py | ||
| test_litellm_proxy_provider.py | ||
| test_minimax_tts.py | ||
| test_mistral_api.py | ||
| test_mistral_audio_transcription_transformation.py | ||
| test_model_cost_map_resilience.py | ||
| test_morph.py | ||
| test_nvidia_nim.py | ||
| test_openai.py | ||
| test_openai_o1.py | ||
| test_openai_record_replay_proxy.py | ||
| test_openrouter.py | ||
| test_optional_params.py | ||
| test_ovhcloud_audio_transcription_transformation.py | ||
| test_ovhcloud_chat_transformation.py | ||
| test_perplexity_reasoning.py | ||
| test_prompt_caching.py | ||
| test_prompt_factory.py | ||
| test_replicate.py | ||
| test_rerank.py | ||
| test_router_llm_translation_tests.py | ||
| test_sambanova_chat_transformation.py | ||
| test_skills_api.py | ||
| test_snowflake.py | ||
| test_text_completion.py | ||
| test_text_completion_unit_tests.py | ||
| test_together_ai.py | ||
| test_triton.py | ||
| test_unit_test_bedrock_invoke.py | ||
| test_v0.py | ||
| test_vcr_classification.py | ||
| test_vcr_conftest_common_banner.py | ||
| test_vcr_filters.py | ||
| test_vcr_leak_guard.py | ||
| test_vcr_redis_persister.py | ||
| test_vertex_ai_image_generation_transformation.py | ||
| test_voyage_ai.py | ||
| test_watsonx.py | ||
| test_ws_vcr.py | ||
| test_xai.py | ||
| test_xiaomi_mimo.py | ||
Unit tests for individual LLM providers.
Name of the test file is the name of the LLM provider - e.g. test_openai.py is for OpenAI.
Redis-backed VCR cache
Every test in this directory is auto-decorated with @pytest.mark.vcr (via
conftest.py). The first time a test runs we hit the live provider and
record the HTTP exchange into Redis under
litellm:vcr:cassette:<test_id>. Every subsequent run within 24h replays
from Redis without touching the network. The 24h TTL means each new day's
first run records again, so upstream API drift surfaces within a day.
The persister, header scrubbing, and 2xx-only filtering are defined in
tests/_vcr_redis_persister.py. Files that already use respx (which
patches the same httpx transport vcrpy does) are excluded from the
auto-marker — see _RESPX_CONFLICTING_FILES in conftest.py.
Requests to localhost, 127.0.0.1, or 0.0.0.0 are never recorded or
replayed (ignore_localhost in vcr_config_dict()): a server the test
process starts itself on an ephemeral port is not a provider, and a cassette
entry for it would replay against whichever later test lands on that port
The same VCR cache is used by other test directories that exercise live
provider APIs. The reusable conftest plumbing lives in
tests/_vcr_conftest_common.py and is wired into:
tests/llm_translation/tests/llm_responses_api_testing/tests/audio_tests/tests/batches_tests/tests/guardrails_tests/tests/image_gen_tests/tests/litellm_utils_tests/tests/local_testing/(coverslocal_testing_part1,local_testing_part2,litellm_router_testing,litellm_assistants_api_testing,langfuse_logging_unit_tests)tests/logging_callback_tests/tests/pass_through_unit_tests/tests/router_unit_tests/tests/unified_google_tests/
Test directories that run LiteLLM proxy in Docker (e.g. build_and_test,
proxy_logging_guardrails_model_info_tests, proxy_store_model_in_db_tests)
are intentionally not included: VCR.py patches the in-process httpx
transport, so it cannot intercept the LLM calls that originate inside the
Docker container.
Required environment
CASSETTE_REDIS_URL — separate Redis instance from the application
Redis (REDIS_URL/REDIS_HOST) so test cassettes are not flushed by
proxy tests. Provider credentials (ANTHROPIC_API_KEY, OPENAI_API_KEY,
AWS_*, etc.) are needed only on cache-miss (the daily re-record), not
on replay.
Flushing the cache
When you want the next run to re-record immediately instead of waiting for the 24h TTL:
make test-llm-translation-flush-vcr-cache
Disabling VCR
Skip the cache entirely (every call goes live, no recording):
LITELLM_VCR_DISABLE=1 uv run pytest tests/llm_translation/test_<file>.py