scriptoriumadmin/litellm
1
0
Fork 0
mirror of https://github.com/BerriAI/litellm.git synced 2026-11-14 09:03:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 214
litellm/tests/e2e/llm_translation
History
Exact
yuneng-jiang 07be67603c
test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 
* test: update the spend tests that still expected the pre-ledger accounting queues

#49872 made the SpendLogs ledger the default accounting path on purpose. A batch with no billable request now writes no cost row, a call the guardrail fails closed during a vendor outage keeps one spend row of either status while its key is still charged once, and charges left at a graceful shutdown are applied by the next start. The daily spend batch shutdown tests exercise the old queue, so they pin use_spend_logs_ledger to false.

* test(integration): price the guardrail relay model so the outage test can assert the charge

The relay model had no price, so every stream cost $0 and the outage test could not tell a charged call from an uncharged one. Both rigs now price it, and the test expects each served stream to cost 9 input and 4 output tokens and the key to be charged for all sixteen.

Also notes the skipped batch cost row case in the e2e rules file, next to the other explicit skips.

* test: repair four tests that fail for reasons outside the product

The OpenAPI docs gate test rebuilds the Rust extension with uv, and the integration lane blocks outbound traffic, so uv could not refresh its PyPI index for maturin. The test now runs uv offline against the cache the dependency install already filled.

The response cache test expected a chat reply blocked with empty content to be cached. Since #50057 a reply with no output is never cached, so that case now expects the repeat to reach the upstream and the real answer to be cached.

The Gemini cache hit e2e test gave the model 16 tokens, which its thinking can use up, and a reply with no text is not cached. It now allows 512 and checks the first reply has text.

The Together cache read e2e test picks the cheapest model with a cache read price. Since #50491 that is a -latest alias, whose target reports no cached tokens. Aliases are now left out of the pick.
2026-11-09 22:07:42 -08:00
..
fixtures test(e2e): serve the vision image from our own fixture 2026-08-27 14:29:53 -07:00
realtime fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
conftest.py test(e2e): replace custom endpoints_client with provider SDK clients 2026-07-23 03:43:20 +00:00
conversational_matrix.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
LLM_TRANSLATION_COVERAGE_MATRIX.md fix(proxy): record streamed /v1/responses container ownership before the response.completed frame (#43140) 2026-09-25 13:45:43 -07:00
passthrough_client.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
responses_helpers.py fix(providers): route Azure Responses past dated api versions, count Grok reasoning tokens on azure_ai, refresh provider sets after register_model, fix gpt-6.1-sol Azure rows (#46377) 2026-10-16 15:23:46 -07:00
sdk_clients.py test(e2e): send no-cache on cacheable SDK calls and accept Bedrock's 400 on the responses leg 2026-09-21 12:39:16 -07:00
structured_output.py test(e2e): move live-provider legacy tests into tests/e2e (#44120) 2026-10-02 00:02:18 -07:00
test_anthropic_completion_e2e.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_audio_speech_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_audio_transcriptions_e2e.py feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
test_bedrock_converse_redacted_thinking_e2e.py fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532) 2026-10-14 19:08:14 -07:00
test_bedrock_guardrail_config_e2e.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_bedrock_kimi_k3_messages_e2e.py fix(bedrock): apply the 16 maxTokens floor to Kimi K3 on Converse (#43875) 2026-10-20 13:01:45 +00:00
test_bedrock_mantle_output_limit_e2e.py fix(bedrock_mantle): send max_completion_tokens for gpt-5.6 and gpt-6 chat completions (#43982) 2026-10-11 18:28:48 +00:00
test_bedrock_native_e2e.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_bedrock_provider_matrix_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_bedrock_web_search_server_tool_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_cache_control.py test(e2e): assert the Vertex cache-creation billing contract in the prompt caching test (#47027) 2026-10-16 18:51:38 -07:00
test_cache_control_injection_tool_calls_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_chat_completion_prediction_e2e.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_chat_completions_contract_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_chat_completions_regression_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_chat_mid_conversation_system_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_chat_stream_contract_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_chat_tool_round_trip_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_completions_endpoint_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_containers_e2e.py fix(proxy)!: preserve Responses SSE event framing (#47980) 2026-11-06 20:17:57 -08:00
test_conversational_matrix_e2e.py test(e2e): add conversational matrix across chat, messages and responses (#42359) 2026-09-21 22:32:56 -07:00
test_count_tokens_source_e2e.py feat(count_tokens)!: configurable token_count_framing and token_count_model, count source in the response (#46777) 2026-10-17 17:11:27 -07:00
test_credential_messages_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_custom_pricing_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_deepseek_reasoning_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_embedding_only_supported_endpoints_e2e.py feat(router): opt in to non-chat deployment filtering (#46257) 2026-10-29 08:33:36 -07:00
test_embeddings_endpoint_e2e.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_files_batches_contract_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_gemini_tool_call_id_length_e2e.py fix(gemini): add thought_signature_in_tool_call_id to keep tool call ids short for clients that cap them (#47692) 2026-10-31 03:20:42 -07:00
test_gemini_url_context_e2e.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_google_interactions_e2e.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_google_native_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_image_edits_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_image_generation_e2e.py test(e2e): fix MCP tool search hit matching and the retired Bedrock image model (#46998) 2026-10-16 15:45:39 -07:00
test_messages_azure_foundry_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_messages_bedrock_e2e.py fix(bedrock): keep tool_reference results as text on converse (#44590) 2026-10-09 17:52:17 -07:00
test_messages_beta_headers_e2e.py fix(anthropic)!: sync the beta header allowlist with what anthropic, vertex_ai, and azure_ai accept (#49409) 2026-11-02 05:57:51 -08:00
test_messages_count_tokens_bedrock_e2e.py fix(bedrock): count tokens on bedrock-mantle when bedrock-runtime cannot count a Claude model (#45317) 2026-10-08 15:26:49 -07:00
test_messages_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_messages_mid_conversation_system_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_messages_mid_conversation_system_native_providers_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_messages_prompt_caching_and_tool_search_e2e.py test: migrate SDK and proxy-booting legacy CircleCI tests to their tiers (#45758) 2026-10-11 00:17:21 -07:00
test_moderations_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_ocr_rust_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_ollama_e2e.py test(e2e): cover ollama and ollama_chat on chat completions, responses and messages (#45079) 2026-10-07 20:17:35 +00:00
test_passthrough_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_passthrough_headers_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_provider_features_e2e.py fix(completion): send prompt_cache_retention to OpenAI instead of silently dropping it (#47480) 2026-10-20 09:41:22 -07:00
test_realtime_http_e2e.py feat(realtime)!: support latest OpenAI audio models (#35600) 2026-11-03 07:27:52 +00:00
test_reasoning_effort_e2e.py test: migrate tests/llm_translation into unit, integration and e2e tiers (#45759) 2026-10-11 07:58:53 +00:00
test_rerank_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_responses_bridge_streaming_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_responses_codex_tool_search_e2e.py feat(responses): support Codex client tool_search on Anthropic and other bridged providers (#48972) 2026-10-29 00:29:40 -07:00
test_responses_e2e.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_responses_retrieve_e2e.py fix(spend)!: save streamed /v1/responses spend rows under the encrypted resp_ id the client received (#49493) 2026-11-02 14:13:17 -08:00
test_sagemaker_nova_e2e.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_together_ai_e2e.py test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
test_token_counter_gemini_contents_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_vector_stores_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00
test_vertex_capabilities_e2e.py test: migrate local_testing SDK and provider tests out of build_and_test (#45760) 2026-10-10 23:14:25 +00:00
test_vertex_passthrough_e2e.py test(e2e): tag llm_translation tests with Subject metadata and record harness steps (#44950) 2026-10-07 10:30:37 -07:00