litellm/tests/e2e
paveln83 05518975a5
test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816)
* test(e2e): move audio and realtime tests off OpenAI model ids with shutdown dates

* test(e2e): move langfuse completion, image, and vertex realtime ids off dated models

* test(e2e): register the weather tool in the first realtime session update and read Langfuse v2 flattened attributes

* test(e2e): expect the OCR upload's file summary in the Langfuse generation input

The OCR upload route has sent the uploaded file, not a data URL, since the native OCR route, and the generation input summarizes that form as the mime type and file name. The broken Langfuse lookup hid the stale data URL expectation

* test(e2e): move the image generation test off gpt-image-1-mini

gpt-image-1-mini has a 2026-12-01 deprecation_date, so the image generation e2e would start failing that day. gpt-image-2.5-flare has no date and already serves the Langfuse image tests

* test(e2e): use gpt-image-2.6-flare and check every n=2 choice reaches Langfuse

OpenAI names gpt-image-2.6-flare as the successor of gpt-image-1-mini, so the image tests use it. The n=2 completions test now needs one logged message per choice, since both choices repeat the same text and a single logged message used to pass

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 09:51:53 -05:00
..
a2a test(a2a): replace the third-party agent e2e test with an integration test (#49541) 2026-10-31 18:59:04 -07:00
access_control fix(proxy/auth): expand all-team-models sentinel for project model access checks (#43734) 2026-10-10 15:49:27 -07:00
batches test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
claude_code chore: move deployment sources under litellm/deploy (#50981) 2026-11-12 04:19:56 +00:00
cli_harness test(e2e): share the coding-CLI harness between suites (#49370) 2026-11-02 13:50:56 -08:00
codex test(e2e): add a Codex row where a sub-agent must read the task its parent hands it (#50494) 2026-11-07 21:16:39 +00:00
coverage_registry chore: move workspace packages under litellm (#51000) 2026-11-12 04:19:57 +00:00
gateway test(e2e): remove the secret manager suite (#51241) 2026-11-11 22:56:36 +00:00
guardrails fix(guardrails)!: check Anthropic and Responses request tools in tool_permission pre_call (#41011) 2026-11-04 20:34:34 +00:00
llm_translation test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816) 2026-11-14 09:51:53 -05:00
logging test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816) 2026-11-14 09:51:53 -05:00
management fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
mcp test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) 2026-11-04 09:55:53 -08:00
migrations refactor(helm)!: connect both charts to a separately deployed Lens (#49169) 2026-10-30 02:29:44 +00:00
other fix(proxy): enforce the personal max_budget on proxy-admin JWTs (#43070) 2026-10-12 16:04:12 -07:00
perf refactor(redis): delete the Python RedisBatch now that every request op is native (#51383) 2026-11-13 03:22:28 +00:00
quota_management test: update the spend tests that still expected the pre-ledger accounting queues (#50814) 2026-11-09 22:07:42 -08:00
router test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) 2026-11-04 09:55:53 -08:00
ui feat(ui): set per-threshold budget alert recipients on virtual keys (#51061) 2026-11-12 15:39:44 -08:00
AGENTS.md chore: move ui and vscode-extension under litellm and delete docs (#50992) 2026-11-12 04:19:57 +00:00
conftest.py test(e2e): remove the secret manager suite (#51241) 2026-11-11 22:56:36 +00:00
CONTRIBUTING.md chore: move deployment sources under litellm/deploy (#50981) 2026-11-12 04:19:56 +00:00
e2e_config.py test(e2e): remove the secret manager suite (#51241) 2026-11-11 22:56:36 +00:00
e2e_db.py test(e2e): guard destructive spend-log truncate behind an explicit opt-in (#33751) 2026-07-20 08:47:39 -07:00
e2e_http.py test(e2e): get_text_external accepts JSON next to text so a pod-log error reads as the Kubernetes Status, not a 406 (#47150) 2026-10-17 13:25:29 -07:00
e2e_metadata.py test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) 2026-10-06 17:03:35 -07:00
fixture_bundle.py test: add strict stateless provider replay identity 2026-09-14 16:55:43 -07:00
fixture_canonical.py feat(e2e): key the provider cache per test and mount Bedrock behind it 2026-09-16 02:15:46 -07:00
fixture_mode.py fix(e2e): own a shared fixture's deployment by the fixture's node, not the first test 2026-09-16 17:35:05 -07:00
fixture_profile.py test: preserve strict replay numeric spelling 2026-09-14 17:11:56 -07:00
idp.py test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) 2026-10-06 17:03:35 -07:00
idp_realm.json test(e2e): harden JWT fixtures and cover management lifecycles 2026-09-11 16:36:12 -07:00
junit_properties.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
lifecycle.py test(e2e): add enum values, auto-discovering label gates and secret hiding for e2e metadata (#44949) 2026-10-06 17:03:35 -07:00
memory_readings.py test(e2e): hold every worker under an idle RSS budget before any traffic (#42552) 2026-09-22 14:47:14 -07:00
models.py fix(guardrails)!: check Anthropic and Responses request tools in tool_permission pre_call (#41011) 2026-11-04 20:34:34 +00:00
otel_client.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
PROVIDER_CACHE.md test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
provider_cache.py fix(e2e): bind provider-cache recordings to the deployment's test, not the serving process 2026-09-16 17:08:17 -07:00
provider_cache_redis.py fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
provider_cache_routing.py revert(e2e): unmount Gemini, its api_base means two things 2026-09-16 08:37:25 -07:00
provider_edge.py test(e2e): record steps for raw transport calls and poll helpers (#45150) 2026-10-07 15:21:26 -07:00
provider_edge_bedrock.py fix(bedrock)!: replay redacted_thinking on Converse and keep its data as Claude's own string (#46532) 2026-10-14 19:08:14 -07:00
proxy_client.py test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) 2026-11-04 09:55:53 -08:00
pytest.ini test(e2e): remove the secret manager suite (#51241) 2026-11-11 22:56:36 +00:00
transport.py test(e2e): record steps for raw transport calls and poll helpers (#45150) 2026-10-07 15:21:26 -07:00