litellm/tests/e2e_harness
paveln83 05518975a5
test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816)
* test(e2e): move audio and realtime tests off OpenAI model ids with shutdown dates

* test(e2e): move langfuse completion, image, and vertex realtime ids off dated models

* test(e2e): register the weather tool in the first realtime session update and read Langfuse v2 flattened attributes

* test(e2e): expect the OCR upload's file summary in the Langfuse generation input

The OCR upload route has sent the uploaded file, not a data URL, since the native OCR route, and the generation input summarizes that form as the mime type and file name. The broken Langfuse lookup hid the stale data URL expectation

* test(e2e): move the image generation test off gpt-image-1-mini

gpt-image-1-mini has a 2026-12-01 deprecation_date, so the image generation e2e would start failing that day. gpt-image-2.5-flare has no date and already serves the Langfuse image tests

* test(e2e): use gpt-image-2.6-flare and check every n=2 choice reaches Langfuse

OpenAI names gpt-image-2.6-flare as the successor of gpt-image-1-mini, so the image tests use it. The n=2 completions test now needs one logged message per choice, since both choices repeat the same text and a single logged message used to pass

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-11-14 09:51:53 -05:00
..
batches test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
claude_code test(e2e): share the coding-CLI harness between suites (#49370) 2026-11-02 13:50:56 -08:00
cli_harness test(e2e): share the coding-CLI harness between suites (#49370) 2026-11-02 13:50:56 -08:00
codex test(e2e): add a Codex row where a sub-agent must read the task its parent hands it (#50494) 2026-11-07 21:16:39 +00:00
coverage_registry test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
guardrails test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
logging test(e2e): move audio, realtime, and Langfuse tests off model ids with shutdown dates (#51816) 2026-11-14 09:51:53 -05:00
perf test(perf): remove the database_upgrade_5rps e2e case and its harness tests (#50076) 2026-11-04 10:39:46 -08:00
AGENTS.md test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) 2026-11-04 09:55:53 -08:00
pytest.ini fix(e2e): restore main qualification across providers and UI (#49763) 2026-11-03 20:38:47 +00:00
test_e2e_http.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_fixture_bundle.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_fixture_canonical.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_fixture_mode.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_idp.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_junit_properties.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_provider_edge.py test(e2e): move the harness self-tests out of tests/e2e (#45172) 2026-10-07 22:20:02 +00:00
test_proxy_client.py test(e2e): retire the quiet_stack lock and batch the claude_code model registration (#49984) 2026-11-04 09:55:53 -08:00