litellm/tests/e2e
Sameer Kankute a16d9c6f9e
test(e2e): add live batches suite across providers and routing scenarios (#30958)
* tests: add e2e tests for spend, budgets and llms

* style: make chained comparison of status_code clearer

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* remove e2e_tests folder

* test: add spend tracking tests

* fix: p0 issues, added types and shared functions for each test suite

* style: carry clearer status_code comparison into renamed e2e dir

* refactor: migrate to gateway client

* fix: add new tests, split gateway

* test(e2e): add live batches suite across providers and routing scenarios

* test(batches): cover real cost tracking on completed batch retrieve

* test(e2e): assert managed vs raw file and batch id shapes per routing scenario

* test(e2e): assert full response shape of each batches and files endpoint

* test(e2e): only accept transitional statuses for a freshly created batch

* test(prompt-factory): make test_convert_url deterministic with a data URL

picsum.photos is down (HTTP 522), so test_convert_url failed on every
run. Swap the live external image for an inline data: URL and assert the
round-trip through convert_url_to_base64 genuinely.

A data URL is already inline base64 image data, so convert_url_to_base64
now short-circuits it instead of attempting an impossible HTTP fetch;
add a regression for that branch in the mapped image_handling test

* fix: pass through async image data urls

* fix(image-handling): short-circuit data URLs in async path too

Bugbot flagged that convert_url_to_base64 returns data: base64 URLs
unchanged but async_convert_url_to_base64 still tried to fetch them,
so async OCR flows (Bedrock, Azure) would reject inline images the sync
path accepts. Add the same guard to the async function and a regression
test that asserts the async path returns the data URL without touching
the HTTP client

* Fix: openai batches lifecycle

* Fix: add e2e azure openai tests

* Fix e2e for vertex ai

* Add all models for testing

* test(managed-files): assert idempotent upsert in store_unified_file_id

store_unified_file_id switched from create to upsert to avoid
UniqueViolationError when re-storing the same unified_file_id (e.g.
batch output files stored before metadata is available). Update the
unit test to assert the upsert call and its create payload instead of
the removed create call.

* test(batches): reconcile vertex_ai native batch-id comment with fallback guard

* fix(test-config): keep rust-ocr models in model_list by moving files_settings after it

* fix(test-config): move batch models after OCR block to keep merge with internal_staging clean

* fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* style: ruff format transformation.py and endpoints.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition

* test(vertex-ai/batches): align completion_window assertion to 24h

* fix: update managed file metadata on upsert

---------

Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com>
Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-02 08:05:23 -07:00
..
batches test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
budgets test(e2e): drop xfail markers for the now-fixed team-budget-JSON and custom-pricing-leak bugs (#31249) 2026-06-24 18:51:10 -07:00
gateway test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
llm_translation test(e2e): move rust OCR e2e into llm_translation on the shared harness 2026-06-26 19:21:55 -07:00
realtime test: add realtime proxy e2e suite across providers (#30960) 2026-06-26 09:36:49 -07:00
spend_tracking test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
CLAUDE.md docs(e2e): document suite-folder layout and the add-a-folder rule in CLAUDE.md 2026-06-27 19:38:57 -07:00
conftest.py test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
CONTRIBUTING.md fix: make changes to contributing 2026-06-27 19:30:55 -07:00
e2e_config.py test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
e2e_gateway.py test(e2e): move rust OCR e2e into llm_translation on the shared harness 2026-06-26 19:21:55 -07:00
e2e_http.py test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
lifecycle.py test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
models.py test(e2e): move rust OCR e2e into llm_translation on the shared harness 2026-06-26 19:21:55 -07:00
pytest.ini test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
test_lifecycle.py test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
transport.py test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00