Model IDs were hardcoded across the e2e suite, so every provider
deprecation meant a scatter-shot edit and stale models (gemini-2.5-flash,
gpt-4o-mini) kept running in CI. Tests now import role-named pins from
tests/e2e/model_matrix.py, and a bump is a one-file change plus the
compose gateway config.
check_e2e_model_freshness.py (wired into test-code-quality) fails when a
pin is missing from model_prices_and_context_window.json, is within 30
days of its deprecation_date, drifts from the docker-compose gateway
config, or when a test hardcodes a versioned model literal again.
Bumps: gemini-2.5-flash -> gemini-3.5-flash, gpt-4o-mini -> gpt-5.4-mini
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects
spend_tracking/spend_e2e_client.py imported SpendTagsResponse and
TagSpend from models, but neither was ever defined, so importing the
client raised ImportError and pytest aborted collection for the whole
e2e session. The tag-spend tests had never run.
Model /spend/tags as it actually answers: a bare array of per-tag
aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the
existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field
that also wouldn't match the array shape; it now reads .root, matching
how spend_logs consumes its RootModel.
* test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction
Adds regression nets and gap-surfacing tests:
A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the
DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document
that reasoning_effort='none' and thinking type='disabled' are silently dropped
(LIT-3686 / GH #27453)
A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py):
parametrized regression net asserting real completion content, not just a 200,
across the configured providers for /chat/completions and /responses (GH #28991)
A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is
honored and prompt-cache read tokens grow on a repeated cacheable prefix
A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call
rate limiter runs, then asserts no unattributed spend row is left behind by the
internal input-file retrieval (LIT-3266)
A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct
key_alias and asserts each alias gets its own labeled series on /metrics
A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict)
regression proving eviction must not close an httpx client still held by an
in-flight caller (LIT-3221 / GH #13034)
Extends tests/e2e/models.py with the typed request and response fields these
tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage
fields, spend-log api_key)
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(e2e): drop unused litellm-regression-tests submodule
The e2e suite migrated the regression cases into this repo; nothing
imports the submodule at runtime (only a provenance comment references
it), so the .gitmodules entry and gitlink pointing at a personal repo
would just make upstream CI init a submodule it never uses. Remove both
to keep the change test-only.
* test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage
The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client
close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread
cleanup vs shared client teardown) and belongs in its own PR, not this e2e
coverage PR, so revert the file to its base state.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
