mirror of
https://github.com/BerriAI/litellm.git
synced 2026-09-13 23:11:40 +00:00
31c1ffc5a4
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH #27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH #28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH #13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com>
52 lines
1.5 KiB
Python
52 lines
1.5 KiB
Python
"""Batches suite's `client` fixture.
|
|
|
|
The shared lifecycle (resources/scoped_key), proxy liveness skip, and e2e marker
|
|
live in the parent tests/e2e/conftest.py. BatchClient holds the shared Gateway, so
|
|
the `resources` fixture cleans up keys through it; tests register file deletes and
|
|
batch cancels via `resources.defer(...)`.
|
|
|
|
Batch deployments (openai-batch, azure-batch, vertex-batch, ...) are registered
|
|
once per session via /model/new and deleted on teardown so they need not live in
|
|
the proxy config.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Iterator
|
|
|
|
import pytest
|
|
|
|
from batch_client import BatchClient, build_client
|
|
from capabilities import PROVIDERS
|
|
from e2e_http import NoBody
|
|
|
|
|
|
def pytest_configure(config: pytest.Config) -> None:
|
|
config.addinivalue_line(
|
|
"markers",
|
|
"covers: registry cell a test covers, e.g. llm.batches.openai.basic.nonstream.works",
|
|
)
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def client() -> BatchClient:
|
|
return build_client()
|
|
|
|
|
|
@pytest.fixture(scope="session")
|
|
def batch_deployments(client: BatchClient) -> Iterator[None]:
|
|
probe = client.gateway.probe("/health/liveliness", params=NoBody())
|
|
if not probe.healthy:
|
|
yield
|
|
return
|
|
|
|
registered: list[str] = []
|
|
try:
|
|
for provider in PROVIDERS:
|
|
registered.append(
|
|
client.create_model(provider.model, provider.litellm_params())
|
|
)
|
|
yield
|
|
finally:
|
|
for model_id in registered:
|
|
client.delete_model(model_id)
|