mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-12 03:48:56 +00:00
4 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mubashir1osmani
|
edd3bce0ec
|
fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265)
The rate-limited batch spend test snapshotted unattributed rows via the unpaginated /spend/logs whole-table read, which grows with the environment (58MB on stage) and OOMKilled the e2e runner at its 512Mi limit on every scheduled run. Gateway.spend_logs_window pages /spend/logs/v2 over an explicit date window instead, and SpendLogsParams now rejects a filterless read so the whole-table call cannot come back |
||
|
mubashir1osmani
|
54d404ef2c
|
fix(e2e): batch credentials wiring and compose harness for live proxy suite (#32744)
* fix(e2e): wire batch provider secrets for docker and k8s
Point batch deployments at the credential field names and os.environ refs
the gateway actually resolves from process env (compose .env or EKS secret
mounts). Missing secrets skip instead of failing red so a red run means a
product bug. Mirror S3 bucket env aliases in docker-compose for provider_fallback
* fix(e2e): drop batch provider_env unit tests
The batches suite is live e2e only; no monkeypatch or unit-level tests
* fix: batch credentials, provider list, and team db lookup
Keep object-storage fields through CredentialLiteLLMParams and resolve
os.environ/ refs when reading deployment credentials so Vertex/Bedrock
batch file uploads see bucket and AWS keys from K8s/docker env
Skip managed batch list when the request is provider-scoped so
/{provider}/v1/batches list works instead of 500
Force DB on check_db_only team lookups and stop masking non-404 errors
as "team doesn't exist"
Drop e2e runner-side skip helpers; hard-fail on missing gateway secrets
* fix: tag reseed, team window spend, and remaining e2e flakes
Reseed spend:tag counters from LiteLLM_TagTable so cold redis still
enforces after the spend writer flushes
When applying post-call cost to team multi-window counters, load the
team from the DB if it is missing from the management cache so window
spend is not dropped on cache misses
Harden cold-counter reseed e2e (namespace-aware keys, burst success,
poll). Give tag budget more headroom. Retry /key/update on redis DNS
blips. Ensure NLTK punkt_tab is present for pipecat realtime audio
* revert: drop product code changes; e2e-only scope
Reverts all litellm/ and unit-test product edits. This branch is limited
to tests/e2e per contributor instruction
* fix(e2e): harden batch list and team member setup races
provider_fallback list falls back when managed batches reject provider
filtering. Team create waits for /team/info and member_add retries on
transient team-not-found so split control-plane lag does not red the suite
* fix(e2e): remove .env.example
Leave local .env and docker-compose env wiring as the secret source
* fix(e2e): wire files_settings and faster budget rescheduler for compose
OpenAI/Azure batch file uploads need files_settings; budget reset e2e needs a
short rescheduler window. Drop unsupported bedrock-encoded create_batch cells,
tolerate bedrock file.bytes=0, and surface team-info wait failures instead of
hanging silently
* chore(e2e): strip verbose comments from batch capabilities
* fix(e2e): assert managed list fallback before provider_fallback skip
When provider-scoped list is rejected, still fetch the unfiltered list and
check the envelope. Only skip membership when the id is a raw
provider_fallback batch that managed list cannot index
|
||
|
mubashir1osmani
|
31c1ffc5a4
|
test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction (#32165)
* fix(e2e): define SpendTagsResponse/TagSpend so spend suite collects spend_tracking/spend_e2e_client.py imported SpendTagsResponse and TagSpend from models, but neither was ever defined, so importing the client raised ImportError and pytest aborted collection for the whole e2e session. The tag-spend tests had never run. Model /spend/tags as it actually answers: a bare array of per-tag aggregates, so SpendTagsResponse is a RootModel[list[TagSpend]] like the existing SpendLogs. spend_by_tags read a nonexistent spend_per_tag field that also wouldn't match the array shape; it now reads .root, matching how spend_logs consumes its RootModel. * test(e2e): close coverage gaps across chat/responses, provider features, batches, prometheus, and langfuse eviction Adds regression nets and gap-surfacing tests: A1 (llm_translation/test_deepseek_reasoning_e2e.py): control case proves the DeepSeek reasoner returns reasoning_content; two xfail(strict) cases document that reasoning_effort='none' and thinking type='disabled' are silently dropped (LIT-3686 / GH #27453) A2 (llm_translation/test_chat_completions_regression_e2e.py and test_responses_e2e.py): parametrized regression net asserting real completion content, not just a 200, across the configured providers for /chat/completions and /responses (GH #28991) A3 (llm_translation/test_provider_features_e2e.py): asserts service_tier is honored and prompt-cache read tokens grow on a repeated cacheable prefix A4 (batches/test_batches_e2e.py): mints a rate-limited key so the batch pre-call rate limiter runs, then asserts no unattributed spend row is left behind by the internal input-file retrieval (LIT-3266) A5 (logging/test_prometheus_cardinality_e2e.py): drives one chat per distinct key_alias and asserts each alias gets its own labeled series on /metrics A6 (test_litellm/.../specialty_caches/test_dynamic_logging_cache.py): xfail(strict) regression proving eviction must not close an httpx client still held by an in-flight caller (LIT-3221 / GH #13034) Extends tests/e2e/models.py with the typed request and response fields these tests read (reasoning_effort, thinking, service_tier, key_alias, cache usage fields, spend-log api_key) Co-authored-by: Cursor <cursoragent@cursor.com> * test(e2e): drop unused litellm-regression-tests submodule The e2e suite migrated the regression cases into this repo; nothing imports the submodule at runtime (only a provenance comment references it), so the .gitmodules entry and gitlink pointing at a personal repo would just make upstream CI init a submodule it never uses. Remove both to keep the change test-only. * test(e2e): drop A6 langfuse-eviction xfail; keep PR to live e2e coverage The dynamic_logging_cache strict-xfail documented an unfixed shared-httpx-client close-on-eviction bug (LIT-3221 / GH #13034). That is a non-trivial fix (thread cleanup vs shared client teardown) and belongs in its own PR, not this e2e coverage PR, so revert the file to its base state. --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Sameer Kankute
|
a16d9c6f9e
|
test(e2e): add live batches suite across providers and routing scenarios (#30958)
* tests: add e2e tests for spend, budgets and llms * style: make chained comparison of status_code clearer Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * remove e2e_tests folder * test: add spend tracking tests * fix: p0 issues, added types and shared functions for each test suite * style: carry clearer status_code comparison into renamed e2e dir * refactor: migrate to gateway client * fix: add new tests, split gateway * test(e2e): add live batches suite across providers and routing scenarios * test(batches): cover real cost tracking on completed batch retrieve * test(e2e): assert managed vs raw file and batch id shapes per routing scenario * test(e2e): assert full response shape of each batches and files endpoint * test(e2e): only accept transitional statuses for a freshly created batch * test(prompt-factory): make test_convert_url deterministic with a data URL picsum.photos is down (HTTP 522), so test_convert_url failed on every run. Swap the live external image for an inline data: URL and assert the round-trip through convert_url_to_base64 genuinely. A data URL is already inline base64 image data, so convert_url_to_base64 now short-circuits it instead of attempting an impossible HTTP fetch; add a regression for that branch in the mapped image_handling test * fix: pass through async image data urls * fix(image-handling): short-circuit data URLs in async path too Bugbot flagged that convert_url_to_base64 returns data: base64 URLs unchanged but async_convert_url_to_base64 still tried to fetch them, so async OCR flows (Bedrock, Azure) would reject inline images the sync path accepts. Add the same guard to the async function and a regression test that asserts the async path returns the data URL without touching the HTTP client * Fix: openai batches lifecycle * Fix: add e2e azure openai tests * Fix e2e for vertex ai * Add all models for testing * test(managed-files): assert idempotent upsert in store_unified_file_id store_unified_file_id switched from create to upsert to avoid UniqueViolationError when re-storing the same unified_file_id (e.g. batch output files stored before metadata is available). Update the unit test to assert the upsert call and its create payload instead of the removed create call. * test(batches): reconcile vertex_ai native batch-id comment with fallback guard * fix(test-config): keep rust-ocr models in model_list by moving files_settings after it * fix(test-config): move batch models after OCR block to keep merge with internal_staging clean * fix(batches): use '24hrs' completion window and allow managed-files listing with provider filter Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: ruff format transformation.py and endpoints.py Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(e2e/batches): set Azure raw_model to gpt-4.1-mini-batch to match deployed model Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(vertex-ai/batches): correct completion_window to 24h per Literal type definition * test(vertex-ai/batches): align completion_window assertion to 24h * fix: update managed file metadata on upsert --------- Co-authored-by: mubashir1osmani <mubashir.osmani777@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |