litellm/tests/e2e
yucheng-berri d6f498ff5c
test(e2e): failed request error span carries the full untruncated message and status (LIT-4179) (#33304)
* test(e2e): failed request error span carries the full untruncated message and status

Covers logging.otel.failure.exports_metric on chat_completions: a request that
fails at the provider (invalid upstream key deployment) must export one
complete trace whose gen-AI span carries the LIT-4179 error contract, declared
as one reviewable payload (EXPECTED_ERROR_SPAN_ATTRIBUTES) plus an untruncated
error.message proven by parsing the embedded provider error JSON back out of
the attribute. The root SERVER span must record the 401 the client received.
Adds STORE_MODEL_IN_DB to the compose stack so /model/new works locally, which
the suite's model-registering tests already assume

* test(e2e): clean failure diagnostics on the error-span contract per review

A truncated error.message with missing braces now fails with a readable
assertion instead of an unhandled ValueError, an unparseable embedded JSON
fails via pytest.fail with the truncation context, and the retry loop now
asserts the upstream provider failure was actually observed so a fresh-key
propagation deadline cannot masquerade as a trace-export failure

* test(e2e): pin the full error attribute set including the litellm.provider.error keys

The LIT-4179 fix restored error.message/code/stack_trace/llm_provider; a later
refactor (#32591) moved the litellm-specific keys under litellm.provider.error.*,
which the initial contract missed. The payload now pins error, error.type,
otel.status_code, litellm.provider.error.code=401, and
litellm.provider.error.llm_provider=anthropic exactly, plus non-empty
litellm.provider.error.stack_trace and the untruncated error.message

* test(e2e): author the error-span test docstring
2026-07-14 22:13:13 -07:00
..
access_control test(e2e): migrate access-control and inference-endpoint regression tests (#32016) 2026-07-05 01:39:10 +00:00
batches fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265) 2026-07-14 13:32:10 -07:00
claude_code test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
coverage_registry test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) 2026-07-14 20:23:54 -07:00
llm_translation test(e2e): scope virtual keys to the deployment under test 2026-07-14 17:12:15 -07:00
logging test(e2e): failed request error span carries the full untruncated message and status (LIT-4179) (#33304) 2026-07-14 22:13:13 -07:00
management test(e2e): cover key rpm/tpm rate limiting, window reset, and pacing headers 2026-07-11 16:15:16 -07:00
quota_management refactor: make the code easier to read 2026-07-14 13:58:23 -07:00
bob_the_builder.py test: litellm fix failing tests (#32577) 2026-07-09 13:54:45 -07:00
CLAUDE.md test(claude_code): move the Claude Code compatibility matrix under tests/e2e (#32548) 2026-07-14 19:19:03 -07:00
conftest.py refactor(e2e): move budgets and spend_tracking suites under quota_management 2026-07-11 16:14:21 -07:00
CONTRIBUTING.md ci: gate tests/e2e on zero basedpyright errors in pre-commit and lint CI 2026-07-11 10:25:22 -07:00
docker-compose.yml test(e2e): failed request error span carries the full untruncated message and status (LIT-4179) (#33304) 2026-07-14 22:13:13 -07:00
e2e_config.py test(e2e): otel trace completeness on /v1/messages (#33133) 2026-07-13 20:19:06 -07:00
e2e_gateway.py fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265) 2026-07-14 13:32:10 -07:00
e2e_http.py test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) 2026-07-14 20:23:54 -07:00
lifecycle.py test(e2e): add live batches suite across providers and routing scenarios (#30958) 2026-07-02 08:05:23 -07:00
models.py test(e2e): otel trace completeness on streaming chat, messages, and responses (LIT-3787) (#33234) 2026-07-14 20:23:54 -07:00
pytest.ini refactor(e2e): move budgets and spend_tracking suites under quota_management 2026-07-11 16:14:21 -07:00
test_e2e_gateway.py fix(e2e): bound spend-log snapshots to a /spend/logs/v2 window (#33265) 2026-07-14 13:32:10 -07:00
test_lifecycle.py test: add e2e tests for spend, budgets and llms (#30869) 2026-06-24 15:01:57 -07:00
test_transport.py test(e2e): probe the full spend read surface including schema-hidden routes (#32267) 2026-07-06 14:02:05 -07:00
transport.py test(e2e): probe the full spend read surface including schema-hidden routes (#32267) 2026-07-06 14:02:05 -07:00