048a1500df
* test(e2e): move the harness self-tests out of tests/e2e The nightly Buildkite run copies tests/e2e into the runner image and runs bare pytest, so the 672 tests of the harness itself (fixture parsing, JUnit properties, the stack lock, the load aggregators, the Claude Code driver) counted as e2e tests on the status page even though none of them reaches a proxy. They now live in tests/e2e_harness, mirroring the tests/e2e layout, and run in the GitHub Actions lint job and the CircleCI provider_replay_harness job instead * fix(ci): point the providers replay controls at tests/e2e_harness The providers integration job still selected the four replay-control tests under tests/e2e/test_provider_edge.py, so pytest exited before they ran. The raw-HTTP check's file walk also drops to one loop per comprehension * style(tests): mark the raw-HTTP check's bindings Final
2 KiB
Tests
Nothing on the other side of the call: tests/unit. A proxy we start with an upstream we script:
tests/integration. Someone else's service with real credentials: tests/e2e. Tests of that harness
itself, no proxy at all: tests/e2e_harness. Two fit, split it
What good looks like
Red when the claim in the name is broken. Prove it: mutate the behaviour, red; restore, green. Put the mutation in the PR body
def test_custom_price_is_reported_and_charged(gateway: Gateway) -> None:
with gateway.scenario() as scenario:
model = scenario.model(input_cost_per_token=0.001, output_cost_per_token=0.002)
response = gateway.request("POST", "/v1/chat/completions", {"model": model, "messages": [{"role": "user", "content": "price control"}]})
assert response.status_code == 200, response.text
assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(20 * 0.001 + 20 * 0.002)
Rates in the test, expected computed by hand, one call, response.text in the assert
Assert the whole value. Iterating expected_body.items() (test_responses_api_request_body.py) cannot
see an extra key; that is the shape of stream_options.include_usage (#19777, #28553)
The linter catches no-assert, mock-echo and credential skips. It cannot see an assert
behind an if (a poll that ends in pytest.fail is fine), except Exception around the call
(test_router.py: except Exception as e: print(f"FAILED TEST")), or blanket --reruns
Where it goes
What the assertion depends on goes in the test; everything else in conftest. A rate in a fixture three directories up makes a failed assertion unreadable. Extend the file that already covers the behaviour
Writing it so a human can read it
Name says what broke: test_send_batched_with_valid_data says nothing. Build, one call, assert, on one
screen. Helpers named for what they return, _pii_prompt(marker, email), not _setup(). Context in the
assert message, not a comment