Merge pull request #42099 from BerriAI/litellm_test_tier_contract

docs(tests): define the tier contract for unit, integration and e2e

tests/AGENTS.md

@@ -0,0 +1,38 @@
+# Tests
+
+Nothing on the other side of the call: `tests/unit`. A proxy we start with an upstream we script:
+`tests/integration`. Someone else's service with real credentials: `tests/e2e`. Two fit, split it
+
+## What good looks like
+
+Red when the claim in the name is broken. Prove it: mutate the behaviour, red; restore, green. Put the
+mutation in the PR body
+
+```python
+def test_custom_price_is_reported_and_charged(gateway: Gateway) -> None:
+    with gateway.scenario() as scenario:
+        model = scenario.model(input_cost_per_token=0.001, output_cost_per_token=0.002)
+        response = gateway.request("POST", "/v1/chat/completions", {"model": model, "messages": [{"role": "user", "content": "price control"}]})
+        assert response.status_code == 200, response.text
+        assert float(response.headers["x-litellm-response-cost"]) == pytest.approx(20 * 0.001 + 20 * 0.002)
+```
+
+Rates in the test, expected computed by hand, one call, `response.text` in the assert
+
+Assert the whole value. Iterating `expected_body.items()` (`test_responses_api_request_body.py`) cannot
+see an extra key; that is the shape of `stream_options.include_usage` (#19777, #28553)
+
+The linter catches no-assert, mock-echo, credential skips and patched internals. It cannot see an assert
+behind an `if` (a poll that ends in `pytest.fail` is fine), `except Exception` around the call
+(`test_router.py`: `except Exception as e: print(f"FAILED TEST")`), or blanket `--reruns`
+
+## Where it goes
+
+What the assertion depends on goes in the test; everything else in conftest. A rate in a fixture three
+directories up makes a failed assertion unreadable. Extend the file that already covers the behaviour
+
+## Writing it so a human can read it
+
+Name says what broke: `test_send_batched_with_valid_data` says nothing. Build, one call, assert, on one
+screen. Helpers named for what they return, `_pii_prompt(marker, email)`, not `_setup()`. Context in the
+assert message, not a comment

tests/e2e/AGENTS.md

@@ -2,6 +2,30 @@
 
 Code-style rules for writing tests under `tests/e2e/`. The harness already encodes the plumbing; your job is the feature-specific behavior, not reinventing it. For what a complete test must do (the lifecycle contract, asserting both recorded state and enforced behavior) and how to run a suite, see `CONTRIBUTING.md` in this directory. Repo-wide conventions live in the root `AGENTS.md`
 
+## What good looks like
+
+Only what a real provider proves. If it holds against our scripted upstream: `tests/integration`
+
+```python
+def test_pre_call_masks_pii_on_chat_completions(self, client: GuardrailsClient, resources: ResourceManager, scoped_key: str) -> None:
+    name = f"e2e-presidio-pre-chat-{unique_marker()}"
+    _register_presidio(client, resources, name=name)
+    email = _fake_email()
+    _assert_eventually_masked(
+        lambda: client.chat(scoped_key, MODEL, _pii_prompt(unique_marker(), email), guardrails=[name], max_tokens=128),
+        _first_content,
+        email=email,
+    )
+```
+
+Marker per run, so a leftover guardrail cannot pass it. `resources.defer(...)` at creation, so a failed
+assert still tears down. Assert what the caller receives
+
+## Where it goes
+
+By the surface a customer would name: `guardrails`, `llm_translation`, `management`. Mutation check
+deferred; it needs credentials
+
 ## Suite folders
 
 Each subdirectory under `tests/e2e/` is one suite, scoped to an endpoint family or behavior area. If you add a new folder, you must add a line here describing what kind of tests belong in it, so the layout stays self-describing. `gateway/` is the exception: it holds proxy configuration only and never tests

tests/integration/AGENTS.md

@@ -0,0 +1,25 @@
+# tests/integration
+
+Real proxy, Postgres, Redis, scripted upstream. `README.md` has shards and CI wiring
+
+## What good looks like
+
+The root example is from here. The spend row lands async: poll, never sleep
+
+```python
+rows = eventually(
+    lambda: read_rows('SELECT spend FROM "LiteLLM_SpendLogs" WHERE request_id=%s', (body["id"],)),
+    lambda values: len(values) == 1,
+    seconds=70,
+)
+assert float(rows[0]["spend"]) == pytest.approx(10 * 0.001 + 5 * 0.0001 + 7 * 0.002 + 4 * 0.002)
+```
+
+`sleep(3)` fails on a slow runner and taxes every fast one. Assert the outbound body in the upstream
+handler; a leaked field is invisible from the response. `monkeypatch.setenv` is fine; patching our own
+function in a full stack is not
+
+## Where it goes
+
+By the domain a user would name: `pricing`, `spend`, `routing`. Add the node and its `covers` ids to
+`contracts.json` or collection fails. Needs no proxy, DB or Redis: `tests/unit`

tests/unit/AGENTS.md

@@ -0,0 +1,37 @@
+# tests/unit
+
+In-process. No network, clock or subprocess
+
+## What good looks like
+
+```python
+def test_send_result_same_version_is_identity_passthrough():
+    rpc = _rpc(V03_MESSAGE)
+    out = normalize_jsonrpc_response(rpc, "0.3", method="message/send")
+    assert out is rpc
+```
+
+`is`, because `==` passes on a copy. Many inputs: parametrize
+(`test_an_incomplete_reservation_accrues_nothing`, fifteen cases, fifteen results)
+
+No doubles on our own code
+
+```python
+with patch.object(streamer, "_group_by_date") as mock_group, patch.object(streamer, "_send_daily_batch") as mock_send:
+    mock_group.return_value = {"2025-01-19": pl.DataFrame({"test": ["data1"]}), "2025-01-20": pl.DataFrame({"test": ["data2"]})}
+    streamer.send_batched(pl.DataFrame({"test": ["data"]}), "replace_hourly")
+    assert mock_send.call_count == 2
+```
+
+Green if `send_batched` drops every row. pydantic doubles in 12 of 203 files, fastapi 11 of 594;
+`tests/test_litellm` 59 percent. Exception: the count is the behaviour
+(`test_dual_cache_async_batch_get_cache_coalesces_concurrent_redis_reads`, fifty readers, `call_count == 1`)
+
+## Where it goes
+
+`tests/unit/<path>` mirrors `litellm/<path>`, so a changed file selects its tests by path, not a mapping
+file. Empty today; new unit tests go here. The examples above live in `tests/test_litellm`
+
+## Writing it so a human can read it
+
+A class only when tests share an arrange