From 3a57d5c9ea50d77634fab8de34b881750424c81c Mon Sep 17 00:00:00 2001 From: Yujong Lee Date: Sun, 30 Aug 2026 09:46:36 -0700 Subject: [PATCH] initiai readme --- tests/route_parity/README.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 tests/route_parity/README.md diff --git a/tests/route_parity/README.md b/tests/route_parity/README.md new file mode 100644 index 00000000000..3fc8e67b972 --- /dev/null +++ b/tests/route_parity/README.md @@ -0,0 +1,29 @@ +# Python/Python parity testing in Python SDK interface + +> Given the same SDK call and identical provider behavior, does the PyO3 implementation behave same as Python? + +## What the harness compares + +- A fixture contains a LiteLLM SDK input and a recorded upstream provider response +- The same LiteLLM input is transformed by isolated Python and Rust workers +- The resulting provider requests must match in method, path, headers, and body, excluding runtime-specific HTTP metadata +- The recorded provider response is then replayed unchanged to both workers +- Each worker serializes its normalized LiteLLM SDK response to JSON, and the results must match + +## Hypothesis and property-based testing + +- Hypothesis is Python libary for property-based testing +- Example-based tests use inputs selected by the test author +- Property-based tests define strategies for valid inputs and properties that must hold for every generated example +- Hypothesis generates combinations from those strategies and normally shrinks a failing example to a smaller reproducible case +- In this harness, Hypothesis is used only during fixture generation to expand the LiteLLM input corpus +- The current OCR strategy varies supported optional parameters while keeping inputs valid +- Fixture generation is deterministic, and each generated input is recorded with the raw provider response it received +- The parity tests use committed fixtures and do not call the provider or generate new Hypothesis examples +- Provider responses are replayed unchanged, so the parity test does not fuzz or validate provider behavior +- Because Hypothesis does not run the parity assertion directly, parity failures are not automatically shrunk + +## References + +- [Hypothesis documentation](https://hypothesis.readthedocs.io/en/latest/) +- [Hypothesis quick start](https://hypothesis.readthedocs.io/en/latest/quickstart.html)