From 2deeeb73a7678f8f802fc320a8966250bdfd8e92 Mon Sep 17 00:00:00 2001
From: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Date: Sat, 11 Jul 2026 17:12:38 -0700
Subject: [PATCH 1/4] docs(github): add QA runbook section to the PR template

---
 .github/pull_request_template.md | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index bd9fc2285d1..fcf894a4dca 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -41,3 +41,7 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
 ✅ Test
 
 ## Changes
+
+## QA runbook
+
+<!-- For PRs that change behavior (features, fixes, tests): list the manual steps a reviewer can follow to QA each change by hand against a live proxy, mapping 1:1 to what the automated proof asserts. One "- [ ]" checklist per test or behavior, each item a concrete action (route, request body, expected response). Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for examples. Delete this section if there is nothing to QA by hand -->

From 1d589832c7b5468f629cf9d028e479a9468f684e Mon Sep 17 00:00:00 2001
From: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Date: Sat, 11 Jul 2026 17:18:22 -0700
Subject: [PATCH 2/4] docs(github): scope the QA runbook to tests/e2e edits and
 add example checklists

---
 .github/pull_request_template.md | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index fcf894a4dca..0c72186d959 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -44,4 +44,18 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
 
 ## QA runbook
 
-<!-- For PRs that change behavior (features, fixes, tests): list the manual steps a reviewer can follow to QA each change by hand against a live proxy, mapping 1:1 to what the automated proof asserts. One "- [ ]" checklist per test or behavior, each item a concrete action (route, request body, expected response). Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for examples. Delete this section if there is nothing to QA by hand -->
+<!-- Only needed when your PR edits tests/e2e; delete this section otherwise
+
+For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one "- [ ]" checklist per test, each item a concrete action (route, request body, expected response). Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
+
+Example checklists:
+
+### test_key_rpm_limit_blocks_third_request
+- [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
+- [ ] Send three /v1/chat/completions requests with that key inside one minute
+- [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
+
+### test_model_create_appears_in_ui
+- [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
+- [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
+-->

From a6c8fb2d1c8b9c8c0c4fc3215e00f3f8d7eee7c8 Mon Sep 17 00:00:00 2001
From: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Date: Sat, 11 Jul 2026 17:21:36 -0700
Subject: [PATCH 3/4] docs(github): shape QA runbook examples as node id plus
 behavior bullets

---
 .github/pull_request_template.md | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index 0c72186d959..df44a4063a1 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -46,16 +46,16 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
 
 <!-- Only needed when your PR edits tests/e2e; delete this section otherwise
 
-For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one "- [ ]" checklist per test, each item a concrete action (route, request body, expected response). Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
+For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one top-level bullet per test giving its pytest node id followed by what it proves in plain words, then a nested "- [ ]" checklist where each item is a concrete action (route, request body, expected response). Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
 
 Example checklists:
 
-### test_key_rpm_limit_blocks_third_request
-- [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
-- [ ] Send three /v1/chat/completions requests with that key inside one minute
-- [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
+- tests/e2e/quota_management/ratelimit/test_rate_limit_e2e.py::TestKeyRateLimits::test_rpm_limit_blocks_over_limit - a key allowed 2 requests a minute serves exactly 2 and refuses the 3rd
+  - [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
+  - [ ] Send three /v1/chat/completions requests with that key inside one minute
+  - [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
 
-### test_model_create_appears_in_ui
-- [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
-- [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
+- tests/e2e/management/test_management_e2e.py::TestModelRoutes::test_model_create_appears_in_ui - a deployment created through the API shows up on the Admin UI models page
+  - [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
+  - [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
 -->

From 0008d96af4d969cc497d829f1a635f7cd3a7df5c Mon Sep 17 00:00:00 2001
From: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Date: Sat, 11 Jul 2026 18:07:37 -0700
Subject: [PATCH 4/4] docs(github): add Final Attestation and per-test
 sanity-check step to QA runbook

---
 .github/pull_request_template.md | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
index df44a4063a1..d7e80b32749 100644
--- a/.github/pull_request_template.md
+++ b/.github/pull_request_template.md
@@ -46,7 +46,7 @@ If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slac
 
 <!-- Only needed when your PR edits tests/e2e; delete this section otherwise
 
-For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one top-level bullet per test giving its pytest node id followed by what it proves in plain words, then a nested "- [ ]" checklist where each item is a concrete action (route, request body, expected response). Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
+For each e2e test you added or changed, list the manual steps a reviewer can follow to reproduce it by hand against a live proxy, mapping 1:1 to what the test asserts: one top-level bullet per test giving its pytest node id followed by what it proves in plain words, then a nested "- [ ]" checklist where each item is a concrete action (route, request body, expected response) and the final item is the sanity-check step shown in the examples. Note environment prerequisites (provider credentials, config flags) and any nuances a manual run will hit. See PRs #32914 and #32963 for full examples
 
 Example checklists:
 
@@ -54,8 +54,14 @@ Example checklists:
   - [ ] Generate a limited key: curl -X POST http://localhost:4000/key/generate -H "Authorization: Bearer sk-1234" -d '{"rpm_limit": 2}'
   - [ ] Send three /v1/chat/completions requests with that key inside one minute
   - [ ] Expect the first two to return 200 and the third to return 429 naming the rpm limit
+  - [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
 
 - tests/e2e/management/test_management_e2e.py::TestModelRoutes::test_model_create_appears_in_ui - a deployment created through the API shows up on the Admin UI models page
   - [ ] POST /model/new with the master key, a bedrock model, and aws_region_name (needs STORE_MODEL_IN_DB=True and AWS credentials)
   - [ ] Open http://localhost:4000/ui/?page=models and expect a deployment row showing the returned model id
+  - [ ] Sanity check: this test makes sense to add and is not hand-wavey (e.g., assert actual expected spend instead of just spend > 0) or potentially flaky
 -->
+
+### Final Attestation
+
+- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR