From f096e4c10bfc1f8db91f3ea3dc649aac1ce98a2b Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:27:07 +0000 Subject: [PATCH 01/10] docs: require a user flow and live-proxy proof in bug reports Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 60 ++++++++++++++++++++++----- CLAUDE.md | 2 + 2 files changed, 51 insertions(+), 11 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 665f8456f0b..4c1cc637f08 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -27,26 +27,64 @@ body: validations: required: true - type: textarea - id: steps-to-reproduce + id: user-flow attributes: - label: Steps to Reproduce - description: Please provide a numbered list of the exact steps to reproduce this bug (include a curl/python snippet to reproduce it). Number each step (1., 2., 3., ...) in the order you performed them. - placeholder: | - 1. config.yaml file/ .env file/ etc. - 2. Run the following code... - 3. Observe the error... + label: User Flow + description: Two numbered lists walking the same end user through the same task, one before a hypothetical fix and one after. Keep the guidance comments in the box while you fill it in, they explain every rule. value: | + + + Before a (hypothetical) fix: + + 1. + 2. + 3. + + After a (hypothetical) fix: + 1. 2. 3. validations: required: true - type: textarea - id: logs + id: proof-of-bug attributes: - label: Relevant log output - description: Please copy and paste any relevant log output. This will be automatically formatted into code, so no need for backticks. - render: shell + label: Proof the bug occurs + description: Paste the commands you ran and their output, captured against a live proxy with no mocks. Keep the guidance comments in the box while you fill it in, they explain every rule. + value: | + + + validations: + required: true - type: dropdown id: component attributes: diff --git a/CLAUDE.md b/CLAUDE.md index 436fa33fa41..e4f027715cc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,6 +31,8 @@ When creating PRs, don't set base to `main`. `litellm_internal_staging` is the d When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule +The same goes for filing a bug report: treat the comments and imperative instructions inside @.github/ISSUE_TEMPLATE/bug_report.yml as rules to follow, not just layout, and read that file from disk before writing an issue body so no stripped HTML comment escapes you + If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`; the Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, just tell me which URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, what fields to fill out, etc. along with the other commands to run in an ordered list, and I'll do it myself and post the screenshots after you make the PR From 7e8faf926740e48bf12f10ffb223534d2c3df233 Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:33:10 +0000 Subject: [PATCH 02/10] docs: ask bug reports for the config and version behind the proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 4c1cc637f08..7b3c9b072d8 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -78,7 +78,7 @@ body: From 7653532e89e134206002672ebb545ba6afc6db26 Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:35:44 +0000 Subject: [PATCH 03/10] docs: spell out secret redaction in the bug report proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 7b3c9b072d8..cedf04a40d6 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -78,10 +78,9 @@ body: + For UI bugs: include screenshots and the page URLs you were on. Scrub keys and tokens out of screenshots too, they show up in headers, request panels, and the Admin UI --> validations: required: true From 550682d5f8e7a41bc09e54a96179edb8eeada9d7 Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:36:11 +0000 Subject: [PATCH 04/10] docs: redact only sensitive env vars in bug report proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index cedf04a40d6..0044c2f2b88 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -78,7 +78,7 @@ body: From f8a5d6a6a1109fc9903960c2c5f3623f99fd820a Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:49:57 +0000 Subject: [PATCH 05/10] docs: make an unfilled bug report proof visibly empty and attested Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 0044c2f2b88..86f66764ab0 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -82,8 +82,21 @@ body: If the bug applies to more than one of the LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), include proof for every one of them, not just one For UI bugs: include screenshots and the page URLs you were on. Scrub keys and tokens out of screenshots too, they show up in headers, request panels, and the Admin UI --> + Config / setup the proxy ran with: + + Version or commit: + + Commands and their full output: + validations: required: true + - type: checkboxes + id: proof-attestation + attributes: + label: About that proof + options: + - label: It came from a live proxy I ran myself, with no mocks, and shows the real commands and their output rather than a `pytest` run + required: true - type: dropdown id: component attributes: From 7a17735473632d6002cdf38f27824f05e12db97b Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:56:13 +0000 Subject: [PATCH 06/10] fix(triage): treat an unfilled bug report scaffold as missing proof Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/scripts/triage_with_llm.py | 4 +++- tests/test_litellm/test_github_triage_with_llm.py | 5 +++++ 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/.github/scripts/triage_with_llm.py b/.github/scripts/triage_with_llm.py index d2536058e01..a7bd145dbc8 100644 --- a/.github/scripts/triage_with_llm.py +++ b/.github/scripts/triage_with_llm.py @@ -582,7 +582,9 @@ def build_issue_prompt(*, title: str, body: str) -> str: Commands whose external dependencies (LLM provider, DB, network) are mocked or stubbed do NOT count. Prose-only "steps to reproduce" with no run output, video, or - screenshot do NOT satisfy (1). + screenshot do NOT satisfy (1). An unfilled template scaffold + (bare headings such as "Version or commit:" with nothing under + them, empty numbered lists) counts as absent, not as evidence. (2) Expected vs. actual behavior (`has_expected_vs_actual`). FAIL the bug report if either (1) or (2) is missing. Do not bias diff --git a/tests/test_litellm/test_github_triage_with_llm.py b/tests/test_litellm/test_github_triage_with_llm.py index f50cf126c36..300fd7c0710 100644 --- a/tests/test_litellm/test_github_triage_with_llm.py +++ b/tests/test_litellm/test_github_triage_with_llm.py @@ -672,6 +672,11 @@ class TestBuildPrompts: assert "mocked or stubbed" in normalized # Prose-only steps are explicitly insufficient now. assert "steps to reproduce" in normalized + # An unedited issue-form scaffold must not read as evidence: the proof + # field ships with visible headings, so the judge has to be told that + # bare headings with nothing under them count as absent. + assert "unfilled template scaffold" in normalized + assert "counts as absent, not as evidence" in normalized def test_should_not_crash_when_pr_body_contains_curly_braces(self, triage_module): """User-supplied content with `{` / `}` must NOT be re-parsed by From 5643a59aa4e0a7322a14ac0645dad21631cd66f8 Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 02:58:21 +0000 Subject: [PATCH 07/10] fix: move bug report guidance out of prefilled values so required means filled Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 61 ++++++++++----------------- CLAUDE.md | 2 +- 2 files changed, 23 insertions(+), 40 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 86f66764ab0..7e08f9297f8 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -23,71 +23,54 @@ body: label: What happened? description: Also tell us, what did you expect to happen? placeholder: Tell us what you see! - value: "A bug happened!" validations: required: true - type: textarea id: user-flow attributes: label: User Flow - description: Two numbered lists walking the same end user through the same task, one before a hypothetical fix and one after. Keep the guidance comments in the box while you fill it in, they explain every rule. - value: | - - - Before a (hypothetical) fix: - - 1. - 2. - 3. - - After a (hypothetical) fix: - - 1. - 2. - 3. validations: required: true - type: textarea id: proof-of-bug attributes: label: Proof the bug occurs - description: Paste the commands you ran and their output, captured against a live proxy with no mocks. Keep the guidance comments in the box while you fill it in, they explain every rule. - value: | - + description: | + The commands (e.g., curl) and their full output, screenshots, or a screen recording demonstrating that the bug happens. Every rule below applies. + - The proof must be completely e2e with no mocks, against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), hitting real LLM provider APIs and costing real $ where the bug involves a provider call. `pytest` commands are not enough + - Show exactly what the end user sees or does, matching the User Flow above step for step + - Start with the config.yaml (or SDK setup) and any env vars the proxy ran with, then the exact version or commit hash the proof was captured at, so a maintainer can stand up the same proxy before running your commands. Keep the real values for env vars that aren't sensitive, they are often the reason the bug happens, and redact only the secrets: never paste a real API key, virtual key, database URL, or other credential, here or anywhere else in the issue + - If the bug applies to more than one of the LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), include proof for every one of them, not just one + - For UI bugs: include screenshots and the page URLs you were on. Scrub keys and tokens out of screenshots too, they show up in headers, request panels, and the Admin UI + placeholder: | Config / setup the proxy ran with: Version or commit: Commands and their full output: - validations: required: true - type: checkboxes diff --git a/CLAUDE.md b/CLAUDE.md index e4f027715cc..3941060afa2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,7 +31,7 @@ When creating PRs, don't set base to `main`. `litellm_internal_staging` is the d When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule -The same goes for filing a bug report: treat the comments and imperative instructions inside @.github/ISSUE_TEMPLATE/bug_report.yml as rules to follow, not just layout, and read that file from disk before writing an issue body so no stripped HTML comment escapes you +The same goes for filing a bug report: treat every field's `description` and `placeholder` in @.github/ISSUE_TEMPLATE/bug_report.yml as rules to follow, not just layout, and read that file from disk before writing an issue body, since the rendered form and any copy injected into your context can drop or reflow that guidance If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank From 0a41b19e5b1d669a0ec59206b8235b128d4e6889 Mon Sep 17 00:00:00 2001 From: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Date: Tue, 11 Aug 2026 08:33:22 -0700 Subject: [PATCH 08/10] chore: make it more brief --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 3941060afa2..0e63b86036d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,7 +31,7 @@ When creating PRs, don't set base to `main`. `litellm_internal_staging` is the d When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule -The same goes for filing a bug report: treat every field's `description` and `placeholder` in @.github/ISSUE_TEMPLATE/bug_report.yml as rules to follow, not just layout, and read that file from disk before writing an issue body, since the rendered form and any copy injected into your context can drop or reflow that guidance +Same applies for filing bug reports and .github/ISSUE_TEMPLATE/bug_report.yml If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank From 10d4213453bcc9bd321892df9682a1f753d70e77 Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 16:25:52 +0000 Subject: [PATCH 09/10] chore: drop the bug report proof attestation and tighten its wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 13 +++---------- 1 file changed, 3 insertions(+), 10 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 7e08f9297f8..48818144caa 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -37,7 +37,7 @@ body: - Every step is something the user does or observes: the HTTP method and full URL they hit, what they sent, and what visibly came back (status code, error text, the shape of an ID). UI steps name the page URL and what is on screen - No LiteLLM internals: never name functions, files, DB tables, config classes, hooks, callbacks, or code paths. "The upload hands back an ID that looks like OpenAI's own `file-abc123` instead of the scrambled one the gateway returned" is right, "no managed-file row was registered" is wrong - Keep the two lists step-for-step identical until they diverge, so the broken step is obvious - - If the bug has a security or authorization consequence, end each list with what another user can or could no longer do + - If the bug has a security or authorization consequence, end each list with what another user can do that they shouldn't be able to, and what they could no longer do after a fix placeholder: | Before a (hypothetical) fix: a developer whose app streams chat completions gets no token counts back, so their cost dashboard reads zero @@ -60,11 +60,11 @@ body: description: | The commands (e.g., curl) and their full output, screenshots, or a screen recording demonstrating that the bug happens. Every rule below applies. - - The proof must be completely e2e with no mocks, against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), hitting real LLM provider APIs and costing real $ where the bug involves a provider call. `pytest` commands are not enough + - The proof must be completely e2e with no mocks, against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), hitting real LLM provider APIs, costing real $$$ if needed, where the bug involves a provider call. `pytest` commands are not enough - Show exactly what the end user sees or does, matching the User Flow above step for step - Start with the config.yaml (or SDK setup) and any env vars the proxy ran with, then the exact version or commit hash the proof was captured at, so a maintainer can stand up the same proxy before running your commands. Keep the real values for env vars that aren't sensitive, they are often the reason the bug happens, and redact only the secrets: never paste a real API key, virtual key, database URL, or other credential, here or anywhere else in the issue - If the bug applies to more than one of the LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), include proof for every one of them, not just one - - For UI bugs: include screenshots and the page URLs you were on. Scrub keys and tokens out of screenshots too, they show up in headers, request panels, and the Admin UI + - For UI bugs: include screenshots and the page URLs you were on. Scrub keys and tokens out of screenshots too (for example, the virtual key is briefly shown in the panel right after you create a virtual key) placeholder: | Config / setup the proxy ran with: @@ -73,13 +73,6 @@ body: Commands and their full output: validations: required: true - - type: checkboxes - id: proof-attestation - attributes: - label: About that proof - options: - - label: It came from a live proxy I ran myself, with no mocks, and shows the real commands and their output rather than a `pytest` run - required: true - type: dropdown id: component attributes: From ad1ff191952c91c6c03cd0837637f12cc82ab4a6 Mon Sep 17 00:00:00 2001 From: mateo Date: Tue, 11 Aug 2026 16:31:19 +0000 Subject: [PATCH 10/10] chore: say real $ instead of $$$ in the bug report proof rules Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- .github/ISSUE_TEMPLATE/bug_report.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index 48818144caa..b93e4add9a7 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -60,7 +60,7 @@ body: description: | The commands (e.g., curl) and their full output, screenshots, or a screen recording demonstrating that the bug happens. Every rule below applies. - - The proof must be completely e2e with no mocks, against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), hitting real LLM provider APIs, costing real $$$ if needed, where the bug involves a provider call. `pytest` commands are not enough + - The proof must be completely e2e with no mocks, against a live proxy you ran yourself (e.g., `litellm --config config.yaml --detailed_debug` on localhost:4000), hitting real LLM provider APIs, costing real $ if needed, where the bug involves a provider call. `pytest` commands are not enough - Show exactly what the end user sees or does, matching the User Flow above step for step - Start with the config.yaml (or SDK setup) and any env vars the proxy ran with, then the exact version or commit hash the proof was captured at, so a maintainer can stand up the same proxy before running your commands. Keep the real values for env vars that aren't sensitive, they are often the reason the bug happens, and redact only the secrets: never paste a real API key, virtual key, database URL, or other credential, here or anywhere else in the issue - If the bug applies to more than one of the LLM endpoints (/v1/responses, /v1/chat/completions, /v1/messages), include proof for every one of them, not just one