diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 12dff300afe..7a9883df356 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,27 +1,162 @@ - + -## What's the problem? +## TLDR -## What's the solution? + - +Problem this solves: -## How does it fix it? +- +- ... - +How it solves it: -## How does the product experience change? +- +- ... - +## User Flow -## What caveats are there, if any? + +Example: + +Before: a developer whose app streams chat completions gets no token counts back, so their cost dashboard reads zero + +1. They send POST https://litellm-domain/v1/chat/completions with `"stream": true` and no `stream_options` +2. The last SSE chunk arrives with `"usage": null`, so their app records 0 prompt and 0 completion tokens +3. They open https://litellm-domain/ui/?page=logs and see the request logged at $0 spend + +After: the same request comes back with real token counts, so the dashboard shows real spend + +1. The proxy admin sets `always_include_stream_usage: true` and restarts the proxy +2. The developer sends the same POST https://litellm-domain/v1/chat/completions with `"stream": true` and no `stream_options` +3. The last SSE chunk now carries a `usage` object with real prompt and completion token counts +4. https://litellm-domain/ui/?page=logs shows that request at non-zero spend +--> + +## Relevant issues + + + +## Affected release + + ## Linear ticket - + -## How did you test this? +## Pre-Submission checklist + +**Please complete all items before asking a LiteLLM maintainer to review your PR** + +- [ ] I have added meaningful tests +- [ ] The handful of test files covering my change pass locally, e.g. `uv run pytest tests/test_litellm/.py -v`. Leave the suites (`make test-unit-*`, `make test-unit`) to CI: it finishes in ~15 minutes where a laptop takes an hour or more +- [ ] My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.) +- [ ] My PR's scope is as isolated as possible; it only solves 1 specific problem +- [ ] I have received a Greptile **Confidence Score of at least 4/5** before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment `@greptileai` to re-request a review after pushing changes) + +## Delays in PR merge? + +If you're seeing a delay in your PR being merged, ping the LiteLLM Team on [Slack (#pr-review)](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA). + +## Screenshots / Proof of Fix + + + +## Type + + + + +๐Ÿ†• New Feature +๐Ÿ› Bug Fix +๐Ÿงน Refactoring +๐Ÿ“– Documentation +๐Ÿš„ Infrastructure +โœ… Test + +## Caveats (if any) + + + +## QA runbook + + + +## Final Attestation + +- [ ] The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR - diff --git a/AGENTS.md b/AGENTS.md index 9e0543753b0..cade08bdd02 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -39,7 +39,7 @@ Same applies for filing bug reports and feature requests, with .github/ISSUE_TEM If you're resolving a linear ticket, in the "## Linear ticket" section of the PR, say "Resolves LIT-1234", replacing "LIT-1234" with the actual ticket id that you're resolving. If you don't have the ticket id, don't make one up or search for it. Just leave the section blank -Never use `pytest` commands or the like as the answer to "How did you test this?". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`; the Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it +Never use `pytest` commands or the like as "Screenshots / Proof of Fix". We prefer curl'ing a live proxy instance running on localhost:4000 (I like to run it with `python litellm/proxy/proxy_cli.py --config litellm/proxy/dev_config.yaml --detailed_debug --reload --use_v2_migration_resolver 2>&1 | tee litellm.log`; the Admin UI dev server is `npm run dev` in `ui/litellm-dashboard`, served on port 3000) and showing both the command run and the output. Also, it should hit real LLM provider APIs, not mocks, and cost real $$$ because that is the most realistic test. The proof of fix should be exactly what the end user / customer would see / do. The run logs in PR #27703 is a prime example of how to do it (not a huge fan of using a python test script that future me and the team will have no visibility into; I prefer just curl commands or a short list of bash commands (e.g., using `for`)). If it's a UI thing, or the main use case runs through a headful agentic coding tool like Claude Code or Codex, drive that surface yourself and embed your own before and after screenshots of it in the PR (the Admin UI page, or what the coding tool shows), next to an ordered list of the URLs to go to (e.g., http://localhost:4000/ui/?page=logs), where to click, and what fields to fill out so a reviewer can reproduce it If you ever write any human-facing text (pull requests, issues, commit messages, discussion posts, github comments, release notes, docs, etc.), always follow these guidelines to sound less AI-y: - don't use emojis