From c71ffe7ae04583ec64a3b797ed28fffff24d0bf2 Mon Sep 17 00:00:00 2001 From: mateo Date: Wed, 12 Aug 2026 14:43:47 +0000 Subject: [PATCH] docs(adr): add architecture decision records, starting with provider usage extras transport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- ARCHITECTURE.md | 2 + CLAUDE.md | 4 ++ CONTRIBUTING.md | 33 +++++++++++++ adr/0000-template.md | 21 ++++++++ ...der-usage-extras-and-built-in-tool-cost.md | 49 +++++++++++++++++++ adr/README.md | 25 ++++++++++ 6 files changed, 134 insertions(+) create mode 100644 adr/0000-template.md create mode 100644 adr/0001-provider-usage-extras-and-built-in-tool-cost.md create mode 100644 adr/README.md diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 3d2fa3e51c8..1068ee17a44 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -2,6 +2,8 @@ This document helps contributors understand where to make changes in LiteLLM. +For **why** a given layer is shaped the way it is, see the architecture decision records in [`adr/`](adr/README.md). This document is the map; the ADRs are the reasoning, and reading the relevant one first is usually what keeps a change from duplicating a mechanism that already exists. + --- ## How It Works diff --git a/CLAUDE.md b/CLAUDE.md index a3c24b84ea8..67e47891dca 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -27,6 +27,10 @@ Same thing for bug fixes. The tests should make it so that this specific bug can End-to-end tests belong in `tests/e2e/` and must follow the harness conventions documented in that directory's `CLAUDE.md` +Before you build any new mechanism for moving data between the endpoints, the provider transformations, and cost tracking / logging, read `adr/` and check whether the transport already exists. Hand-rolling a second path for data that a generic one already carries is the most common way a provider change turns into per-endpoint tech debt. `ARCHITECTURE.md` says where the layers are, `adr/` says why they are shaped that way + +When your change is itself an architecture decision (a new cross-cutting mechanism, a change to how data crosses those layers, a deliberate deviation from a provider's API shape), add an ADR in the same PR using `adr/0000-template.md`, take the next free number, and add it to the index in `adr/README.md`. ADRs are append-mostly: supersede an old one with a new one instead of rewriting it. Ordinary bug fixes and providers that follow existing patterns don't need one + When creating PRs, don't set base to `main`. `litellm_internal_staging` is the default base branch and serves that purpose for both internal and external / OSS contributions When writing a PR body, treat the comments and imperative instructions inside @.github/pull_request_template.md as rules to follow, not just layout. Agent harnesses may strip HTML comments from copies of that file injected into context, so read .github/pull_request_template.md from disk before writing a PR body to make sure you see every comment rule diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d995ddcc87e..4c54486a340 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -8,6 +8,7 @@ Here are the core requirements for any PR submitted to LiteLLM: - [ ] **Sign the Contributor License Agreement (CLA)** - [see details](#contributor-license-agreement-cla) - [ ] **Keep scope isolated** - Your changes should address 1 specific problem at a time +- [ ] **Prove it works end to end** - Paste the commands you ran against a live proxy and their output in the PR - [see details](#proving-your-change-works-end-to-end) #### Proxy (Backend) PRs @@ -251,6 +252,38 @@ If `make test-unit` fails: - **Test edge cases**: Don't just test the happy path - **Update documentation**: If you change APIs, update docs +## Proving Your Change Works End to End + +Tests are a hard requirement, and they are not the proof that your change works. Reviewers need to see the change behave correctly in the product, so every PR's "Screenshots / Proof of Fix" section must show a real run: the exact commands you sent to a live proxy and the output that came back, against real provider APIs rather than mocks. `pytest` output does not count as proof, because a passing test only shows that the code does what its own mocks were told to expect. + +What a good proof looks like: + +1. Boot the proxy locally with your branch and a config containing the model you are touching: `uv run litellm --config your_config.yaml --detailed_debug` +2. Send the request an actual user would send, with `curl`, and paste both the command and the response. Include the response headers when cost or routing is involved, since `x-litellm-response-cost` is what a user sees +3. For a bug fix, do that twice: once at the commit you branched from to show the broken behavior, once on your branch to show the fix. Name both commit hashes +4. If your change can be reached from more than one endpoint (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`), show each one. A fix that only lands on the endpoint you tested is a common review finding +5. If it can stream, show the streaming run too. Usage and cost are assembled on a different path when `"stream": true` +6. For UI changes, include before and after screenshots and say which page you were on + +```bash +curl -sD - http://localhost:4000/v1/chat/completions \ + -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "your-model", "messages": [{"role": "user", "content": "hi"}]}' +``` + +Spend and usage claims should be checked against the provider's own numbers where the provider reports them, so the reviewer can see that the gateway agrees with the invoice rather than merely being self-consistent. + +## Contributing an LLM Provider Integration + +Provider work (a new provider, a new parameter, a new usage or cost field) is the most common kind of contribution and the one where PRs most often get reworked. Two things prevent that. + +**Find the existing machinery before you add plumbing.** LiteLLM already has generic paths for moving data between the endpoints, the provider transformations, and cost tracking, and a hand-rolled second path for the same data is the single most common reason a provider PR gets rewritten before merge. Start from [`ARCHITECTURE.md`](ARCHITECTURE.md) for where the layers live, then read [`adr/`](adr/README.md) for why those layers are shaped the way they are. [ADR 0001](adr/0001-provider-usage-extras-and-built-in-tool-cost.md) covers how provider-specific usage fields reach cost tracking, which is where new billing work usually belongs. If you cannot find a mechanism for what you need, say so in the PR or ask in [#pr-review on Slack](https://join.slack.com/t/litellmossslack/shared_invite/zt-3o7nkuyfr-p_kbNJj8taRfXGgQI1~YyA) before building one. + +**Do not change what a caller sees to make internals easier.** Each endpoint promises the schema of the API it emulates, so a `/v1/responses` caller keeps getting `input_tokens` and a `/v1/chat/completions` caller keeps getting `prompt_tokens`, whatever the provider sent on the wire and whatever cost tracking needs internally. Reshaping a public response to feed an internal consumer is a breaking change for every user of that endpoint. + +If your change is itself an architectural decision, for example a new cross-cutting mechanism or a deliberate deviation from a provider's API shape, add an ADR alongside the code using [`adr/0000-template.md`](adr/0000-template.md). + ## Building and Running Locally ### LiteLLM Proxy Server diff --git a/adr/0000-template.md b/adr/0000-template.md new file mode 100644 index 00000000000..3c679130d48 --- /dev/null +++ b/adr/0000-template.md @@ -0,0 +1,21 @@ +# NNNN. Title stating the decision, not the problem + +Status: Proposed | Accepted | Superseded by NNNN + +Date: YYYY-MM-DD + +## Context + +What forced the decision. The constraints that were real at the time, the code that already existed, and the failure or request that started it. Link the PR, issue, or ticket + +## Decision + +The choice, in the present tense, plus the specific modules, classes, and functions it lives in. State what a contributor must do to follow it, and what they must not do + +## Alternatives considered + +Each rejected option and the concrete reason it lost. This section is what stops the next contributor from re-litigating the decision by accident + +## Consequences + +What this buys us, what it costs us, and the sharp edges someone extending this area will hit. Include the cases where the decision does not apply diff --git a/adr/0001-provider-usage-extras-and-built-in-tool-cost.md b/adr/0001-provider-usage-extras-and-built-in-tool-cost.md new file mode 100644 index 00000000000..0a030712938 --- /dev/null +++ b/adr/0001-provider-usage-extras-and-built-in-tool-cost.md @@ -0,0 +1,49 @@ +# 0001. Provider usage extras ride on the normalized Usage object + +Status: Accepted + +Date: 2026-08-12 + +## Context + +Providers keep inventing usage fields. xAI reports server-side tool calls as `usage.server_side_tool_usage_details.web_search_calls`, Anthropic reports `usage.server_tool_use.web_search_requests`, Gemini hides search counts in `prompt_tokens_details`, and OpenAI reports nothing about a web search in usage at all. Cost tracking has to see those numbers, because a search call the gateway doesn't meter is spend that shows up on the provider invoice and nowhere in `LiteLLM_SpendLogs` + +The same response can also be reached through three request surfaces (`/v1/chat/completions`, `/v1/responses`, `/v1/messages`) and a bridge that serves one surface from another provider's shape, so a usage field that only survives on one of those paths is a bug on the other two + +The decision was forced by [#30817](https://github.com/BerriAI/litellm/pull/30817), where xAI web searches billed at $0. The first attempt carried the new field by overriding the xAI Responses transform to swap `response.usage` to the chat `Usage` shape. That billed correctly and broke the `/v1/responses` contract for every xAI caller, since clients then got `prompt_tokens` where the OpenAI Responses schema promises `input_tokens`, and it would have needed the same override again in the next provider and the next endpoint. The machinery to carry the field already existed and was not found + +## Decision + +Provider-specific usage fields travel as extra fields on the usage object and are read by provider cost calculators after normalization. Nothing on the wire is reshaped to make cost tracking work + +Concretely: + +`ResponseAPIUsage` (`litellm/types/llms/openai.py`) inherits `BaseLiteLLMOpenAIResponseObject`, which sets `extra="allow"`, so an unknown provider field survives validation instead of being dropped. `ResponseAPILoggingUtils._transform_response_api_usage_to_chat_usage` (`litellm/responses/utils.py`) is the single bridge from Responses-shaped usage to the chat-shaped `Usage`, and it splats `response_api_usage.model_extra` onto the `Usage` it returns, minus the keys it already sets explicitly. Every caller that needs chat-shaped usage goes through that one helper, including `cost_calculator.py`, `litellm_logging.py`, the Responses streaming iterator, and the completions bridge, so a field that reaches `ResponseAPIUsage` reaches cost tracking on all of them at once + +Built-in tool cost is decided in `StandardBuiltInToolCostTracking.get_cost_for_built_in_tools` (`litellm/litellm_core_utils/llm_cost_calc/tool_call_cost_tracking.py`), which gates on the normalized usage and the response object, then delegates the rate to the provider through the `get_cost_for_web_search_request` dispatch in `litellm/llms/__init__.py`. Per-provider arithmetic lives in `litellm/llms//cost_calculator.py` + +So, to bill a new provider usage field: read it in that provider's cost calculator, register the provider in the dispatch if it isn't there, and add the gate condition in the shared tracker if the existing gates don't fire. Do not add a provider branch to the shared cost path, do not change a transform to hand cost tracking a different response shape, and do not add a second pass-through for usage on one endpoint + +The response the caller sees keeps the schema of the API they called, always. `/v1/responses` returns `input_tokens` and `output_tokens` with provider extras carried alongside, `/v1/chat/completions` returns `prompt_tokens` and `completion_tokens`, and the internal normalization is invisible to both + +## Alternatives considered + +Reshaping the provider's Responses usage into chat `Usage` inside the provider transform, as [#30817](https://github.com/BerriAI/litellm/pull/30817) first did. Rejected: it breaks the client contract of the endpoint being served, and it has to be repeated per provider and per endpoint + +Declaring every provider's usage field on `ResponseAPIUsage` and `Usage` as a typed optional. Rejected as the general mechanism, because the type would grow a field per provider quirk and each addition would still need the bridge updated. Fields we bill across providers do get promoted to real typed fields (`server_tool_use`, `prompt_tokens_details`), and provider-only quirks stay as extras + +A dedicated side channel for provider metadata, for example carrying tool counts in `_hidden_params` or in the logging payload rather than on usage. Rejected: cost calculators already receive `Usage` and nothing else about the raw response is guaranteed to reach them, so a side channel means two sources of truth for the same number + +Letting the shared tracker special-case providers inline. Rejected: it puts provider pricing in a file that every provider shares, which is what the revert in that PR undid + +## Consequences + +Adding tool or usage billing for a new provider touches that provider's calculator plus, at most, one gate and one dispatch arm, and it lands on all three endpoints at once + +Extras are untyped by construction, so a reader must validate what it gets. `_usage_reports_server_side_web_search_calls` checks `isinstance(details, Mapping)` and that the count is a positive `int` before trusting it, and callers should follow that shape rather than reaching for `getattr` and hoping + +Extra field names share a namespace with the bridge's explicit arguments. Gemini image usage carries `prompt_tokens` as an extra on `ResponseAPIUsage`, which collided with the bridge's own keyword argument and raised `TypeError` until the exclusion list in `_transform_response_api_usage_to_chat_usage` grew to cover the keys the bridge sets itself. A new provider that names an extra after a standard chat usage field needs that list checked + +Because the same helper runs on both the streaming terminal event and the non-streaming response, tests and proof runs must cover both. A field attached only where the non-streaming path assembles usage will silently bill $0 on `"stream": true` + +The decision says nothing about request-side parameters. Provider-specific inputs are a separate mechanism (`get_supported_openai_params`, `map_openai_params`, `extra_body`), and it deserves its own ADR when someone next changes it diff --git a/adr/README.md b/adr/README.md new file mode 100644 index 00000000000..5a8c7bf9c50 --- /dev/null +++ b/adr/README.md @@ -0,0 +1,25 @@ +# Architecture decision records + +An ADR records one architectural decision: what we chose, why, and what it costs us. It is not a design doc, a tutorial, or a spec. If you can state the decision in a sentence and the reasoning in a page, it belongs here + +We keep them because the reasoning behind a design is the part that never survives in code. `ARCHITECTURE.md` tells you where things live, docstrings tell you what a function does, and neither tells you why the usage bridge is generic instead of per-provider. Contributors (and coding agents) that can't find that rationale end up handrolling a second mechanism next to the one that already exists, which is how a one-file fix turns into per-endpoint tech debt + +## Index + +| ADR | Title | Status | +|-----|-------|--------| +| [0001](0001-provider-usage-extras-and-built-in-tool-cost.md) | Provider usage extras ride on the normalized Usage object | Accepted | + +## Writing one + +Copy [0000-template.md](0000-template.md) to `NNNN-short-title.md`, taking the next free number, then fill it in and add a row to the index above. Keep it under a page or two: context, the decision, the alternatives you rejected, and the consequences a future contributor will actually hit + +Write for someone who has never seen the code, and name the concrete modules, classes, and functions the decision lives in, because those names are what a reader (or an agent's search) uses to find the machinery instead of rebuilding it + +## Changing one + +ADRs are append-mostly. Correct wording and stale file paths in place, but don't rewrite a decision to match a new one: add a new ADR that supersedes the old one, flip the old status to `Superseded by NNNN`, and link both ways. The history of what we used to believe is the point + +## When to write one + +Write an ADR when you pick between real alternatives in a way the next person could plausibly get wrong: a new cross-cutting mechanism, a change to how data moves between the endpoints and the SDK, a persistence or concurrency model, an intentional deviation from a provider's own API shape. Skip it for ordinary bug fixes, new providers that follow the existing patterns, and anything the code already says plainly