Every bedrock batch output line went through the Anthropic usage parser, which
reads snake_case input_tokens/output_tokens. Converse-family models (Nova and
friends) report camelCase inputTokens/outputTokens, so their usage came back
0/0/0 and the batch billed $0 despite real token consumption.
Usage is now selected by the shape of the payload: a Converse-shaped block goes
through the same transform the live Converse path uses, so a batch and an
equivalent non-batch call agree on tokens, including cache reads and writes.
Anthropic-shaped bedrock output is unchanged.
A shape neither parser understands (an InvokeModel-native payload from Titan,
Cohere, or Llama, which name their counts differently again) still reads zero,
but now warns with the keys it saw instead of silently billing $0.
Exposes the Converse usage transform as public, since batch parsing is a second
legitimate caller; that also removes the private-member access invoke_handler
was already making.
Resolves the transform_create_file_response conflict by keeping the
_uploaded_object_size handoff over the response Content-Length read,
and adds the rebind-ok justification LIT011 now requires for the
upload-size litellm_params handoff after the base budget ratcheted.
* fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5
The Bedrock twins of both models already carry
supports_native_structured_output, but the Anthropic-direct entries do not,
so response_format requests to anthropic/claude-sonnet-5 and
anthropic/claude-haiku-4-5 fall back to the json_tool_call emulation and
inherit its nested-envelope failure modes (#8898) despite the API supporting
output_format natively.
Verified live against the Anthropic API on 2026-08-05: both models accept
output_format (structured outputs beta header) and return exact schema
instances, including a large nested production schema validated with
pydantic. Same two lines applied to the bundled backup map.
* fix(model_map): cover the versioned claude-haiku-4-5-20251001 alias
Exact-match capability lookup of anthropic/claude-haiku-4-5-20251001
resolved the versioned entry, which lacked the flag, so response_format
for that identifier still took the tool-emulation path. Flag it in both
the root and bundled maps, matching its unversioned alias.
* fix(anthropic): bound $defs inlining in output_format with the shared schema-bomb budget
map_response_format_to_anthropic_output_format called unpack_defs with
no max_inlined_bytes, so an authenticated caller could send a compact
schema whose repeated $refs expand without bound before reaching the
provider. Reuse the existing 10MB inlining budget (renamed from
_LEGACY_DEFS_MAX_INLINED_BYTES to DEFS_MAX_INLINED_BYTES now that two
call sites share it); overflow raises ValueError instead of
materialising the expansion.
Regression tests: a compact schema bomb is rejected, a normal $defs
schema still resolves; the bomb test fails when the bound is removed.
* chore: retrigger CI (benchmarks job flaked on a PyPI download timeout)
---------
Co-authored-by: Anmol Jaiswal <anmolg1997@users.noreply.github.com>
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
* feat(search): add Nimble as a search provider
Adds `NimbleSearchConfig` so `search_provider: nimble` works across the SDK,
the proxy /v1/search endpoint, the Search Tools dashboard, and spend tracking.
Nimble's /v2/search already uses the Perplexity unified spec's parameter names,
so the request transform is close to a pass-through. `search_domain_filter`
splits into include_domains/exclude_domains on the spec's `-` prefix, `country`
is upper-cased to the ISO form Nimble documents, and everything else is
forwarded so focus, search_depth, time_range and the rest stay reachable. On the
response side, snippet prefers `content` and falls back to `description`, and a
malformed body raises an attributed error rather than reporting an empty search.
Also tightens `BaseSearchConfig.get_supported_perplexity_optional_params` to
return `frozenset[str]` instead of a bare mutable `set`, which every caller
already treats as read-only.
* fix(search): surface Nimble error bodies instead of empty results
Greptile flagged that a null or absent `results` degraded to a successful empty
search. A search with no hits comes back as `"results": []`, verified against the
live API, so the field is now required and anything else raises the attributed
schema error the other malformed bodies already take.
Also unwraps Nimble's second error envelope. Collection failures return
`{"success", "task_id", "message"}` rather than the `{"detail"}` shape validation
errors use, and only the latter was being read.
Drops comments that restated the adjacent code.
* docs(search): drop the Nimble param list from the transform docstring
It restated the vendor's API reference, which the module docstring already links,
and would go stale the moment Nimble adds a focus mode.
Anthropic's Models API declares max_input_tokens and max_tokens as nullable, not
optional, and the live vendor endpoint returns both keys on every entry. The
merged Anthropic-native listing dropped either key whenever LiteLLM could not
resolve a limit, so a client validating against a nullable-but-required schema
saw a malformed entry for any model the cost map does not know.
OpenAI and Azure GPT-5.x answer a chat request whose output budget cannot fit a
single visible token with a 400, while the same models return a length-truncated
200 one or two tokens higher. Agents that probe a model with a hardcoded
max_tokens of 1 read that 400 as "model unavailable".
The four chat request helpers now recognise the provider's own sentence and hand
back the length-truncated response the provider gives at a slightly larger
budget: finish_reason "length", empty content, zero completion tokens. Any other
400 still raises. Streaming is covered by the same seam, and the caller's budget
is never raised on their behalf.
The provider bills the prompt it processed but sends no usage object with the
400, so the prompt tokens are estimated with the same token_counter every other
usage-less path uses. Reporting zero would let a caller send an arbitrarily
large prompt with max_tokens 1 and be charged nothing.
cost_per_second treated a declared-but-zero output_cost_per_second as a real
rate, so the output branch claimed the call and the elif locked out
input_cost_per_second. Every transcription model shipping
output_cost_per_second 0.0 next to a real input rate billed $0, which covers
43 of the 55 per-second entries in the cost map: all 36 deepgram models, both
assemblyai, both elevenlabs scribe, both groq whisper and azure-stt. Custom
deployments pairing the two fields the same way billed $0 as well
Take the output branch only when that rate is actually billable, so a zero
falls through to the input rate. Entries that duplicate one rate into both
fields, whisper-1 among them, keep billing exactly what they bill today
The two new `translate_tools_to_responses_api` calls carried
`# type: ignore[arg-type]`, which CLAUDE.md bans as LIT009: pyrightconfig.json
sets enableTypeIgnoreComments to false, so the comment silently does nothing and
the reportArgumentType error stands. Annotating the fixtures as
list[AllAnthropicToolsValues] makes both calls check clean with no suppression
at all.
Translating Anthropic tools left the outbound function-tool `strict` unset,
which the Responses API does not read as non-strict. OpenAI's function-calling
docs say strict mode requires every field in `properties` to be marked
required, and with `strict` omitted the schema gets normalized to satisfy that
instead of being rejected. What users see is a tool whose `required` lists
every property, so models fill optional Anthropic tool arguments with empty
values. Send `strict` explicitly so an unset value stays non-strict and an
explicit `strict: true` still reaches the provider
On the Chat Completions adapter, `strict` was also missing from
`mapped_tool_params`, so a tool-level `strict` was merged into the OpenAI
function `parameters` schema (mutating the caller's `input_schema` along the
way) instead of being set on the function. Map it to `function.strict` and
leave it unset when the caller omits it, since Chat Completions already
defaults to non-strict
* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery
* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils
* fix(proxy): make model_list request param optional for direct callers
* style: apply ruff format to changed lines
* style: satisfy ruff strict-rule budget (UP006, I001)
* style: satisfy type-discipline budget (LIT002 mutable-ok, LIT009 pyright ignore)
* style: satisfy LIT001/LIT010 and drop explanatory comment per contributor rules
* fix(proxy): translate team model names in the Anthropic /v1/models response
* ci: trigger buildkite status report
* feat(proxy): carry token limits into the Anthropic-native /v1/models entries
* fix(proxy): cast the injected request so the anthropic-version guard is a real comparison
* fix(proxy): explain the model listing casts so the type-discipline gate passes
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
chunk_parser built ModelResponseStream without passing usage, so the
cache_read_input_tokens and cache_creation_input_tokens that Databricks
returns for Anthropic models never reached the cost calculator. Every
streamed request was billed at the full input rate even when served
from cache.
ModelResponseStream already coerces a usage dict into Usage, which maps
those keys into prompt_tokens_details, so passing the chunk's usage
through is sufficient.
* feat(azure_ai): add Fireworks FW model pricing on Azure AI Foundry
* fix(azure_ai): drop incorrect FW-Kimi-K2.6-Code alias
* test(azure-ai): assert FW max token metadata
* feat(azure_ai): add Inkling and Nemotron 3 Ultra pricing
Address review feedback on #36762:
- Only use the parsed 5m/1h split when it fully accounts for
cacheWriteInputTokens; an unrecognized ttl or missing entry now falls
back to the aggregate (previous behavior) instead of silently
understating cost.
- Mark TypedDict fields ReadOnly (AWS response data, never constructed
by us) to satisfy the repo's type-discipline lint gate.
- Trim comments and add Final to locals per repo style.
Co-Authored-By: pi (Claude/GPT via @earendil-works/pi-coding-agent) <noreply@earendil.works>
