* fix(model_map): flag native structured outputs on Anthropic-direct claude-sonnet-5 and claude-haiku-4-5
The Bedrock twins of both models already carry
supports_native_structured_output, but the Anthropic-direct entries do not,
so response_format requests to anthropic/claude-sonnet-5 and
anthropic/claude-haiku-4-5 fall back to the json_tool_call emulation and
inherit its nested-envelope failure modes (#8898) despite the API supporting
output_format natively.
Verified live against the Anthropic API on 2026-08-05: both models accept
output_format (structured outputs beta header) and return exact schema
instances, including a large nested production schema validated with
pydantic. Same two lines applied to the bundled backup map.
* fix(model_map): cover the versioned claude-haiku-4-5-20251001 alias
Exact-match capability lookup of anthropic/claude-haiku-4-5-20251001
resolved the versioned entry, which lacked the flag, so response_format
for that identifier still took the tool-emulation path. Flag it in both
the root and bundled maps, matching its unversioned alias.
* fix(anthropic): bound $defs inlining in output_format with the shared schema-bomb budget
map_response_format_to_anthropic_output_format called unpack_defs with
no max_inlined_bytes, so an authenticated caller could send a compact
schema whose repeated $refs expand without bound before reaching the
provider. Reuse the existing 10MB inlining budget (renamed from
_LEGACY_DEFS_MAX_INLINED_BYTES to DEFS_MAX_INLINED_BYTES now that two
call sites share it); overflow raises ValueError instead of
materialising the expansion.
Regression tests: a compact schema bomb is rejected, a normal $defs
schema still resolves; the bomb test fails when the bound is removed.
* chore: retrigger CI (benchmarks job flaked on a PyPI download timeout)
---------
Co-authored-by: Anmol Jaiswal <anmolg1997@users.noreply.github.com>
Images nested inside an Anthropic `tool_result` block were dropped when the
request was adapted for an OpenAI-compatible provider, because the OpenAI tool
message shape only carried text. Hoist those images out of the tool result and
into a following user message so the model can still see them, and widen the
tool message content type to accept image parts.
* feat(search): add Nimble as a search provider
Adds `NimbleSearchConfig` so `search_provider: nimble` works across the SDK,
the proxy /v1/search endpoint, the Search Tools dashboard, and spend tracking.
Nimble's /v2/search already uses the Perplexity unified spec's parameter names,
so the request transform is close to a pass-through. `search_domain_filter`
splits into include_domains/exclude_domains on the spec's `-` prefix, `country`
is upper-cased to the ISO form Nimble documents, and everything else is
forwarded so focus, search_depth, time_range and the rest stay reachable. On the
response side, snippet prefers `content` and falls back to `description`, and a
malformed body raises an attributed error rather than reporting an empty search.
Also tightens `BaseSearchConfig.get_supported_perplexity_optional_params` to
return `frozenset[str]` instead of a bare mutable `set`, which every caller
already treats as read-only.
* fix(search): surface Nimble error bodies instead of empty results
Greptile flagged that a null or absent `results` degraded to a successful empty
search. A search with no hits comes back as `"results": []`, verified against the
live API, so the field is now required and anything else raises the attributed
schema error the other malformed bodies already take.
Also unwraps Nimble's second error envelope. Collection failures return
`{"success", "task_id", "message"}` rather than the `{"detail"}` shape validation
errors use, and only the latter was being read.
Drops comments that restated the adjacent code.
* docs(search): drop the Nimble param list from the transform docstring
It restated the vendor's API reference, which the module docstring already links,
and would go stale the moment Nimble adds a focus mode.
Anthropic's Models API declares max_input_tokens and max_tokens as nullable, not
optional, and the live vendor endpoint returns both keys on every entry. The
merged Anthropic-native listing dropped either key whenever LiteLLM could not
resolve a limit, so a client validating against a nullable-but-required schema
saw a malformed entry for any model the cost map does not know.
OpenAI and Azure GPT-5.x answer a chat request whose output budget cannot fit a
single visible token with a 400, while the same models return a length-truncated
200 one or two tokens higher. Agents that probe a model with a hardcoded
max_tokens of 1 read that 400 as "model unavailable".
The four chat request helpers now recognise the provider's own sentence and hand
back the length-truncated response the provider gives at a slightly larger
budget: finish_reason "length", empty content, zero completion tokens. Any other
400 still raises. Streaming is covered by the same seam, and the caller's budget
is never raised on their behalf.
The provider bills the prompt it processed but sends no usage object with the
400, so the prompt tokens are estimated with the same token_counter every other
usage-less path uses. Reporting zero would let a caller send an arbitrarily
large prompt with max_tokens 1 and be charged nothing.
cost_per_second treated a declared-but-zero output_cost_per_second as a real
rate, so the output branch claimed the call and the elif locked out
input_cost_per_second. Every transcription model shipping
output_cost_per_second 0.0 next to a real input rate billed $0, which covers
43 of the 55 per-second entries in the cost map: all 36 deepgram models, both
assemblyai, both elevenlabs scribe, both groq whisper and azure-stt. Custom
deployments pairing the two fields the same way billed $0 as well
Take the output branch only when that rate is actually billable, so a zero
falls through to the input rate. Entries that duplicate one rate into both
fields, whisper-1 among them, keep billing exactly what they bill today
* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery
* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils
* fix(proxy): make model_list request param optional for direct callers
* style: apply ruff format to changed lines
* style: satisfy ruff strict-rule budget (UP006, I001)
* style: satisfy type-discipline budget (LIT002 mutable-ok, LIT009 pyright ignore)
* style: satisfy LIT001/LIT010 and drop explanatory comment per contributor rules
* fix(proxy): translate team model names in the Anthropic /v1/models response
* ci: trigger buildkite status report
* feat(proxy): carry token limits into the Anthropic-native /v1/models entries
* fix(proxy): cast the injected request so the anthropic-version guard is a real comparison
* fix(proxy): explain the model listing casts so the type-discipline gate passes
---------
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
chunk_parser built ModelResponseStream without passing usage, so the
cache_read_input_tokens and cache_creation_input_tokens that Databricks
returns for Anthropic models never reached the cost calculator. Every
streamed request was billed at the full input rate even when served
from cache.
ModelResponseStream already coerces a usage dict into Usage, which maps
those keys into prompt_tokens_details, so passing the chunk's usage
through is sufficient.
* feat(azure_ai): add Fireworks FW model pricing on Azure AI Foundry
* fix(azure_ai): drop incorrect FW-Kimi-K2.6-Code alias
* test(azure-ai): assert FW max token metadata
* feat(azure_ai): add Inkling and Nemotron 3 Ultra pricing
Python keeps only the last binding for a name, so when a file defines the same
test twice the earlier one is unreachable. pytest cannot collect a function that
no longer exists, so nothing reports it and the file still looks like it covers
the scenario.
These ten are cases where the two definitions have different bodies, meaning a
real test was replaced rather than duplicated. Each is renamed to say what it
actually covers, which makes it reachable again:
- test_gemini_frequency_penalty: the dead copy checks the parameter is listed in
get_supported_openai_params for vertex_ai; the survivor checks get_optional_params
maps a value for gemini. Different function and different provider.
- test_async_log_success_event_adds_to_queue and the failure variant: the dead
copies run without mocking asyncio.create_task, so they exercise the real task
path the survivors mock out.
- test_async_send_batch_triggers_tasks: the dead copy asserts send is not awaited
directly; the survivor asserts create_task was called.
- test_model_id_in_required_metrics: the dead copy checks the model_id label on
twelve further metrics the survivor dropped.
- test_anthropic_messages_pt_file_block_preserves_cache_control: the dead copy
passes model and llm_provider explicitly and uses real base64 PDF content.
- test_translate_streaming_openai_chunk_to_anthropic_with_thinking: the dead copy
covers thinking_delta; the survivor covers signature_delta.
- test_client_initialization and test_client_without_api_key: the dead copies
assert the resource clients are wired with the right base URL and key; the
survivors only construct the object.
- test_client_initialization_strips_trailing_slash: the dead copy constructs
ModelsManagementClient directly rather than going through Client.
Verification: collecting the seven touched files gives 401 node IDs before and
411 after, the ten new names and nothing else, with nothing lost. All ten pass.
Running the touched files in full gives 299 passed, and test_optional_params.py
goes from 111 passed to 112.
Two further shadowed definitions were left alone rather than renamed: the dead
copies of test_prompt_caching and test_cost_calculator_with_base_model_with_router
have no assertions at all, one being a bare pass and the other a lone import, so
restoring them would add tests that cannot fail.
Three groups, all verified by running the suite rather than by inspection.
18 files whose every test function carries an unconditional @pytest.mark.skip,
39 test functions in total. They are collected on every CI run and always skip,
so they advertise coverage the suite does not have. Reasons on the marks include
"AWS Suspended Account", "lakera deprecated their v1 endpoint" and "moved to
using 'otel' for logging"; 26 of the marks predate 2025.
30 test functions with a byte-identical body and identical decorators to a
sibling in the same file and class, differing only in name. Deleting one of each
pair removes no coverage. Four further candidates were excluded because they
override an inherited test, where deleting the override un-shadows the base
class implementation instead of removing a duplicate.
9 test functions that a later definition of the same name shadows, so Python
never binds them and pytest cannot collect them.
One file that is a demo script rather than a test; its own docstring says to run
it with python.
Verification: collecting the 26 edited files gives 2,492 node IDs before and
2,462 after. The 30 duplicate deletions account for exactly 30 removals, the 9
shadowed deletions account for 0 (confirming at runtime that they were never
collectable), nothing unexplained disappeared, and nothing new appeared. No
other test or module imports any deleted symbol.
Drop the transform overrides that swapped response.usage to the chat
shape, which broke the /v1/responses client contract. Provider extras
like server_side_tool_usage_details already survive validation via
ResponseAPIUsage extra fields, so the shared usage bridge now carries
them onto the bridged chat Usage generically. The web_search_call
output gate also reads dict output items, since items that fail SDK
validation stay plain dicts, and the chat path gains billing tests.
