Models using only the legacy max_tokens field now have proper
max_input_tokens and max_output_tokens based on provider documentation.
Providers updated: OpenRouter (13), Gradient AI (13), Heroku (4),
Aleph Alpha (6), Together AI (2).
The observability "View all" page only listed 9 integrations while
27 more had documentation. Added all missing integration cards so
users can discover the full catalog from the integrations landing page.
Two independent bugs prevented post-call OpenAI Moderation guardrail
results from reaching downstream logging callbacks (Langfuse, Datadog).
Bug 1: process_output_response() created a throwaway request_data dict,
so guardrail info written by @log_guardrail_information was discarded.
Fixed by threading the real request_data from the unified guardrail
dispatcher through all 13 BaseTranslation handlers, with litellm_metadata
injection preserved for third-party guardrails (Zscaler, Prompt Security).
Also extended to process_output_streaming_response for consistency.
Bug 2: The @log_guardrail_information decorator collapsed the full
moderation API response (categories, scores, flagged status) to "allow".
Fixed by overriding _process_response/_process_error on
OpenAIModerationGuardrail to stash and log the full response, following
the established Model Armor pattern.
Both bridges (Responses→CC and CC→Responses) independently encoded the
same field mapping knowledge. This extracts 4 shared mappings into a
single module so future changes only need to happen in one place.
Shared mappings:
- status ↔ finish_reason bidirectional dicts and functions
- response_format ↔ text.format paired conversion functions
- provider_specific_fields normalization helper
- usage field name translation (input_tokens ↔ prompt_tokens, etc.)
No behavioral changes — bridge methods now delegate to the shared module.
The `dimensions` parameter was correctly mapped to `outputDimensionality`
in `optional_params` but never placed in the request body. The Vertex AI
predict endpoint expects it under a `parameters` field.
Add `parameters` dict to `VertexMultimodalEmbeddingRequest` TypedDict and
populate it from `optional_params` in `transform_embedding_request`.
Fixes#24392
Gemini API returns a DOCUMENT modality in promptTokensDetails for PDF
inputs, but the token parser only handled TEXT, IMAGE, AUDIO, and VIDEO.
DOCUMENT tokens were silently dropped, causing cost to be undercounted
by up to 99% for PDF-heavy requests.
Map DOCUMENT tokens to text_tokens since Gemini bills documents at the
text token rate. Applied to all four modality parser loops:
promptTokensDetails, cacheTokensDetails, responseTokensDetails, and
candidatesTokensDetails.
Fixes#24375
The vertex_ai cost calculator hardcoded $0.035 and charged for every
call with a PromptTokensDetailsWrapper (not just web search calls).
Delegate to the shared Gemini calculator which reads pricing and
billing unit from model_info, fixing both issues for vertex_ai models.
Replace _is_gemini_3_model() substring check with a
web_search_billing_unit field in model_prices JSON:
- "per_query": each search query billed individually (Gemini 3.x)
- "per_prompt" (default): flat fee per grounded API call (Gemini 2.x)
Add web_search_billing_unit to 23 Gemini 3.x model entries.
Update docs and tests accordingly.
Document how each provider bills for web search, the
search_context_cost_per_query field in model_prices JSON,
how to override pricing via proxy config, and how LiteLLM
extracts web_search_requests from each provider's response.
Add tests for the gpt-5.1/5.2/5.4 reasoning.effort interaction:
- gpt-5.1 with no reasoning allows flexible temperature
- gpt-5.1 with effort='high' drops temperature
- gpt-5.4 with effort='none' allows flexible temperature
- Gemini 2.x charges per grounded prompt (flat $0.035), clamped to 1
regardless of internal query count
- Gemini 3.x charges per search query ($0.014 each)
- Extract web_search_requests from groundingMetadata in non-streaming
responses (parity with streaming path)
- Add search_context_cost_per_query to vertex_ai and base Gemini entries
- Move tests to tests/test_litellm/ (CI directory)
The Gemini web search cost calculator hardcoded $0.035 per request,
which is only correct for Gemini 2.x models. Gemini 3.x models
charge $0.014 per request.
Read from search_context_cost_per_query in model_info (same field
used by Anthropic, OpenAI, and Perplexity) with fallback to the
legacy $0.035 for models not yet updated in the JSON.
Also add search_context_cost_per_query to all 25 Gemini models
that support web search in model_prices_and_context_window.json.
Fixes#24369
The Responses API map_openai_params passed all params through without
applying model-specific validation. GPT-5 models (except gpt-5-chat)
only accept temperature=1 unless reasoning.effort="none" on models
that support it (5.1, 5.2, 5.4).
Reuse the existing OpenAIGPT5Config logic from chat completions to
validate temperature in the Responses API path. With drop_params=True,
unsupported temperature values are silently dropped; without it,
UnsupportedParamsError is raised.
Fixes#16090
The Gemini batch embedding transformation was spreading all
optional_params into the request body via **gemini_params. Params
like max_tokens (injected by add_provider_specific_params_to_optional_params)
would reach the Gemini API and cause a 400 BadRequestError.
Extract _filter_embed_params() that maps dimensions/task_type and
keeps only the fields Gemini embeddings actually accept
(outputDimensionality, taskType, title). Applied to both
transform_openai_input_gemini_content and
transform_openai_input_gemini_embed_content.
This also fixes drop_params: true not preventing the error, since
the param was re-injected after the drop_params check.
Fixes#24293
When the Responses API converts function_call and message output items
into chat completion messages, they can become two consecutive assistant
messages. The Bedrock Converse transformer merges these into one, but
the merge preserves input order — so if function_call came first, the
toolUse block ends up before the text block.
Claude models (Sonnet 4, Haiku 3.5+) reject this ordering with:
"tool_use ids were found without tool_result blocks immediately after"
Add _sort_bedrock_assistant_content_blocks() that reorders content
blocks within assistant messages: reasoningContent → text → toolUse.
Applied in both sync and async Bedrock Converse transformation paths.
Fixes#24361
When Gemini sends tool call arguments in the same streaming chunk as a
content block transition, the Anthropic adapter discarded the
processed_chunk containing the input_json_delta. This caused tool_use
blocks to arrive with empty input: {}.
Queue the processed_chunk alongside the block transition events when it
contains input_json_delta data. Applied to both sync and async paths.
Fixes#24134
When Azure sends stream_options.include_usage=True, it emits an initial
chunk with choices=[] (prompt_filter_results) before the first content
chunk. Previously, LiteLLM inflated this empty-choices chunk with a
default StreamingChoices, which consumed the sent_first_chunk flag and
caused strip_role_from_delta to strip role from the real first chunk.
Additionally, the first real chunk with role='assistant' and content=''
was discarded by is_chunk_non_empty as "empty".
This fix:
- Forwards chunks with choices=[] faithfully (no inflated default)
- Only marks sent_first_chunk for chunks with real choices
- Treats chunks with role in delta as non-empty
- Guards choices[0] access in __next__/__anext__ and stream_chunk_builder
Fixes#24221
