* (sap) ensure tool parameters have type='object' for SAP compatibility
Fix SAP GenAI Hub Orchestration Service rejecting tool calls with error:
"400 - LLM Module: tools.0.custom.input_schema.type: Input should be 'object'"
Root cause: When Claude Code uses tools (like web_search) with the SAP provider
through LiteLLM's Anthropic experimental pass-through adapter, Anthropic's
input_schema format doesn't always include the required type="object" field.
The adapter's translate_anthropic_tools_to_openai() function was directly
copying input_schema to OpenAI's parameters field without ensuring the
type="object" requirement that SAP's API strictly enforces.
Changes:
- Modified translate_anthropic_tools_to_openai() to check if input_schema
is missing the type field and add type="object" if absent
- Preserves existing type field if already present
- Added comprehensive test suite (6 tests) covering:
- Missing type field scenario (now adds type="object")
- Existing type preservation
- Empty input_schema handling
- Multiple tools transformation
- Additional schema properties preservation
- SAP-specific compatibility regression test
Testing:
- All new tests pass (6/6 in test_anthropic_tool_schema_fix.py)
- All existing Anthropic tool tests pass (57/57 tool-related tests)
- SAP tool parameter validation tests pass (9/9 in test_sap_tool_parameters.py)
* (sap) enable native response_format for anthropic models
* (sap) filter strict param from model_params for GPT models only
* (sap) revert Anthropic adapter type='object' fix
The SAP FunctionTool Pydantic validator in litellm/llms/sap/chat/models.py
already ensures type='object' is added to all tool parameters for SAP
API compatibility.
The Anthropic adapter change affected ALL consumers, not just SAP, which
was broader scope than intended for this PR.
- Revert input_schema modification in Anthropic adapter
- Remove Anthropic-specific test file (SAP tests still cover this case)
* (sap) gate markdown stripping to Anthropic models only
SAP GenAI Hub with Anthropic models sometimes returns JSON wrapped in
markdown code blocks. GPT/Gemini/Mistral models don't exhibit this
behavior, so stripping is now gated to avoid accidentally modifying
valid responses that may contain markdown in JSON string values.
The user-specified async client was being overwritten by
`litellm.module_level_aclient` in `streaming_handler.py` when using
async+streaming with Gemini.
This fix adds a `gemini_client` parameter to `make_call()` (matching
the existing pattern in `make_sync_call()`) so the user's custom client
is preserved and not overwritten.
Fixes#17148
- Log provider token counting failures instead of silently swallowing
- Fall back to local tokenizer when provider returns error response
- Map assistant tool_calls to Responses API function_call items
- Concatenate multiple system messages instead of overwriting
- Hide internal error details from proxy API responses
- Narrow exception catch in handler to network/JSON errors only
- Update test to match new fallback behavior
- Add OpenAITokenCounter using POST /v1/responses/input_tokens endpoint
- Add litellm.acount_tokens() public async API that auto-routes to provider APIs
- Add proxy endpoint POST /v1/responses/input_tokens for OpenAI-compatible counting
- Transform chat tools format to Responses API format for correct token counting
- Fall back to local tiktoken when provider API unavailable
Fixes#22302
Kontext models (flux-kontext-pro, flux-kontext-max) support both
text-to-image and image editing. Add them to IMAGE_GENERATION_MODELS
and update supported_endpoints in model prices JSON.
- Create handler.py for image generation and image edit
- Move polling logic from transformation to handlers
- Handlers use _get_httpx_client() / get_async_httpx_client()
- Transformation files now only transform request/response data
- Follows Bedrock pattern for provider-specific handlers
Addresses feedback: transformation files should not make HTTP requests
Add native text-to-image generation for Black Forest Labs Flux models
(flux-pro-1.1, flux-pro-1.1-ultra, flux-dev, flux-pro).
- Polling-based async API with sync and async support
- OpenAI-compatible parameter mapping (size, n, quality)
- Reuses shared HTTP clients via _get_httpx_client()
- 39 unit tests added
Replace direct httpx.get() calls with _get_httpx_client() to reuse
cached HTTP client, following the pattern used by other providers
(RunwayML, Azure AI OCR, Sagemaker, etc.).
Add native integration for Black Forest Labs image editing models
(flux-kontext-pro, flux-kontext-max, flux-pro-1.0-fill, flux-pro-1.0-expand).
Changes:
- Add BlackForestLabsImageEditConfig for BFL API transformation
- Add BLACK_FOREST_LABS to LlmProviders enum
- Add use_multipart_form_data() to BaseImageEditConfig for JSON vs form-data
- Modify image_edit_handler to support JSON request bodies
- Add comprehensive unit tests
Closes#11401
Providers like Cerebras return delta.reasoning in streaming responses
for gpt-oss models, but LiteLLM's Delta class expects reasoning_content.
This causes reasoning content to be silently dropped during streaming.
Fixes#13300
Add global media_resolution support for Gemini 2.x models (2.0, 2.5) when
using OpenAI's detail parameter on images. Previously, the detail parameter
was only working for Gemini 3+ models (per-part) and was silently ignored
for older Gemini models.
- Add _get_highest_media_resolution() and _extract_max_media_resolution_from_messages()
to extract highest detail from all images/files in a request
- Update _transform_request_body() to add mediaResolution to generationConfig
for Gemini 2.x models only (not 1.x which doesn't support it, not 3+ which
uses per-part)
- Add mediaResolution field to GenerationConfig TypedDict
- Support detail extraction from both image_url and file content types
- Add comprehensive unit tests and update documentation
Add MistralAudioTranscriptionConfig for Mistral's /v1/audio/transcriptions
endpoint, enabling litellm.transcription() with mistral/voxtral-mini-latest
and other Voxtral models. Supports multipart form-data with OpenAI-compatible
params (language, temperature, response_format, timestamp_granularities)
plus Mistral-specific params like diarize.
* azure content enhancement...
* rafactored to increase confidence score
* improvements based on additional feedback
* removed unused import
* Force-split any word longer than max length allowed
* preserve whitespace in text splitting
* moving common initialization to base class
* consolidate enforcement into async_make_request as single point, remove redundant caller-side checks, extract shared init/HTTP logic into base, and fix stale log messages
* clean up
* clean up tests
Add cache_read_input_token_cost_per_audio_token, supports_code_execution,
and supports_file_search to the JSON schema used by the model prices
validation test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The _encrypt_response_id method now receives request_cache=None as a
keyword argument from async_post_call_success_hook. Updated the mock
assertion to expect this parameter.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests used call_policy throughout but the actual API model uses
input_policy and output_policy. Updated _make_tool_row helper,
list filter query param, and policy update request/response assertions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The aresponses_websocket CallType was recently added but not included
in the test exclusion list. It uses WebSocket passthrough (not Azure SDK
client initialization), so it correctly doesn't call
initialize_azure_sdk_client.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix translate_thinking_to_reasoning in responses_adapters/transformation.py
to make summary opt-in (was hardcoded to "detailed")
- Update e2e test to mock litellm.responses (new OpenAI routing path)
- Add tests for Responses API adapter summary preservation
- Resolve merge conflict in test file
Remove hardcoded summary="detailed" injection — summary is opt-in per
OpenAI spec and increases costs. Users opt-in per-request via LiteLLM
extension: thinking={"type": "enabled", "budget_tokens": N, "summary": "concise"}.
Also preserve summary in translate_thinking_for_model() which previously
dropped it when converting thinking → reasoning_effort for non-Claude models.
Fixes#20998
Add test_parallel_tool_calls_comprehensive_streaming_integration which
synthesizes the full 10-event Responses API SSE sequence with split
argument deltas and asserts all fix invariants together:
1. output_item.done emits no finish_reason (no premature stream end)
2. Each call_id appears exactly once (no duplicate tool_call chunks)
3. Split argument deltas assemble to correct final JSON
4. Exactly one finish event, at the terminal response.completed chunk
5. Parallel tool calls have distinct indices (output_index 0 and 1)
All 24 unit tests pass.
* add explicit caching to litellm proxy for gemini models via injection
* fix: add missing `supports_function_calling` for deepinfra models
All 55 deepinfra models that had `supports_tool_choice: true` were
missing the `supports_function_calling` flag, causing
`litellm.supports_function_calling()` to incorrectly return False.
Fixes#22619
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Managed batches - Address PR bot comments from #22464
* feat(togetherai): add support for TogetherAI Qwen3.5-397B-A17B model
* Agent Tracing - support context_id based trace id propogation + nested llm calls (#22626)
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [Feat] UI - Add Open in New Tab on leftnav Bar (#22731)
* Add minimal dev_config.yaml for proxy development
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): wrap left nav items in <a> tags for open-in-new-tab support
Nav items are now rendered as <a> elements with proper href attributes,
enabling right-click → 'Open in new tab', Ctrl/Cmd+click, and
middle-click to open any sidebar page in a new browser tab.
Normal clicks continue to use SPA navigation (no full page reload).
Applied to both leftnav.tsx (query-param routing) and Sidebar2.tsx
(Next.js file-based routing).
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* [Feat] Add Tool Policies for AI Gateway (#22732)
* fix: fix ui render
* fix: fix minor bugs
* refactor: use prisma functions instead of raw sql (safer)
* fix(add-new-tiles-to-tool-policies): allow developer to see what's available
* feat: ensure tool allowlist runs correctly for tool names + mcp's
* refactor: more ui improvements
* feat: working key tool blocking
* feat(tools): show tool logs
* refactor: backend code improvements
* refactor: improve log viewer for tools
* fix: address PR review feedback for tool access control
- Add missing blocked_tools column to root schema.prisma (schema drift)
- Invalidate ToolPolicyRegistry after policy mutations so changes take effect immediately
- Remove dead code: unused get_effective_policies, get_tool_policies_cached, and helpers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: race condition in permission resolution and remove duplicate allowlist check
- Use atomic update_many with object_permission_id=None to prevent concurrent
requests from creating orphaned permission rows and losing tool blocks
- Remove duplicate allowed_tools enforcement from guardrail (already enforced
in auth layer via check_tools_allowlist)
- Move inline uuid import to module level
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* update to account for userAgent
* UI - Add ToolDetails
* input/output policy
* LiteLLM_PolicyAttachmentTable
* LiteLLM_PolicyAttachmentTable
* fix: add _enqueue_tool_registry_upsert
* fix: tool mgmt endpoints
* tool mgmt endpoints
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update tests/test_litellm/proxy/db/test_tool_registry_writer.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix: sync root schema.prisma and fix test_tool_registry_writer for input/output policy
- Migrate root schema.prisma LiteLLM_ToolTable from call_policy to
input_policy/output_policy, add missing user_agent and last_used_at columns
(now consistent with litellm/proxy/schema.prisma and litellm-proxy-extras)
- Fix SpendLogToolIndex comment across all three schema files
- Fix all call_policy references in test_tool_registry_writer.py:
swapped update_tool_policy arguments, wrong get_tools_by_names return type
assertions, _mock_tool_row setting call_policy instead of input_policy
Addresses Greptile review feedback on PR #22732.
Made-with: Cursor
---------
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(proxy): add key_alias, key_hash, requested_model DD APM span tags (#22710)
* feat(proxy): add key_alias, key_hash, requested_model tags to DD APM spans
* refactor(proxy): consolidate DD APM tag helpers into DDSpanTagger class
* refactor(proxy): move DDSpanTagger to its own file litellm/proxy/dd_span_tagger.py
---------
Co-authored-by: liweiguang <codingpunk@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ephrim Stanley <ephrim.stanley@point72.com>
Co-authored-by: Varad Khonde <varadkhonde@gmail.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Sameer Kankute <sameer@berri.ai>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* feat(proxy): add key_alias, key_hash, requested_model tags to DD APM spans
* refactor(proxy): consolidate DD APM tag helpers into DDSpanTagger class
* refactor(proxy): move DDSpanTagger to its own file litellm/proxy/dd_span_tagger.py
* style(ui/): distinguish agent calls from llm calls on ui
* feat: initial grouping working
* feat: set stable contextid for a2a calls - allows for easily passing to downstream llm/mcp calls
* feat(a2a_endpoints.py): fix tracing to avoid recreating logging objects for the same call
allows stable trace id usage
* fix(guardrail_endpoints): handle string ui_type values in _build_field_dict
_build_field_dict unconditionally called .value on ui_type, which crashes
for guardrail configs that use plain strings (e.g. BlockCodeExecutionGuardrailConfigModel
uses "multiselect" and "percentage"). Now checks with hasattr before calling .value.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: propagate trace/session id from headers in MCP server calls
Cherry-picked mcp_server/server.py fixes from 6feb9bab: adds
get_chain_id_from_headers to extract x-litellm-trace-id /
x-litellm-session-id from raw headers, and uses it in call_tool
and list_tools to keep spend logs and tracing consistent with A2A.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Change status codes from 400 to 500 for team metadata misconfig errors
(callers can't fix admin-set config, 400 is misleading)
- Add anchor value validation to batch endpoint (matching files endpoint)
- Coerce seconds to int to handle string values from metadata
- Add error-path tests: missing keys, invalid anchor, status code assertions
- Add happy-path test: team injects expiry when caller sends nothing
The route-level auth check was blocking internal_user role (team admins)
from reaching /key/{key}/reset_spend because KEY_RESET_SPEND was missing
from key_management_routes. Added it so team admins pass the route check
and the endpoint's existing _check_proxy_or_team_admin_for_key enforces
actual authorization.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The _resolve_model_for_cost_lookup function was only checking
litellm_params.model when resolving model names from the router.
For Azure custom deployment names (e.g. azure/openai/gpt-5.3-codex),
this deployment name doesn't exist in the model cost map, so cost
returned /bin/zsh.
Now checks model_info.base_model and litellm_params.base_model first,
falling back to litellm_params.model only if no base_model is set.
This matches how the router resolves base_model everywhere else.
