- Static Methods for Streaming Handler Function
- Remove the afile_content_streaming wrapper function. Enabled with a stream boolean in afile_content
- Cleaned up test cases after refactor
- Removed unused imports and streamlined type hints in `litellm/utils.py` and `litellm/files/main.py`.
- Moved `FileContentStreamingResult` to a new `litellm/files/types.py` for better organization.
- Updated `FileContentStreamingResponse` in `litellm/files/streaming.py` to include asynchronous close methods and improved logging capabilities.
- Enhanced tests to ensure proper closure of streaming iterators in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Introduced `afile_content_streaming` and `file_content_streaming` functions in `litellm/files/main.py` to handle asynchronous and synchronous file content streaming.
- Added `FileContentStreamingResponse` class in `litellm/files/streaming.py` to manage streaming responses with logging capabilities.
- Updated OpenAI API integration in `litellm/llms/openai/openai.py` to support new streaming methods.
- Enhanced file content retrieval in `litellm/proxy/openai_files_endpoints/files_endpoints.py` to route requests for streaming.
- Added unit tests for the new streaming functionality in `tests/test_litellm/llms/openai/test_openai_file_content_streaming.py` and `tests/test_litellm/proxy/openai_files_endpoint/test_files_endpoint.py`.
- Refactored type hints and imports for better clarity and organization across modified files.
- Change chunk["id"] to chunk.get("id") for compatibility with MiniMax
- ModelResponseStream auto-generates id when None is passed
- Add regression test test_chunk_parser_without_id_field
Add _is_reasoning_effort_level_explicitly_disabled to use opt-out semantics
for minimal effort: unknown/unlisted models pass through, only blocked when
the model map explicitly sets supports_minimal_reasoning_effort=false.
xhigh keeps opt-in semantics (must be explicitly supported).
Adds test for unknown-model passthrough and explicit-disabled detection.
Made-with: Cursor
OpenAI rejects any reasoning_effort (even 'none') with tools in
/v1/chat/completions for gpt-5.4. Update the guard to drop reasoning_effort
regardless of value. Add docs explaining the auto-drop behavior.
- Add _get_effort_level() to extract effective effort from string or dict
- Use effective_effort for xhigh validation, tool-drop, sampling, temperature guards
- Preserve dict format when it has summary/generate_summary for Responses API
- Add tests: xhigh-dict validation, none-dict for tools/sampling/temperature
- Update tests: dict-with-summary now preserved (not normalized)
Made-with: Cursor
When reasoning_effort is passed as a dict with additional fields like 'summary' or 'generate_summary', preserve the full dict format instead of normalizing it to a string. This ensures that when requests are routed to the OpenAI Responses API, all reasoning parameters are correctly included.
The normalization to string format now only happens for simple dicts with just the 'effort' key, which is appropriate for the Chat Completions API.
Fixes issue where summary field was being dropped when routing gpt-5.4+ requests with tools + reasoning to Responses API.
Made-with: Cursor
Function calls not supported with reasoning_effort != 'none' on gpt-5.4.
Drop reasoning_effort when tools are in the request (small minority of volume).
Made-with: Cursor
The OpenAI chat completion API expects reasoning_effort as a string
('none', 'low', 'medium', 'high', 'xhigh'). Config/deployments may pass
the Responses API format: {'effort': 'high', 'summary': 'detailed'}.
Fix BadRequestError when model config uses dict format by extracting
the 'effort' value before passing to the API.
Made-with: Cursor
- Shift from hardcoded model checks to dynamic lookup via _supports_factory
- Add supports_none_reasoning_effort for gpt-5.1/5.2/5.4 chat variants
- Add supports_xhigh_reasoning_effort for gpt-5.1-codex-max, gpt-5.2, gpt-5.4+
- Update model_prices_and_context_window.json and backup
- Add ProviderSpecificModelInfo types for new fields
- Fix Azure: use _supports_reasoning_effort_level instead of removed is_model_gpt_5_1_model
Made-with: Cursor
* Add Prometheus child_exit cleanup for gunicorn workers
When a gunicorn worker exits (e.g. from max_requests recycling), its
per-process prometheus .db files remain on disk. For gauges using
livesum/liveall mode, this means the dead worker's last-known values
persist as if the process were still alive. Wire gunicorn's child_exit
hook to call mark_process_dead() so live-tracking gauges accurately
reflect only running workers.
* docs: update AssemblyAI docs with Universal-3 Pro, Speech Understanding, and LLM Gateway (#21130)
* docs: update AssemblyAI docs with Universal-3 Pro, Speech Understanding, and LLM Gateway provider config
* feat: add AssemblyAI LLM Gateway as OpenAI-compatible provider
* fix(mcp): update test mocks to use renamed filter_server_ids_by_ip_with_info
Tests were mocking the old method name `filter_server_ids_by_ip` but production
code at server.py:774 calls `filter_server_ids_by_ip_with_info` which returns
a (server_ids, blocked_count) tuple. The unmocked method on AsyncMock returned
a coroutine, causing "cannot unpack non-iterable coroutine object" errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): update realtime guardrail test assertions for voice violation behavior
Tests were asserting no response.create/conversation.item.create sent to
backend when guardrail blocks, but the implementation intentionally sends
these to have the LLM voice the guardrail violation message to the user.
Updated assertions to verify the correct guardrail flow:
- response.cancel is sent to stop any in-progress response
- conversation.item.create with violation message is injected
- response.create is sent to voice the violation
- original blocked content is NOT forwarded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bedrock): restore parallel_tool_calls mapping in map_openai_params
The revert in 8565c70e53 removed the parallel_tool_calls handling from
map_openai_params, and the subsequent fix d0445e1e33 only re-added the
transform_request consumption but forgot to re-add the map_openai_params
producer that sets _parallel_tool_use_config. This meant parallel_tool_calls
was silently ignored for all Bedrock models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): update Azure pass-through test to mock litellm.completion
Commit 99c62ca40e removed "azure" from _RESPONSES_API_PROVIDERS,
routing Azure models through litellm.completion instead of
litellm.responses. The test was not updated to match, causing it
to assert against the wrong mock.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add in_flight_requests metric to /health/backlog + prometheus (#22319)
* feat: add in_flight_requests metric to /health/backlog + prometheus
* refactor: clean class with static methods, add tests, fix sentinel pattern
* docs: add in_flight_requests to prometheus metrics and latency troubleshooting
* fix(db): add missing migration for LiteLLM_ClaudeCodePluginTable
PR #22271 added the LiteLLM_ClaudeCodePluginTable model to
schema.prisma but did not include a corresponding migration file,
causing test_aaaasschema_migration_check to fail.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update stale docstring to match guardrail voicing behavior
Addresses Greptile review feedback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Feat] Agent RBAC Permission Fix - Ensure Internal Users cannot create agents (#22329)
* fix: enforce RBAC on agent endpoints — block non-admin create/update/delete
- Add /v1/agents/{agent_id} to agent_routes so internal users can
access GET-by-ID (previously returned 403 due to missing route pattern)
- Add _check_agent_management_permission() guard to POST, PUT, PATCH,
DELETE agent endpoints — only PROXY_ADMIN may mutate agents
- Add user_api_key_dict param to delete_agent so the role check works
- Add comprehensive unit tests for RBAC enforcement across all roles
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: mock prisma_client in internal user get-agent-by-id test
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): hide agent create/delete controls for non-admin users
Match MCP servers pattern: wrap '+ Add New Agent' button in
isAdmin conditional so internal users see a read-only agents view.
Delete buttons in card and table were already gated.
Update empty-state copy for non-admin users.
Add 7 Vitest tests covering role-based visibility.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: exclude gpt-5.2-chat from temperature passthrough (#21911)
gpt-5.2-chat and gpt-5.2-chat-latest only support temperature=1
(like base gpt-5), not arbitrary values (like gpt-5.2).
Update is_model_gpt_5_1_model() to exclude gpt-5.2-chat variants
so drop_params correctly drops unsupported temperature values.
Fixes#21911
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Ryan Crabbe <rcrabbe@berkeley.edu>
Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com>
Co-authored-by: Dylan Duan <dylan.duan@assemblyai.com>
Co-authored-by: Julio Quinteros Pro <jquinter@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
LiteLLM was adding a `duration` field to audio transcription responses
for internal cost tracking. The OpenAI Python SDK uses "best match
deserialization" to determine the response type from present fields —
seeing `duration` caused it to incorrectly match plain Transcription
responses as TranscriptionVerbose/TranscriptionDiarized types.
Move the internally-calculated duration to `_hidden_params` so it
remains available for cost calculation without polluting the response
body. Provider-returned duration (e.g. from verbose_json format) is
still preserved in the response as expected.
These params were silently dropped for Chat Completions because they
were missing from the supported params whitelist. Also adds
prompt_cache_retention to the Responses API TypedDict and fixes
misleading cache_control comments in OpenAI prompt caching docs.
Verify that extra_headers are correctly forwarded to OpenAI's
images.generate() in both sync and async paths, and that they
are absent when not provided.
logprobs, top_p, top_logprobs are only accepted by OpenAI when
reasoning_effort="none". Add validation matching the existing
temperature logic: raise UnsupportedParamsError or drop when
reasoning_effort is set to other values.
Remove logit_bias, modalities, prediction, audio, web_search_options
from supported params for all GPT-5 reasoning models (OpenAI rejects
them). Add logprobs, top_p, top_logprobs for gpt-5.1/5.2 which support
them when reasoning_effort="none".
Related to #21572
