* Add Prometheus child_exit cleanup for gunicorn workers
When a gunicorn worker exits (e.g. from max_requests recycling), its
per-process prometheus .db files remain on disk. For gauges using
livesum/liveall mode, this means the dead worker's last-known values
persist as if the process were still alive. Wire gunicorn's child_exit
hook to call mark_process_dead() so live-tracking gauges accurately
reflect only running workers.
* docs: update AssemblyAI docs with Universal-3 Pro, Speech Understanding, and LLM Gateway (#21130)
* docs: update AssemblyAI docs with Universal-3 Pro, Speech Understanding, and LLM Gateway provider config
* feat: add AssemblyAI LLM Gateway as OpenAI-compatible provider
* fix(mcp): update test mocks to use renamed filter_server_ids_by_ip_with_info
Tests were mocking the old method name `filter_server_ids_by_ip` but production
code at server.py:774 calls `filter_server_ids_by_ip_with_info` which returns
a (server_ids, blocked_count) tuple. The unmocked method on AsyncMock returned
a coroutine, causing "cannot unpack non-iterable coroutine object" errors.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): update realtime guardrail test assertions for voice violation behavior
Tests were asserting no response.create/conversation.item.create sent to
backend when guardrail blocks, but the implementation intentionally sends
these to have the LLM voice the guardrail violation message to the user.
Updated assertions to verify the correct guardrail flow:
- response.cancel is sent to stop any in-progress response
- conversation.item.create with violation message is injected
- response.create is sent to voice the violation
- original blocked content is NOT forwarded
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(bedrock): restore parallel_tool_calls mapping in map_openai_params
The revert in 8565c70e53 removed the parallel_tool_calls handling from
map_openai_params, and the subsequent fix d0445e1e33 only re-added the
transform_request consumption but forgot to re-add the map_openai_params
producer that sets _parallel_tool_use_config. This meant parallel_tool_calls
was silently ignored for all Bedrock models.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(test): update Azure pass-through test to mock litellm.completion
Commit 99c62ca40e removed "azure" from _RESPONSES_API_PROVIDERS,
routing Azure models through litellm.completion instead of
litellm.responses. The test was not updated to match, causing it
to assert against the wrong mock.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add in_flight_requests metric to /health/backlog + prometheus (#22319)
* feat: add in_flight_requests metric to /health/backlog + prometheus
* refactor: clean class with static methods, add tests, fix sentinel pattern
* docs: add in_flight_requests to prometheus metrics and latency troubleshooting
* fix(db): add missing migration for LiteLLM_ClaudeCodePluginTable
PR #22271 added the LiteLLM_ClaudeCodePluginTable model to
schema.prisma but did not include a corresponding migration file,
causing test_aaaasschema_migration_check to fail.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update stale docstring to match guardrail voicing behavior
Addresses Greptile review feedback.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* [Feat] Agent RBAC Permission Fix - Ensure Internal Users cannot create agents (#22329)
* fix: enforce RBAC on agent endpoints — block non-admin create/update/delete
- Add /v1/agents/{agent_id} to agent_routes so internal users can
access GET-by-ID (previously returned 403 due to missing route pattern)
- Add _check_agent_management_permission() guard to POST, PUT, PATCH,
DELETE agent endpoints — only PROXY_ADMIN may mutate agents
- Add user_api_key_dict param to delete_agent so the role check works
- Add comprehensive unit tests for RBAC enforcement across all roles
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: mock prisma_client in internal user get-agent-by-id test
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* feat(ui): hide agent create/delete controls for non-admin users
Match MCP servers pattern: wrap '+ Add New Agent' button in
isAdmin conditional so internal users see a read-only agents view.
Delete buttons in card and table were already gated.
Update empty-state copy for non-admin users.
Add 7 Vitest tests covering role-based visibility.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix: exclude gpt-5.2-chat from temperature passthrough (#21911)
gpt-5.2-chat and gpt-5.2-chat-latest only support temperature=1
(like base gpt-5), not arbitrary values (like gpt-5.2).
Update is_model_gpt_5_1_model() to exclude gpt-5.2-chat variants
so drop_params correctly drops unsupported temperature values.
Fixes#21911
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Ryan Crabbe <rcrabbe@berkeley.edu>
Co-authored-by: ryan-crabbe <128659760+ryan-crabbe@users.noreply.github.com>
Co-authored-by: Dylan Duan <dylan.duan@assemblyai.com>
Co-authored-by: Julio Quinteros Pro <jquinter@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Register HostedVLLMResponsesAPIConfig so that litellm.responses(model="hosted_vllm/...")
routes directly to vLLM's /v1/responses endpoint instead of falling back to the
chat completions → responses conversion pipeline.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
when a key/team/end-user has mcp_tool_permissions for a server but that
server is not in mcp_servers, the server was excluded from the allowed
list — making the tool permissions useless.
now we union the keys from mcp_tool_permissions into the allowed server
set alongside direct servers and access group servers.
fixes#21954
When the Prisma query engine process is alive but not accepting
connections (e.g., startup race condition in containerized
deployments), lightweight reconnects (disconnect + connect) will
never succeed. The health watchdog retries indefinitely without
escalating to a full Prisma client recreation.
Adds a consecutive failure counter that triggers a heavy reconnect
(full Prisma client and engine recreation) after 3 consecutive
lightweight reconnect failures (configurable via
PRISMA_RECONNECT_ESCALATION_THRESHOLD env var).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The upsert update branches for model_cost_map_reload_config were
overwriting param_value with only the force_reload flag, dropping
interval_hours. This caused scheduled reloads to self-destruct
after their first execution.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
_route_streaming_logging_to_handler is called via asyncio.create_task()
after streaming chunks are already delivered to the client. Any
unhandled exception in this logging task propagates as an unhandled
asyncio task exception, polluting error logs.
Wraps the method body in try/except to log errors without propagating.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The OpenAI SDK raw_response.parse() can return a plain str instead
of a Pydantic model when Azure returns a non-JSON content type (e.g.,
HTML error page, proxy error). Calling .model_dump() on the str then
raises AttributeError.
Adds isinstance(response, str) checks before all 4 model_dump() call
sites in the Azure chat completion and embedding paths.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Spend logs are written asynchronously in batches, so a DB lookup for a
just-completed response's spend log races and returns zero rows. Replace the
DB-only fallback with an in-memory session store (_session_history) that is
populated after each response.completed event and consulted at the start of
the next response.create. This makes same-connection multi-turn reliable
without any timing dependency on the DB write queue.
Made-with: Cursor
Also suppress pre-existing pyright type error on model_response.close() call
which is guarded by hasattr at runtime.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Also fix pyrightconfig.json to use the conda venv for type checking, and remove
redundant inline import of ResponsesAPIRequestUtils that was confusing pyright.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The response.output_item.done handler for function_call type was emitting
finish_reason='tool_calls' and a duplicate tool_call delta. This caused
premature stream termination after the first tool call in multi-tool
scenarios — downstream wrappers (e.g. AnthropicStreamWrapper) would close
the stream before subsequent tool calls arrived.
The response.completed event already inspects the response output list and
emits finish_reason='tool_calls' when function_call items are present, so
output_item.done does not need to (and must not) do so.
This mirrors the existing fix for message-type output_item.done (#17246).
Updated test_function_call_done_emits_is_finished (renamed) to assert
finish_reason=None and no duplicate delta. Updated test_text_plus_tool_calls_sequence
to match. Added test_multi_tool_call_stream_no_premature_finish which exercises
a synthetic 2-tool-call stream and verifies no premature termination.
Renamed test_build_custom_pricing_entry_kwargs_take_precedence_over_model_info
to test_build_custom_pricing_entry_setdefault_does_not_override_existing.
The original test claimed to verify kwargs precedence over model_info but
had no overlapping keys between the two sources. CustomPricingLiteLLMParams
fields and the model_info metadata keys (mode, supports_prompt_caching,
max_tokens) do not currently overlap. Updated the test to verify that
model_info fields are correctly merged, and added an explicit setdefault
assertion demonstrating that pre-existing keys would not be overwritten.
Co-authored-by: openhands <openhands@all-hands.dev>
Previously, register_model() was called with only input_cost_per_token,
output_cost_per_token, and litellm_provider. This dropped ~40+ other
pricing fields from CustomPricingLiteLLMParams (cache_read_input_token_cost,
cache_creation_input_token_cost, output_cost_per_reasoning_token, etc.)
as well as model_info metadata (mode, supports_prompt_caching, max_tokens).
For DB-sourced custom-priced models, the first request after a pod restart
would register a partial entry in litellm.model_cost, causing cost
calculations to miss cache token discounts and other extended pricing
until the entry was later enriched by deployment_callback_on_success
mutating the lru_cache.
Changes:
- Add _build_custom_pricing_entry() helper that iterates over all
CustomPricingLiteLLMParams.model_fields and merges model_info metadata
- Replace hardcoded 3-field dicts in both completion() and embedding()
with the new helper
- Add 7 tests covering field collection, model_info merging, precedence,
None skipping, and end-to-end register_model behavior
Co-authored-by: openhands <openhands@all-hands.dev>
- Parse VIDEO modality in promptTokensDetails → prompt_tokens_details.video_tokens
- Parse VIDEO modality in candidatesTokensDetails → completion_tokens_details.video_tokens
- Parse VIDEO modality in cacheTokensDetails and subtract from prompt video tokens
- Add video_tokens field to PromptTokensDetailsWrapper and CompletionTokensDetailsWrapper
- Fix implicit caching text-token fallback to not fire when cacheTokensDetails is present
- Add 4 unit tests covering: prompt video tokens, response video tokens,
auto-calculated text fallback with video, and explicit video cache subtraction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers:
- Region + modelId correctly extracted for ap-northeast-1, us-east-1, us-west-2
- No region in path leaves modelId and optional_params unchanged
- Cross-region inference prefixes (us., eu., ap.) are not treated as region segments
- Explicitly set aws_region_name is not overridden by region in model path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a user passes model="bedrock/ap-northeast-1/moonshotai.kimi-k2.5", get_llm_provider
strips the "bedrock/" prefix and passes "ap-northeast-1/moonshotai.kimi-k2.5" to the
converse handler. Two bugs occurred:
1. modelId was encoded as "ap-northeast-1%2Fmoonshotai.kimi-k2.5" (region included),
which AWS rejects as "not a valid model identifier"
2. The region ap-northeast-1 was never extracted, so the request went to the wrong
default region instead
Fix: after stripping routing prefixes in converse_handler.py completion(), check if the
remaining path starts with a known AWS region and strip it from modelId, injecting it
into optional_params so _get_aws_region_name picks it up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract the repeated usage-combining block from both
_completion_streaming_iterator and _acompletion_streaming_iterator into a
shared static helper method _combine_fallback_usage. This brings both
functions under the PLR0915 50-statement limit, removing the noqa
suppressions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add `# noqa: PLR0915` suppression to match the async twin
`_acompletion_streaming_iterator` which already carries the same
suppression. The function's complexity is inherent (nested class,
generator with fallback logic, cleanup code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The observatory test workflow failed because the "Verify tunnel
connectivity" step used a single curl with no retries. Cloudflare quick
tunnels need time for DNS propagation, and the first lookup can return
NXDOMAIN (curl exit 6). Replace with a retry loop (10 attempts, 5s
apart) matching the pattern already used in the health check step.
Also add `# noqa: PLR0915` to `_completion_streaming_iterator` in
router.py, matching the suppression already on its async twin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
actions/checkout treats short commit hashes as branch names, causing
fetch failures. The checkout only needs the config file from the
repo, so use the default branch instead of a specific ref.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The secrets context is not available in step-level if: conditions,
causing the workflow file to fail validation. Move the conditional
check into the shell script instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass AZURE_API_KEY, AZURE_API_BASE, OBSERVATORY_URL,
OBSERVATORY_API_KEY, and REQUEST_ID through step-level env
blocks so they are never interpolated directly into shell scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Validate inputs.tag matches vX.Y.Z format to prevent script
injection via workflow_dispatch
- Pass tag via env var instead of direct interpolation in shell
- Add cleanup step to kill cloudflared and remove docker container
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add timeout-minutes: 30 to prevent runaway jobs
- Build /run-test payload with jq --arg to safely escape
TUNNEL_URL and LITELLM_MASTER_KEY values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fail early if request_id is missing or null from the /run-test
response instead of polling /run-status/null for 15 minutes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoids shell quoting issues with single quotes in JSON and
multi-line output truncation when using GITHUB_OUTPUT.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add permissions block (contents: read) per GitHub security scan
- Poll /run-status/{request_id} instead of global /queue-status
to avoid race conditions with concurrent test runs
- Add result verification step that fails the workflow if tests
did not pass or the run errored
- Fix auth header to use X-LiteLLM-Observatory-API-Key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New reusable workflow that spins up a LiteLLM container from the
release image, exposes it via cloudflared tunnel, and triggers
test runs on the Railway-hosted observatory
- Integrates into ghcr_deploy.yml for RC and stable releases
- Can also be triggered manually via workflow_dispatch
- Add placeholder litellm_config.yaml for observatory test models
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The transformation.py file was using FEATHERLESS_API_KEY (missing _AI_)
while the rest of the codebase (get_llm_provider_logic.py, utils.py)
correctly uses FEATHERLESS_AI_API_KEY. This caused 401 auth errors when
the user set FEATHERLESS_AI_API_KEY as documented.
Now checks FEATHERLESS_AI_API_KEY first (canonical name) with fallback
to FEATHERLESS_API_KEY (legacy compatibility). Same fix applied to
FEATHERLESS_AI_API_BASE.
Refs: #22490
