The /v1/messages/count_tokens endpoint was hardcoding the Bedrock runtime
URL, ignoring api_base and aws_bedrock_runtime_endpoint settings. This
aligns it with invoke/converse handlers by using the existing
get_runtime_endpoint() method for consistent endpoint resolution.
Signed-off-by: stias <seokjun.yang@mycraft.kr>
Adds a control plane capability that enables a central admin instance
to manage multiple regional worker proxies from a single UI.
Backend:
- Worker registry loaded from YAML config (worker_id, name, url)
- /.well-known/litellm-ui-config exposes is_control_plane and workers list
- /v3/login + /v3/login/exchange: opaque code exchange for cross-origin
username/password auth (JWT never in URL/logs, single-use 60s TTL)
- SSO cookie handoff with return_to → opaque code → exchange
- _validate_return_to: full origin validation (scheme+hostname+port)
- Startup warning when control_plane_url set without Redis
- Both /v3 endpoints gated behind control_plane_url config
Frontend:
- Worker selector dropdown on login page (gated behind is_control_plane)
- Cross-origin SSO code exchange handling on callback
- switchToWorkerUrl: localStorage-persisted worker URL for API calls
- useWorker hook: shared worker state management
- WorkerDropdown in navbar for switching workers
- Logout/switch clears worker state from localStorage
Tests:
- 7 tests for /v3/login + /v3/login/exchange
- 10 tests for _validate_return_to
- 2 tests for control plane discovery endpoint
- Skip short-circuit for providers that have a BaseAnthropicMessagesConfig
(bedrock, vertex_ai, azure_ai, anthropic) — they use the agentic loop
which includes a follow-up LLM synthesis step. Short-circuiting would
return raw search text instead of an LLM-synthesized answer.
- Add fallback to litellm.get_llm_provider() for custom_llm_provider
derivation when litellm_params is overwritten by kwargs.
- Add test for bedrock guard.
Addresses Greptile review comments #3 and #4.
Bedrock does not surcharge for >200k token context windows.
All *_above_200k_tokens prices should equal their base prices.
Affected models (anthropic., global., us., eu., au. regions):
- anthropic.claude-opus-4-6-v1
- anthropic.claude-sonnet-4-6
Tests were outdated after _get_and_validate_existing_key was refactored
to use prisma_client.db.litellm_verificationtoken.find_unique() and
ProxyException. Also add ProxyException handling in bulk_update_keys
error extractor so error messages aren't empty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests were outdated after _get_and_validate_existing_key was refactored
to use prisma_client.db.litellm_verificationtoken.find_unique() instead
of prisma_client.get_data(), and to raise ProxyException instead of
HTTPException. Also fix bulk_update_keys error handler to extract
ProxyException.message (str(ProxyException) returns empty string).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace hand-rolled _extract_search_query with existing
get_last_user_message from common_utils
- Use full UUID (str(uuid.uuid4())) to match codebase convention
- Move uuid import to module level per CLAUDE.md
Addresses Greptile review feedback:
- Save original stream flag before pre-request hooks convert it, so
streaming callers get SSE events instead of a plain dict
- Propagate custom_llm_provider derived inside _execute_pre_request_hooks
when it was not explicitly passed by the caller
- Add tests covering both scenarios
For providers like github_copilot that don't natively support web search,
Claude Code's search sub-conversations were falling through to the adapter
path which strips the web_search tool and has no stream reconversion.
Instead of routing search requests through the full LLM pipeline, detect
web-search-only requests early (all tools are web_search, simple prompt)
and execute the search directly via Tavily/Perplexity, returning a
synthetic Anthropic response. No adapter, no backend LLM call needed.
Fixes#21733
* add DD Tracing (#24033)
* feat(models): add Azure GPT-5.4 mini and nano variants (#24045)
Add `azure/gpt-5.4-mini` and `azure/gpt-5.4-nano` to the model
database with official pricing from Azure OpenAI:
- GPT-5.4 mini: $0.75/M input, $0.075/M cached, $4.5/M output
- GPT-5.4 nano: $0.20/M input, $0.02/M cached, $1.25/M output
Both models support:
- 1.05M input / 128K output context window
- Chat, batch, and responses endpoints
- Function calling, tools, vision, reasoning
- Prompt caching with automatic tiered pricing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add new model pricing details for volcengine Doubao-Seed-2.0 series (#23871)
Add entries for volcengine Doubao-Seed-2.0 series
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23701)
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23700)
The .well-known/oauth-authorization-server metadata advertises
refresh_token as a supported grant type, but the token endpoint
rejected it with HTTP 400. This adds refresh_token grant support
so MCP clients can refresh expired tokens without re-authenticating.
* test(mcp): add tests for refresh_token grant type in OAuth token endpoint
* fix(mcp): move code_verifier guard into authorization_code branch
code_verifier is only relevant for authorization_code grants (PKCE).
Move it inside the else branch so it doesn't apply to refresh_token.
* fix(mcp): guard None client_secret and forward scope in token exchange
- Conditionally include client_secret in form data to prevent httpx
from sending the literal string "None" (applies to both
authorization_code and refresh_token branches)
- Forward optional scope parameter per RFC 6749 §6, allowing clients
to request a subset of originally-granted scopes on refresh
* fix(mcp): validate code param in authorization_code grant
Guard against None code being form-encoded as literal string "None"
by httpx, symmetric with the existing refresh_token guard.
* docs: add incident report for guardrail logging secret exposure (#24059)
Add blog post documenting the guardrail logging path exposing internal
request data (e.g. Authorization headers) in spend logs and OTEL traces.
Fix available in LiteLLM 1.82.3+.
Made-with: Cursor
* [Fix] Datadog LLM Observability tags format (env, service, version missing) (#23673)
* tag fix
* greptile comment
* fix(ci): stabilize 6 failing CI jobs
1. mypy: remove duplicate type annotation for token_data in discoverable_endpoints.py
2. integrations tests: add parameterized to CI test deps
3. doc quality: document OTEL_IGNORE_CONTEXT_PROPAGATION env key
4. security: allowlist CVE-2026-2673, CVE-2026-3644, CVE-2026-4224 (no fix available)
5. proxy_store_model_in_db: fix missing x-litellm-call-id header on error responses
6. google tests: add --retries 3 for transient Vertex AI rate limits
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(streaming): handle RuntimeError during model_copy in streaming handler
The race condition occurs when model_copy(deep=True) tries to deepcopy
_hidden_params dict while it's being concurrently modified by logging
callbacks. Fall back to shallow copy if the deep copy fails.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(cost): handle non-string traffic_type in cost calculator + add retries
1. Fix AttributeError in _map_traffic_type_to_service_tier when traffic_type
is an integer (cast to str before calling .upper()). This was causing
pass-through vertex spend logging to fail silently.
2. Add --retries to llm_translation_testing for flaky external API calls.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: ExMatics HydrogenC <33123710+HydrogenC@users.noreply.github.com>
Co-authored-by: Jack Venberg <jack.venberg@rover.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Wrap _run_deferred_stream_guardrails initialization (UnifiedLLMGuardrails
constructor and _check_and_merge_model_level_guardrails) in try/finally
so logging always fires even if init throws. Prevents silent logging loss
on transient errors.
Move fastapi.HTTPException import from module-level to local test-function
scope. Add test_logging_fires_even_if_guardrail_init_raises to verify the
try/finally guard.
Use the merged guardrail_data dict (from _check_and_merge_model_level_guardrails)
for hook invocations in _run_deferred_stream_guardrails, instead of the original
captured_data. This ensures model-level non-default guardrails are visible to
inner should_run_guardrail re-checks inside UnifiedLLMGuardrails.
Rewrite three hand-crafted closure tests to exercise the production
_run_deferred_stream_guardrails exception-handling path. Add three new tests
that use deep-copy mocks to prove hooks receive the merged dict.
guardrail_information is None in StandardLoggingPayload because logging
fires before post-call guardrails write to metadata.
Non-streaming: wrapper_async stores a closure instead of calling
create_task immediately. The proxy fires it in a try/finally after
post_call_success_hook so the SLP is built with guardrail info.
Streaming: a closure on logging_obj is called by CSW.__anext__ at
stream end. The closure runs only guardrail hooks (not all callbacks)
on the assembled response, then fires both logging handlers. This
avoids behavioral changes for non-guardrail callbacks on streaming.
- Change `if team_limit:` to `if team_limit is not None:` in both
get_key_model_rpm_limit and get_key_model_tpm_limit so that an
explicitly-empty team rate-limit map ({}) is returned as-is instead
of silently falling through to deployment defaults (P1 fix).
- Replace the bare `int()` list comprehension in _get_deployment_default_limit
with a loop that catches ValueError/TypeError so malformed config strings
do not raise an unhandled exception during request handling (P2 fix).
- Add corresponding unit tests for both edge cases.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Merge _get_deployment_default_rpm_limit and _get_deployment_default_tpm_limit
into a single _get_deployment_default_limit(model_name, field) helper; the two
thin wrappers are preserved for callers but share one implementation
- Compute _success_tpm_limit / _success_rpm_limit once before the guard condition
in async_log_success_event, eliminating the previous two unconditional
get_key_model_* calls (each of which could hit llm_router.get_model_list)
- Replace fragile llm_model_list=[{}] sentinel in test with []
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Aligns proxy default with litellm.AZURE_DEFAULT_API_VERSION (2025-02-01-preview)
so Azure response_format + json_schema works without tools fallback.
Made-with: Cursor
Enable deployment_affinity, responses_api_deployment_check, and session_affinity to be configured per model group via router_settings.model_group_affinity_config, falling back to global settings for unconfigured groups.
- Add model_group_affinity_config parameter to Router and DeploymentAffinityCheck
- Add _get_effective_flags helper to resolve flags per model group
- Update async_filter_deployments and async_pre_call_deployment_hook to use per-group config
- Add 4 comprehensive tests covering per-group config, fallback, and override scenarios
This allows fine-grained control of affinity behavior across model groups, e.g., enabling stickiness only for cross-provider deployments while leaving other groups free to load-balance.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously the test called common_processing_pre_call_logic in isolation,
making generate_polling_id.assert_not_called() vacuously true. Now the test
calls responses_api() end-to-end so it actually verifies that a rate-limited
request never receives a polling ID.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Guard logging_obj for None when skip_pre_call_logic=True: raise ValueError
if litellm_logging_obj not in data, preventing AttributeError downstream
- Add model=None to common_processing_pre_call_logic call in endpoints.py
to match style of other call sites
- Add test verifying rate-limited request never receives polling ID
