- Skip short-circuit for providers that have a BaseAnthropicMessagesConfig
(bedrock, vertex_ai, azure_ai, anthropic) — they use the agentic loop
which includes a follow-up LLM synthesis step. Short-circuiting would
return raw search text instead of an LLM-synthesized answer.
- Add fallback to litellm.get_llm_provider() for custom_llm_provider
derivation when litellm_params is overwritten by kwargs.
- Add test for bedrock guard.
Addresses Greptile review comments #3 and #4.
Bedrock does not surcharge for >200k token context windows.
All *_above_200k_tokens prices should equal their base prices.
Affected models (anthropic., global., us., eu., au. regions):
- anthropic.claude-opus-4-6-v1
- anthropic.claude-sonnet-4-6
Tests were outdated after _get_and_validate_existing_key was refactored
to use prisma_client.db.litellm_verificationtoken.find_unique() and
ProxyException. Also add ProxyException handling in bulk_update_keys
error extractor so error messages aren't empty.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tests were outdated after _get_and_validate_existing_key was refactored
to use prisma_client.db.litellm_verificationtoken.find_unique() instead
of prisma_client.get_data(), and to raise ProxyException instead of
HTTPException. Also fix bulk_update_keys error handler to extract
ProxyException.message (str(ProxyException) returns empty string).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace hand-rolled _extract_search_query with existing
get_last_user_message from common_utils
- Use full UUID (str(uuid.uuid4())) to match codebase convention
- Move uuid import to module level per CLAUDE.md
Addresses Greptile review feedback:
- Save original stream flag before pre-request hooks convert it, so
streaming callers get SSE events instead of a plain dict
- Propagate custom_llm_provider derived inside _execute_pre_request_hooks
when it was not explicitly passed by the caller
- Add tests covering both scenarios
For providers like github_copilot that don't natively support web search,
Claude Code's search sub-conversations were falling through to the adapter
path which strips the web_search tool and has no stream reconversion.
Instead of routing search requests through the full LLM pipeline, detect
web-search-only requests early (all tools are web_search, simple prompt)
and execute the search directly via Tavily/Perplexity, returning a
synthetic Anthropic response. No adapter, no backend LLM call needed.
Fixes#21733
* add DD Tracing (#24033)
* feat(models): add Azure GPT-5.4 mini and nano variants (#24045)
Add `azure/gpt-5.4-mini` and `azure/gpt-5.4-nano` to the model
database with official pricing from Azure OpenAI:
- GPT-5.4 mini: $0.75/M input, $0.075/M cached, $4.5/M output
- GPT-5.4 nano: $0.20/M input, $0.02/M cached, $1.25/M output
Both models support:
- 1.05M input / 128K output context window
- Chat, batch, and responses endpoints
- Function calling, tools, vision, reasoning
- Prompt caching with automatic tiered pricing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add new model pricing details for volcengine Doubao-Seed-2.0 series (#23871)
Add entries for volcengine Doubao-Seed-2.0 series
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23701)
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23700)
The .well-known/oauth-authorization-server metadata advertises
refresh_token as a supported grant type, but the token endpoint
rejected it with HTTP 400. This adds refresh_token grant support
so MCP clients can refresh expired tokens without re-authenticating.
* test(mcp): add tests for refresh_token grant type in OAuth token endpoint
* fix(mcp): move code_verifier guard into authorization_code branch
code_verifier is only relevant for authorization_code grants (PKCE).
Move it inside the else branch so it doesn't apply to refresh_token.
* fix(mcp): guard None client_secret and forward scope in token exchange
- Conditionally include client_secret in form data to prevent httpx
from sending the literal string "None" (applies to both
authorization_code and refresh_token branches)
- Forward optional scope parameter per RFC 6749 §6, allowing clients
to request a subset of originally-granted scopes on refresh
* fix(mcp): validate code param in authorization_code grant
Guard against None code being form-encoded as literal string "None"
by httpx, symmetric with the existing refresh_token guard.
* docs: add incident report for guardrail logging secret exposure (#24059)
Add blog post documenting the guardrail logging path exposing internal
request data (e.g. Authorization headers) in spend logs and OTEL traces.
Fix available in LiteLLM 1.82.3+.
Made-with: Cursor
* [Fix] Datadog LLM Observability tags format (env, service, version missing) (#23673)
* tag fix
* greptile comment
* fix(ci): stabilize 6 failing CI jobs
1. mypy: remove duplicate type annotation for token_data in discoverable_endpoints.py
2. integrations tests: add parameterized to CI test deps
3. doc quality: document OTEL_IGNORE_CONTEXT_PROPAGATION env key
4. security: allowlist CVE-2026-2673, CVE-2026-3644, CVE-2026-4224 (no fix available)
5. proxy_store_model_in_db: fix missing x-litellm-call-id header on error responses
6. google tests: add --retries 3 for transient Vertex AI rate limits
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(streaming): handle RuntimeError during model_copy in streaming handler
The race condition occurs when model_copy(deep=True) tries to deepcopy
_hidden_params dict while it's being concurrently modified by logging
callbacks. Fall back to shallow copy if the deep copy fails.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(cost): handle non-string traffic_type in cost calculator + add retries
1. Fix AttributeError in _map_traffic_type_to_service_tier when traffic_type
is an integer (cast to str before calling .upper()). This was causing
pass-through vertex spend logging to fail silently.
2. Add --retries to llm_translation_testing for flaky external API calls.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: ExMatics HydrogenC <33123710+HydrogenC@users.noreply.github.com>
Co-authored-by: Jack Venberg <jack.venberg@rover.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Wrap _run_deferred_stream_guardrails initialization (UnifiedLLMGuardrails
constructor and _check_and_merge_model_level_guardrails) in try/finally
so logging always fires even if init throws. Prevents silent logging loss
on transient errors.
Move fastapi.HTTPException import from module-level to local test-function
scope. Add test_logging_fires_even_if_guardrail_init_raises to verify the
try/finally guard.
Use the merged guardrail_data dict (from _check_and_merge_model_level_guardrails)
for hook invocations in _run_deferred_stream_guardrails, instead of the original
captured_data. This ensures model-level non-default guardrails are visible to
inner should_run_guardrail re-checks inside UnifiedLLMGuardrails.
Rewrite three hand-crafted closure tests to exercise the production
_run_deferred_stream_guardrails exception-handling path. Add three new tests
that use deep-copy mocks to prove hooks receive the merged dict.
guardrail_information is None in StandardLoggingPayload because logging
fires before post-call guardrails write to metadata.
Non-streaming: wrapper_async stores a closure instead of calling
create_task immediately. The proxy fires it in a try/finally after
post_call_success_hook so the SLP is built with guardrail info.
Streaming: a closure on logging_obj is called by CSW.__anext__ at
stream end. The closure runs only guardrail hooks (not all callbacks)
on the assembled response, then fires both logging handlers. This
avoids behavioral changes for non-guardrail callbacks on streaming.
- Change `if team_limit:` to `if team_limit is not None:` in both
get_key_model_rpm_limit and get_key_model_tpm_limit so that an
explicitly-empty team rate-limit map ({}) is returned as-is instead
of silently falling through to deployment defaults (P1 fix).
- Replace the bare `int()` list comprehension in _get_deployment_default_limit
with a loop that catches ValueError/TypeError so malformed config strings
do not raise an unhandled exception during request handling (P2 fix).
- Add corresponding unit tests for both edge cases.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Merge _get_deployment_default_rpm_limit and _get_deployment_default_tpm_limit
into a single _get_deployment_default_limit(model_name, field) helper; the two
thin wrappers are preserved for callers but share one implementation
- Compute _success_tpm_limit / _success_rpm_limit once before the guard condition
in async_log_success_event, eliminating the previous two unconditional
get_key_model_* calls (each of which could hit llm_router.get_model_list)
- Replace fragile llm_model_list=[{}] sentinel in test with []
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Aligns proxy default with litellm.AZURE_DEFAULT_API_VERSION (2025-02-01-preview)
so Azure response_format + json_schema works without tools fallback.
Made-with: Cursor
Enable deployment_affinity, responses_api_deployment_check, and session_affinity to be configured per model group via router_settings.model_group_affinity_config, falling back to global settings for unconfigured groups.
- Add model_group_affinity_config parameter to Router and DeploymentAffinityCheck
- Add _get_effective_flags helper to resolve flags per model group
- Update async_filter_deployments and async_pre_call_deployment_hook to use per-group config
- Add 4 comprehensive tests covering per-group config, fallback, and override scenarios
This allows fine-grained control of affinity behavior across model groups, e.g., enabling stickiness only for cross-provider deployments while leaving other groups free to load-balance.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Previously the test called common_processing_pre_call_logic in isolation,
making generate_polling_id.assert_not_called() vacuously true. Now the test
calls responses_api() end-to-end so it actually verifies that a rate-limited
request never receives a polling ID.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Guard logging_obj for None when skip_pre_call_logic=True: raise ValueError
if litellm_logging_obj not in data, preventing AttributeError downstream
- Add model=None to common_processing_pre_call_logic call in endpoints.py
to match style of other call sites
- Add test verifying rate-limited request never receives polling ID
Move pre-call checks (rate limits, guardrails, budget) to run BEFORE
polling ID creation in the background streaming flow. This prevents the
edge case where a rate-limited request receives a polling ID that
immediately fails.
Changes:
- Add skip_pre_call_logic parameter to base_process_llm_request to allow
skipping pre-call checks (avoiding double-counting of RPM/parallel requests)
- Run common_processing_pre_call_logic before generating polling ID in the
responses API endpoint. If rate limits/guardrails fail, return error
immediately without creating a polling ID
- Background streaming task passes skip_pre_call_logic=True to avoid re-running
pre-call checks that were already done before polling ID creation
- Add tests verifying skip_pre_call_logic parameter works correctly
Fixes the edge case where polling_via_cache would return a polling ID
for a request that immediately fails due to rate limiting.
- Change chunk["id"] to chunk.get("id") for compatibility with MiniMax
- ModelResponseStream auto-generates id when None is passed
- Add regression test test_chunk_parser_without_id_field
Replace bare _get_deployment_default_tpm/rpm_limit calls in the
async_log_success_event condition with get_key_model_tpm/rpm_limit
(model_name=model_group). The higher-level getters short-circuit on
key/team metadata hits before ever reaching the router, so requests
that don't use deployment defaults incur no extra router lookup. Remove
the now-unused bare helper imports.
Also fix invalid `int = None` type hints in test helper signatures
to `Optional[int] = None`.
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Use min() across all matching deployments instead of first-wins when
resolving default_api_key_tpm/rpm_limit for a model group, so
load-balanced setups with different per-deployment limits always apply
the most conservative value
- Replace the global SensitiveDataMasker non_sensitive_overrides change
with a targeted excluded_keys set at the remove_sensitive_info_from_deployment
call site, avoiding unintended suppression of other fields
- Update the v1 parallel request limiter to pass model_name to
get_key_model_tpm/rpm_limit so deployment defaults apply there too
- Add 4 tests covering multi-deployment min semantics
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
Adds `default_api_key_tpm_limit` and `default_api_key_rpm_limit` to
`GenericLiteLLMParams` so operators can set per-deployment rate limit
defaults in config.yaml. When a key has no model-specific tpm/rpm limit
configured, the proxy falls back to these deployment defaults (Case 2 in
spec). Key-level limits always take priority (Case 1).
- Extends `get_key_model_tpm_limit` / `get_key_model_rpm_limit` with a
`model_name` param and a priority-4 deployment-default fallback
- Passes `model_name=requested_model` in the parallel request limiter so
the fallback is triggered at enforcement time
- Adds `"limit"` to `SensitiveDataMasker` non-sensitive overrides so
`*_limit` fields are not masked in `/model/info` responses
- Adds 17 unit tests covering both spec cases and the `/model/info` path
Co-Authored-By: Claude (claude-sonnet-4-6) <noreply@anthropic.com>
- Only remove wildcard path from openai_routes when the route entry has
type="subpath", avoiding accidental removal when two endpoints share
the same base path but differ in include_subpath
- Clean up _registered_pass_through_routes in the test finally block to
prevent stale entries from polluting subsequent tests on failure
