- Add litellm/proxy/auto_benchmark.py: start_auto_benchmark_from_app(host, port)
runs from proxy lifespan so server binds and prints port before benchmark.
Single-process file lock ensures only one worker starts the benchmark.
Subprocess spawned with start_new_session (Unix) / DETACHED_PROCESS (Win).
- Call start_auto_benchmark_from_app from proxy_server lifespan (before yield).
- Remove benchmark start from proxy_cli before uvicorn.run(); benchmark now
runs only from app startup so main process does not block on init.
Use subprocess.Popen() instead of subprocess.run() so the benchmark starts
and the auto-benchmark thread returns immediately. Benchmark runs in a
separate process; output still goes to the configured .txt files.
- scripts/run_benchmark_rps_suite.py: run benchmark at 1k, 2k, ..., 10k RPS sequentially
- Each run writes to benchmark_results_<rps>rps.txt (e.g. benchmark_results_1000rps.txt)
- First run waits for proxy; runs 2–10 use --no-wait. --output-dir, --requests supported
- proxy_cli: LITELLM_AUTO_BENCHMARK_RPS_SUITE=1 runs suite when proxy is up
- Optional: LITELLM_AUTO_BENCHMARK_OUTPUT_DIR, LITELLM_AUTO_BENCHMARK_REQUESTS
Mock chat API (off by default):
- LITELLM_MOCK_CHAT_API=1 to start mock on port (from LITELLM_MOCK_CHAT_ENDPOINT)
- LITELLM_MOCK_CHAT_ENDPOINT: base URL or port (default http://localhost:8090)
- Mock started once in main process before workers; log_level=error, _log() when MOCK_CHAT_VERBOSE
Auto-benchmark when proxy is up:
- LITELLM_AUTO_BENCHMARK=1 runs benchmark after proxy health, saves to .txt
- Auto-starts mock when auto-benchmark enabled so benchmark has provider to hit
- Health check uses 127.0.0.1 on Windows; LITELLM_AUTO_BENCHMARK_OUTPUT, LITELLM_AUTO_BENCHMARK_ARGS
chat_completions_api.py: mock OpenAI-compatible chat completions for testing
- Wait for proxy health (LITELLM_PROXY_URL base + /health) then run benchmark
- Save stdout to --output file (default benchmark_results_YYYYMMDD_HHMMSS.txt)
- Env: LITELLM_PROXY_URL, PROVIDER_URL; pass-through args (e.g. --rps-control, --requests)
- --no-wait to skip health wait
- Add ProxyInitializationHelpers._start_mock_chat_api_once() in proxy_cli
- Start mock (chat_completions_api.app) in a daemon thread on 8090 before forking workers
- Remove mock startup from proxy_server lifespan so workers never start it
- Single mock instance regardless of --num_workers; no port conflict
- Add --rps-control N to cap requests started per second (no concurrency cap)
- Add make_request_with_rps_limit() and _rps_refill_task() for rate limiting
- benchmark_endpoint() accepts rps_control; when set, ignores max_concurrent
- --requests still controls total requests; works with --rps-control
- Document RPS example in docstring and --help epilog
* Add request tags documentation for spend tracking
- Add new concise doc explaining how to tag model requests
- Include Python SDK and cURL examples
- Show where tags appear in spend logs
- Add common use cases table (AWS accounts, teams, projects)
- Include how to set default tags on API keys
- Add to Spend Tracking section in sidebar
Co-authored-by: ishaan <ishaan@berri.ai>
* Simplify request tags doc for AI Gateway usage
- Focus on config.yaml setup with default_key_generate_params
- Show both request body and header methods for sending tags
- Remove SDK examples, keep concise cURL examples
- Streamline for quick reference
Co-authored-by: ishaan <ishaan@berri.ai>
* Update request tags doc to show model-level config
- Set tags directly on model deployments in litellm_params
- Requests just specify model, tags applied automatically
- Use clear naming: AWS_IAM_PROD, AWS_IAM_DEV
Co-authored-by: ishaan <ishaan@berri.ai>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: ishaan <ishaan@berri.ai>
* docs: add FAQ for setting up and verifying LITELLM_LICENSE
Added two new FAQ entries to the Enterprise docs page:
- How to set up your Enterprise License (LITELLM_LICENSE) via .env, Docker, or docker-compose
- How to verify the license is active by checking for 'Enterprise Edition' in the Swagger UI
* docs: trim license FAQ to essential steps only
* perf: cache _get_relevant_args_to_use_for_logging() as module-level frozenset
The set of valid LLM API parameter names for logging was being rebuilt
on every request from 8 OpenAI SDK type annotations + set operations.
Since these are static TypedDict annotations that never change at
runtime, compute once at import time and store as a class-level
frozenset.
Line profiler: get_standard_logging_model_parameters() dropped from
774ms to 77ms across 12K calls (90% reduction, ~25µs/req saved).
* test: add tests for cached ModelParamHelper logging args
Verify cached frozenset matches dynamic computation and that
prompt content keys (messages, prompt, input) are excluded from
logged model parameters.
- Cache CallTypes enum values as module-level dict to avoid repeated list
comprehension and enum construction on every call
- Hoist update_response_metadata getattr lookup to top of function
- Guard verbose print_verbose call behind _is_debugging_on() check
Pass-through endpoints (like vLLM classify) were not setting
standard_logging_object because _get_assembled_streaming_response
returns None for non-ModelResponse results.
This caused model_max_budget_limiter.async_log_success_event to raise
ValueError('standard_logging_payload is required').
The fix adds an elif branch in async_success_handler that mirrors the
non-pass-through code path.
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Add x-api-key header to CountTokens handler to match chat completion
authentication. Azure AI Anthropic requires this header per Microsoft's
native API format.
Bedrock rejects requests when toolResult or toolUse blocks within a
single message contain duplicate IDs. The Converse message transformer
merges consecutive tool/assistant messages without checking for
duplicate toolUseId values, causing BedrockException errors.
Add _deduplicate_bedrock_content_blocks() — a generalized helper that
removes duplicate blocks by ID, logs a warning for each dropped
duplicate via verbose_logger, and preserves non-tool blocks (e.g.
cachePoint). Apply it at all four merge sites (sync/async × toolResult/
toolUse).
The Anthropic /messages path was fixed in PR #19324; this applies the
equivalent fix to the Bedrock Converse path.
Fixes#20048
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Fix Nova grounding web_search_options={} not applying systemTool
Two bugs prevented web_search_options={} from working for Nova grounding:
1. Empty dict falsy check: The condition `value and isinstance(value, dict)`
short-circuits to False when value is {} (empty dict is falsy in Python).
Changed to `isinstance(value, dict)` to match Anthropic's implementation.
2. Pre-formatted tools mangled by _bedrock_tools_pt: The systemTool
(already in Bedrock format) was added to optional_params["tools"], but
_process_tools_and_beta passed all tools through _bedrock_tools_pt which
expects OpenAI-format tools. This corrupted the systemTool into an empty
toolSpec. Fixed by separating systemTool blocks before transformation
and appending them after.
Fixes follow-up to #19598
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* Fix python-multipart Python version constraint for Poetry lock
python-multipart ^0.0.22 requires Python >=3.10 but the project supports
>=3.9. Add python = ">=3.10" marker so Poetry can resolve dependencies
for Python 3.9.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Previously, when Model Armor guardrail blocked a request/response,
the `applied_guardrails` field was not populated in the logs because
`add_guardrail_to_applied_guardrails_header()` was called after the
HTTPException was raised.
This fix moves the `add_guardrail_to_applied_guardrails_header()` call
to before the blocking check in all hooks:
- async_pre_call_hook (pre_call mode)
- async_moderation_hook (during_call mode)
- async_post_call_success_hook (post_call mode)
- async_post_call_streaming_iterator_hook (streaming)
This ensures that even when a guardrail blocks content, the guardrail
name is properly recorded in the logs for observability.
Added regression tests to verify applied_guardrails is populated when
content is blocked.
Co-authored-by: Cursor <cursoragent@cursor.com>
When using LiteLLM's Anthropic /v1/messages endpoint to route requests to
OpenAI models, requests fail if any tool name exceeds OpenAI's 64-character
limit. Anthropic API has no such limit, causing compatibility issues.
Changes:
- Add truncate_tool_name() function using {55-char-prefix}_{8-char-hash} format
- Modify translate_anthropic_tools_to_openai() to truncate and return mapping
- Modify translate_anthropic_tool_choice_to_openai() to truncate tool name
- Restore original tool names in responses using the mapping
- Support tool name restoration in streaming responses
- Add backwards-compatible API (existing methods still work)
The fix only applies when routing Anthropic requests to OpenAI models.
Native Anthropic/Claude requests pass through unchanged.
* fix: models loadbalancing billing issue by filter (#18891)
* fix: models loadbalancing billing issue by filter
* fix: separate key and team access groups in metadata
* fix: lint issues
Fixes#19788
- Add `supported_regions: ["global"]` to Qwen MaaS models in model_prices_and_context_window.json
- Update `get_supported_regions()` to read directly from `model_cost` dict
- Update `get_complete_vertex_url()` to use `get_vertex_region()` for global-only models
- Update `create_vertex_url()` to generate correct URL for global location (without region prefix)
- Add tests for Qwen global endpoint support
Per review feedback, thought_signature should not be a root-level
param on ImageObject as it's not OpenAI compatible. Moved to
provider_specific_fields dict to match the pattern used in chat
completions (Message, Delta, Choices, etc).
Fixes#17184 - Gemini 3 Pro image preview model returns a thoughtSignature
field required for interactive image editing. This change:
- Adds thought_signature field to ImageObject class
- Updates Gemini and Vertex AI transformations to extract thoughtSignature
- Adds test for thought_signature in response transformation
