- Add test_memory_baseline_1m for router (1,000,000 requests)
- Add test_proxy_memory_baseline_1m for proxy (1,000,000 requests)
- Add clarifying comment: high-volume tests verify memory limit strictness
- These extreme scale tests help detect very gradual leaks (~1-2 KB per 1000 requests)
- Estimated runtime: router ~15-20 min, proxy ~50-60 min
- For manual execution only, not included in CI
- Run with: pytest tests/load_tests/test_*_memory_growth.py::test_*_1m -v
- Add test_memory_baseline_500k for router (500,000 requests)
- Add test_proxy_memory_baseline_500k for proxy (500,000 requests)
- These high-volume tests are designed for manual execution and deep memory leak analysis
- Not included in CI due to long runtime (~15-30 minutes each)
- Useful for detecting very gradual memory accumulation patterns
- Run individually with: pytest tests/load_tests/test_*_memory_growth.py::test_*_500k -v
- Rename test_memory_baseline_100k to test_memory_baseline_50k (router)
- Rename test_proxy_memory_baseline_250k to test_proxy_memory_baseline_50k (proxy)
- Update CI configuration to use new 50k test names
- Reduces test time while still providing comprehensive scale testing
- Router 50k test: ~90 seconds vs 3-4 minutes for 100k
- Proxy 50k test: ~2.5 minutes vs 16 minutes for 250k
- Add multi-generation garbage collection (3 generations)
- Add 5 full GC passes to catch circular references
- Check for uncollectable objects (memory leak detection)
- Add 0.5s wait for OS memory stabilization
- Add final GC pass after wait
- Add detailed logging of cleanup results
- Match cleanup routine used in proxy tests
- Reports actual memory usage vs test artifacts separately
- Add test_memory_baseline_100k for testing at scale (100,000 requests)
- Increase MEMORY_LIMIT from 50 MB to 60 MB for all router tests
- Memory limit adjustment accounts for observed usage patterns in 30k test
- Increase batch size from 20 to 100 requests (matches proxy test pattern)
- Add memory tracking and detailed logging throughout test execution
- Add periodic memory checks every 10 batches
- Direct task creation without intermediate validation for maximum speed
- Explicit cleanup of responses after each batch
- Average 300-400 req/s (was ~50-100 req/s before)
- Increase memory limit from 40 MB to 50 MB to account for larger batch size
- 1k test now completes in ~6.5 seconds (was ~20+ seconds)
- Rename test_linear_memory_growth.py to test_router_acompletion_memory_growth.py for clarity
- Add new test_proxy_chat_completions_memory_growth.py to test full proxy server memory usage
- Implement dynamic port allocation (18888-18919) in mock_server fixture for parallel test execution
- Add httpx client diagnostics to differentiate test artifacts from actual proxy memory growth
- Increase MEMORY_LIMIT to 170 MB based on empirical testing showing ~174 MB stable usage at scale
- Add aggressive cleanup routines with detailed logging to ensure accurate memory measurements
- Fix Windows Unicode compatibility by removing emoji characters from test output
- Tests now properly detect slow memory accumulation (~170 KB per 1000 requests) in proxy server
* fix: align max_tokens with max_output_tokens for consistency
Fixed inconsistent max_tokens definitions in model_prices_and_context_window.json.
According to LiteLLM convention, max_tokens should equal max_output_tokens when available.
Models fixed:
- deepseek-chat: 131072 → 8192 (now equals max_output_tokens)
- dashscope/qwen-flash: 1000000 → 32768 (now equals max_output_tokens)
- databricks/databricks-gemma-3-12b: 128000 → 32000 (now equals max_output_tokens)
This ensures consistency across all providers where max_tokens represents
the maximum number of tokens that can be generated in the output.
* fix: align max_tokens with max_output_tokens for 244 models
- Fix 244 models where max_tokens != max_output_tokens
- Add test to validate max_tokens consistency and prevent regressions
According to model_prices_and_context_window.json spec:
- max_tokens is a LEGACY parameter
- Should always equal max_output_tokens when both are present
This ensures consistency across all model definitions.
Fix router embedding methods to properly propagate proxy model
configuration headers to LLM API calls by calling
_update_kwargs_before_fallbacks() just like completion() does.
Previously, router.embedding() and router.aembedding() manually
set num_retries and metadata but didn't call
_update_kwargs_before_fallbacks(), which meant default_litellm_params
(including headers) were not propagated correctly.
Changes:
- Replace manual kwargs setup with _update_kwargs_before_fallbacks()
in _embedding method (litellm/router.py:3318)
- Apply Black formatting to router.py for consistency
- Add comprehensive unit tests for header propagation
- Add integration tests for various router configurations
Tests verify:
- Headers from default_litellm_params are included in embedding calls
- Metadata (model_group) is properly set
- Consistency between completion() and embedding() behavior
- Support for deployment-specific headers, fallbacks, and retries
* fix(azure): add logprobs support for Azure OpenAI GPT-5 models
Azure OpenAI GPT-5 models (including gpt-5.2) support logprobs
parameters, unlike OpenAI's GPT-5 reasoning models. This fix
overrides the parent class restriction to enable logprobs for Azure.
Changes:
- Override get_supported_openai_params() in AzureOpenAIGPT5Config
- Add "logprobs" and "top_logprobs" to supported params
- Add comprehensive tests for logprobs functionality
Testing:
- Verified with direct Azure API calls to gpt-5.2
- API version: 2025-01-01-preview
- Successfully returns logprobs data
Related: #7974, #4022
* refactor: restrict logprobs support to gpt-5.2 only
Only gpt-5.2 has been verified to support logprobs on Azure.
Other gpt-5 variants (gpt-5, gpt-5.1) have not been tested.
Changes:
- Add conditional check for is_model_gpt_5_2_model()
- Update tests to be specific to gpt-5.2
- Add negative tests for gpt-5 and gpt-5.1
- Update documentation to reflect gpt-5.2 specificity
