Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
- Add test_memory_baseline_1m for router (1,000,000 requests)
- Add test_proxy_memory_baseline_1m for proxy (1,000,000 requests)
- Add clarifying comment: high-volume tests verify memory limit strictness
- These extreme scale tests help detect very gradual leaks (~1-2 KB per 1000 requests)
- Estimated runtime: router ~15-20 min, proxy ~50-60 min
- For manual execution only, not included in CI
- Run with: pytest tests/load_tests/test_*_memory_growth.py::test_*_1m -v
- Add test_memory_baseline_500k for router (500,000 requests)
- Add test_proxy_memory_baseline_500k for proxy (500,000 requests)
- These high-volume tests are designed for manual execution and deep memory leak analysis
- Not included in CI due to long runtime (~15-30 minutes each)
- Useful for detecting very gradual memory accumulation patterns
- Run individually with: pytest tests/load_tests/test_*_memory_growth.py::test_*_500k -v
- Rename test_memory_baseline_100k to test_memory_baseline_50k (router)
- Rename test_proxy_memory_baseline_250k to test_proxy_memory_baseline_50k (proxy)
- Update CI configuration to use new 50k test names
- Reduces test time while still providing comprehensive scale testing
- Router 50k test: ~90 seconds vs 3-4 minutes for 100k
- Proxy 50k test: ~2.5 minutes vs 16 minutes for 250k
- Add test_memory_baseline_100k to router memory tests in CI
- Test runs after 30k test with same 30m timeout
- Provides comprehensive scale testing up to 100k requests
- Add multi-generation garbage collection (3 generations)
- Add 5 full GC passes to catch circular references
- Check for uncollectable objects (memory leak detection)
- Add 0.5s wait for OS memory stabilization
- Add final GC pass after wait
- Add detailed logging of cleanup results
- Match cleanup routine used in proxy tests
- Reports actual memory usage vs test artifacts separately
- Add test_memory_baseline_100k for testing at scale (100,000 requests)
- Increase MEMORY_LIMIT from 50 MB to 60 MB for all router tests
- Memory limit adjustment accounts for observed usage patterns in 30k test
- Increase batch size from 20 to 100 requests (matches proxy test pattern)
- Add memory tracking and detailed logging throughout test execution
- Add periodic memory checks every 10 batches
- Direct task creation without intermediate validation for maximum speed
- Explicit cleanup of responses after each batch
- Average 300-400 req/s (was ~50-100 req/s before)
- Increase memory limit from 40 MB to 50 MB to account for larger batch size
- 1k test now completes in ~6.5 seconds (was ~20+ seconds)
- Add test_memory_baseline_30k to Router tests
- Add test_proxy_memory_baseline_30k and test_proxy_memory_baseline_250k to Proxy tests
- Increase proxy tests timeout from 30m to 45m to accommodate 250k test (~16 min runtime)
- Update memory_leak_tests job to use test_router_acompletion_memory_growth.py (renamed)
- Add new step to run proxy server memory growth tests
- Test both Router (1k, 2k, 4k, 10k) and Proxy (1k, 10k, 20k) in CI
- Reduce timeout from 60m to 30m per section for faster feedback
- Remove 30k router test from CI to keep runtime reasonable
- Rename test_linear_memory_growth.py to test_router_acompletion_memory_growth.py for clarity
- Add new test_proxy_chat_completions_memory_growth.py to test full proxy server memory usage
- Implement dynamic port allocation (18888-18919) in mock_server fixture for parallel test execution
- Add httpx client diagnostics to differentiate test artifacts from actual proxy memory growth
- Increase MEMORY_LIMIT to 170 MB based on empirical testing showing ~174 MB stable usage at scale
- Add aggressive cleanup routines with detailed logging to ensure accurate memory measurements
- Fix Windows Unicode compatibility by removing emoji characters from test output
- Tests now properly detect slow memory accumulation (~170 KB per 1000 requests) in proxy server
This fixes a breaking change where batch completion requests with comma-separated
model strings (e.g., 'gpt-3.5-turbo,fake-openai-endpoint') were failing validation.
The validate_model_access function now:
- Detects comma-separated model strings
- Validates each model individually
- Provides clear error messages for inaccessible models in batch requests
- Maintains backward compatibility for single model validation
Fixes test_batch_chat_completions test failure.
