* fix(apscheduler): prevent memory leaks from jitter and frequent job intervals
Fixes critical memory leak in APScheduler that causes 35GB+ memory allocations
during proxy startup and operation. The leak was identified through Memray
analysis showing massive allocations in normalize() and _apply_jitter()
functions.
Key changes:
1. Remove jitter parameters from all scheduled jobs - jitter was causing
expensive normalize() calculations leading to memory explosion
2. Configure AsyncIOScheduler with optimized job_defaults:
- misfire_grace_time: 3600s (increased from 120s) to prevent backlog
calculations that trigger memory leaks
- coalesce: true to collapse missed runs
- max_instances: 1 to prevent concurrent job execution
- replace_existing: true to avoid duplicate jobs on restart
3. Increase minimum job intervals:
- PROXY_BATCH_WRITE_AT: 30s (was 10s)
- add_deployment/get_credentials jobs: 30s (was 10s)
4. Use fixed intervals with small random offsets instead of jitter for
job distribution across workers
5. Explicitly configure jobstores and executors to minimize overhead
6. Disable timezone awareness to reduce computation
Memory impact:
- Before: 35GB with 483M allocations during startup
- After: <1GB with normal allocation patterns
Performance notes:
- Minimum job intervals increased from 10s to 30s (configurable via env vars)
- Jobs can still be distributed across workers using random start offsets
- No functional changes to job behavior, only timing and memory optimization
Testing:
- Added comprehensive test suite for scheduler configuration
- Verified no job execution backlog on startup
- Tested duplicate job prevention with replace_existing
Related issue: Memory leak in production proxy servers with APScheduler
\ud83e\udd16 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: update PROXY_BATCH_WRITE_AT default value from 10s to 30s
Update documentation to reflect the new default value for PROXY_BATCH_WRITE_AT
changed in PR #15846. The default was increased from 10 seconds to 30 seconds
to prevent memory leaks in APScheduler.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: Move APScheduler config to constants.py
Address code review feedback from ishaan-jaff:
- Move scheduler configuration variables (coalesce, misfire_grace_time,
max_instances, replace_existing) to litellm/constants.py
- Update all references in proxy_server.py to use the constants
- Improves maintainability and makes configuration values centralized
Requested-by: @ishaan-jaff
Related: #15846🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Fixes#15995
When using the Responses API with providers that don't have a native
ResponsesAPIConfig implementation (like Gemini, Anthropic, Cohere, etc.),
the text.format parameter was being ignored. This happened because these
providers fall back to using the LiteLLMCompletionTransformationHandler,
which converts Responses API requests to Chat Completion API requests.
Changes:
- Added 'text' to the list of supported parameters in
LiteLLMCompletionResponsesConfig.get_supported_openai_params()
- Added transformation logic to convert text.format (Responses API format)
to response_format (Chat Completion API format) in
transform_responses_api_request_to_chat_completion_request()
- Created _transform_text_format_to_response_format() method to handle
the conversion between the two format structures
The transformation supports:
- json_schema: Converts to Chat Completion's json_schema format with
proper nested structure
- json_object: Converts to Chat Completion's json_object mode
- text: Returns None (default text format)
This fix benefits all providers that use the fallback transformation
handler, ensuring consistent behavior with OpenAI's native Responses API
implementation.
* fix: Preserve Bedrock inference profile IDs in health checks
- Fixes issue where health checks were stripping inference profile IDs
- Preserves cross-region inference profile prefixes (us., eu., apac., jp., au., us-gov., global.)
- Strips only AWS region routing while preserving routes and handlers
- Resolves both issue #15807 and inference profile requirement errors
- Adds comprehensive tests for all Bedrock model format combinations
Issue #15807 attempted to fix regional Bedrock model health checks but was too
aggressive, stripping cross-region inference profile prefixes that AWS requires.
This caused errors: "Invocation of model ID X with on-demand throughput isn't
supported. Retry your request with the ID or ARN of an inference profile."
The fix now correctly:
- Strips AWS regions (us-west-2, eu-central-1, etc.) from routing
- Preserves CRIS prefixes (us., eu., etc.) required by AWS
- Preserves routes (converse/, invoke/)
- Preserves handlers (llama/, deepseek_r1/)
- Only affects Bedrock models (checked via startswith)
Test coverage includes 20+ scenarios for all Bedrock model format combinations.
* Remove unused traceback import
* perf: Skip sleep delays in base_mail.py during tests to improve test speed
* perf: Mock datetime.now in parallel_request_limiter_v3.py to improve test speed
* pref: Mock urllib system calls in test_aiohttp_transport.py to improve test speed
* chore: add --durations=50 to visualize slowest tests
* pref: reduce setup phase overhead by widening fixture scope in conftest.py
* test: stabilize flaky tests
* fix: minor issue
