The error message for DISABLE_ADMIN_ENDPOINTS incorrectly said
"DISABLING LLM API ENDPOINTS is an Enterprise feature" instead of
"DISABLING ADMIN ENDPOINTS is an Enterprise feature".
This was a copy-paste bug from the is_llm_api_route_disabled() function.
Added regression tests to verify both error messages are correct.
* feat(generic_api_callback.py): make generic api OSS + support multiple generic API's
Enables https://github.com/BerriAI/litellm/pull/17094#discussion_r2562832967
* feat(callback_utils.py): support custom generic api callbacks
* feat(generic_api_callback.py): support specifying which event types to run the generic api for
* fix(litellm_logging.py): log system prompt for anthropic messages
* feat(generic_api_callback.py): support generic api compatible api's - e.g. rubrik agent cloud
* docs(sidebars.js): document new OSS generic api
* docs(generic_api.md): document new OSS Generic API
* docs(custom_webhook_api.md): document custom webhook api integration tutorial
* docs(custom_webhook_api.md): cleanup
* docs(custom_webhook_api.md): document what get's logged to custom webhook api
* Refactor: Pass callback config to GenericAPILogger
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
* Fix: Handle empty messages list in logging payload
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
* Checkpoint before follow-up message
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
* feat: Cache GenericAPILogger instances to improve performance
Co-authored-by: krrishdholakia <krrishdholakia@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* Add v1 cut of container api
* fix lint errors
* Add proxy support to container apis & logging support (#16049)
* Add proxy support to container apis
* Add logging support
* Add cost tracking support for containers and documentation
* Add new constant documentation
* Add container cost in model map
* fix failing azure tests
* Update tests based on model map changes
* fix model map tests
* fix model map tests
* Container modeshould be container
* Container tests fix
* Merge branch 'main' into litellm_sameer_oct_staging_2
* Add Prometheus metric to track callback logging failures in S3 (#16102)
* Add proxy support to container apis
* Add logging support
* prometheus metric measures how often s3_v2 is failing
* remove not needed files
* remove not needed files
* remove not needed files
* fix mypy errors
* Use logging_callback_manager to get all the callbacks
---------
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* fix(apscheduler): prevent memory leaks from jitter and frequent job intervals
Fixes critical memory leak in APScheduler that causes 35GB+ memory allocations
during proxy startup and operation. The leak was identified through Memray
analysis showing massive allocations in normalize() and _apply_jitter()
functions.
Key changes:
1. Remove jitter parameters from all scheduled jobs - jitter was causing
expensive normalize() calculations leading to memory explosion
2. Configure AsyncIOScheduler with optimized job_defaults:
- misfire_grace_time: 3600s (increased from 120s) to prevent backlog
calculations that trigger memory leaks
- coalesce: true to collapse missed runs
- max_instances: 1 to prevent concurrent job execution
- replace_existing: true to avoid duplicate jobs on restart
3. Increase minimum job intervals:
- PROXY_BATCH_WRITE_AT: 30s (was 10s)
- add_deployment/get_credentials jobs: 30s (was 10s)
4. Use fixed intervals with small random offsets instead of jitter for
job distribution across workers
5. Explicitly configure jobstores and executors to minimize overhead
6. Disable timezone awareness to reduce computation
Memory impact:
- Before: 35GB with 483M allocations during startup
- After: <1GB with normal allocation patterns
Performance notes:
- Minimum job intervals increased from 10s to 30s (configurable via env vars)
- Jobs can still be distributed across workers using random start offsets
- No functional changes to job behavior, only timing and memory optimization
Testing:
- Added comprehensive test suite for scheduler configuration
- Verified no job execution backlog on startup
- Tested duplicate job prevention with replace_existing
Related issue: Memory leak in production proxy servers with APScheduler
\ud83e\udd16 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* docs: update PROXY_BATCH_WRITE_AT default value from 10s to 30s
Update documentation to reflect the new default value for PROXY_BATCH_WRITE_AT
changed in PR #15846. The default was increased from 10 seconds to 30 seconds
to prevent memory leaks in APScheduler.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: Move APScheduler config to constants.py
Address code review feedback from ishaan-jaff:
- Move scheduler configuration variables (coalesce, misfire_grace_time,
max_instances, replace_existing) to litellm/constants.py
- Update all references in proxy_server.py to use the constants
- Improves maintainability and makes configuration values centralized
Requested-by: @ishaan-jaff
Related: #15846🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
* fix(support-model-specific-tpm/rpm-limits): Allows setting rate limits by tpm/rpm for models by team
* fix(key_management_endpoints.py): enforce guaranteed throughput with key-level model tpm/rpm limits, when team-level tpm/rpm limits are set
* test: add unit testing
* feat(schema.prisma): add metadata to litellm budget table
* feat(proxy/utils.py): add org limits to user api key auth
allows org level tpm/rpm limiting to work
* feat: add org level tpm/rpm limits + inherit org id in key from team
enables org level tpm/rpm limits
* feat: validated working org tpm/rpm limits
* feat: support updating org level, model specific tpm/rpm limits
* fix: working key validation for org level tpm/rpm limits
* fix: working validation for orgs when giving tpm/rpm to teams
* fix(key_management_endpoints.py): fix tpm/rpm limits on orgs
* fix(key_management_endpoints.py): support limits
* refactor: remove duplicate var
* fix: refactor to avoid ruff errors
* fix: fix typign
* fix: fix linting error
* fix: fix testing
* fix(key_management_endpoints.py): document params
* fix(managed_files.py): don't raise error if managed object is not found
* feat(vector_stores): add azure ai search vector store support
Enables direct querying a vector store on azure
* fix(azure/vector_stores): working azure ai search api vector stores
allows azure direct querying on vector stores
* test: update env vars
* docs(docs/): document new azure ai vector store search
* docs(azure_ai_vector_stores.md): add table
* docs: clarify support for 'create' vector stores
* fix(vector_stores/endpoints.py): Fixes https://github.com/BerriAI/litellm/issues/14606
* fix: fix linting errors
* Addd v2/chat support for cohere
* fix streaming
* Use v2_transformation for logging passthrough:
* Use v2_transformation for logging passthrough:
* Add test for checking if document and citation_options is getting passed
* Update the cohere model
* Add cost tracking for vertex ai passthrough batch jobs
* Add full passthrough support
* refactor code according to the comments
* Add passthrough handler
* remove invalid params
* Updated documentation
* Updated documentation
* Updated documentation
* Correct the import
* Add openai videos generation and retrieval support
* add retrieval endpoint
* Add docs
* Add imports
* remove orjson
* remove double import
* fix openai videos format
* remove mock code
* remove not required comments
* Add tests
* Add tests
* Add other video endpoints
* Fix cost calculation and transformation
* Fixed mypy tests
* remove not used imports
* fix documentation for get batch req (#15742)
* Add grounding info to responses API (#15737)
* Add grounding info to responses API
* fix lint errors
* Use typed objects for annotations
* Use typed objects for annotations
* fix mypy error
* Litellm fix json serialize alreting 2 (#15741)
* fix json serializable error for alerts
* Add test
* fix mypt errors
* fix mypt errors
* Add Qwen3 imported model support for AWS Bedrock (#15783)
* Add qwen imported model support
* fix mypy errors
* fix empty user message error (#15784)
* fix typed dict for list
* Add azure supported videos endpoint
* fix mapped tests
* add azure sora models to model map
* Add OpenAI video generation and content retrieval support (#15745)
* Add openai videos generation and retrieval support
* add retrieval endpoint
* Add docs
* Add imports
* remove orjson
* remove double import
* fix openai videos format
* remove mock code
* remove not required comments
* Add tests
* Add tests
* Add other video endpoints
* Fix cost calculation and transformation
* Fixed mypy tests
* remove not used imports
* fix typed dict for list
* fix mypy errors
* move directory
* make v2 chat default
* Fix mypy tests
* Fix mypy tests
* Fix mypy tests
* Fix mypy tests
* Revert "Add Azure Video Generation Support with Sora Integration"
* refactor videos repo
* add test
* Add azure openai videos support
* Add azure openai videos support
* Add router endpoint support for videos
* fix mypy error
* add azure models
* fix mapped test
* fix mypy error
* Add proxy router test
* Add proxy router test
* remove deprecated model name from tests
* fix import error
* fix import error
* Add gaurdrail integration in videos endpoint
* Add logging support for videos endpoint
* Add final documentation supporting videos integration
* fix model name and document input
* Update literals to avoid mypy errors
* Remove unused imports and print statements
* revert guardrail support for video generation and video remix
* revert guardrail support for video generation and video remix
* Fix failing mapped and llm translation tests
