* fix(unified_guardrail.py): correctly map a v1/messages call to the anthropic unified guardrail
* fix: add more rigorous call type checks
* fix(anthropic_endpoints/endpoints.py): initialize logging object at the beginning of endpoint
ensures call id + trace id are emitted to guardrail api
* feat(anthropic/chat/guardrail_translation): support streaming guardrails
sample on every 5 chunks
* fix(openai/chat/guardrail_translation): support openai streaming guardrails
* fix: initial commit fixing output guardrails for responses api
* feat(openai/responses/guardrail_translation): handler.py - fix output checks on responses api
* fix(openai/responses/guardrail_translation/handler.py): ensure responses api guardrails work on streaming
* test: update tests
* test: update tests
* fix: support multiple kinds of input to the guardrail api
* feat(guardrail_translation/handler.py): support extracting tool calls from openai chat completions for guardrail api's
* feat(generic_guardrail_api.py): support extracting + returning modified tool calls on generic_guardrails_api
allows guardrail api to analyze tool call being sent to provider - to run any analysis on it
* fix(guardrails.py): support anthropic /v1/messages tool calls
* feat(responses_api/): extract tool calls for guardrail processing
* docs(generic_guardrail_api.md): document tools param support
* docs: generic_guardrail_api.md
improve documentation
* fix(unified_guardrail.py): correctly map a v1/messages call to the anthropic unified guardrail
* fix: add more rigorous call type checks
* fix(anthropic_endpoints/endpoints.py): initialize logging object at the beginning of endpoint
ensures call id + trace id are emitted to guardrail api
* feat(anthropic/chat/guardrail_translation): support streaming guardrails
sample on every 5 chunks
* fix(openai/chat/guardrail_translation): support openai streaming guardrails
* fix: initial commit fixing output guardrails for responses api
* feat(openai/responses/guardrail_translation): handler.py - fix output checks on responses api
* fix(openai/responses/guardrail_translation/handler.py): ensure responses api guardrails work on streaming
* test: update tests
* test: update tests
* test: update tests
* fix(bedrock_guardrails.py): fix post call streaming iterator logic
* fix: fix return
* fix(bedrock_guardrails.py): fix
* fix: lazy load utils.py imports
Lazy-load most functions and response types from utils.py to avoid loading
tiktoken and other heavy dependencies at import time. This significantly
reduces memory usage when importing completion from litellm.
* fix: prevent memory leak in aiohttp connection pooling
Add connection limits to aiohttp TCPConnector to prevent unbounded
connection growth that causes memory leaks. Without these limits,
aiohttp's _wrap_create_connection can accumulate connections
indefinitely in long-running processes.
Changes:
- Set default limit of 300 total connections and 50 per host
- Apply limits to shared proxy session initialization
- Apply limits to HTTP handler transport creation
- Configurable via AIOHTTP_CONNECTOR_LIMIT and
AIOHTTP_CONNECTOR_LIMIT_PER_HOST environment variables
- Set to 0 for unlimited (not recommended for production)
This fix covers:
- All standard LLM provider API calls (OpenAI, Anthropic, etc.)
- Proxy server shared session
- Most guardrail HTTP calls
Impact: Prevents memory exhaustion in high-traffic deployments and
long-running proxy servers that make thousands of API calls.
Testing: Verified connection limits are applied correctly and
existing functionality remains unchanged.
Add Agent Lightning, Microsoft's open-source framework for training
AI agents with RL, APO, and SFT. Uses LiteLLM Proxy for LLM routing
and trace collection.
