Two-pronged fix for the reported memory leak where RSS grows from ~1.9 GiB
to ~3.5 GiB under load (1000-1500 concurrent users) and never returns to
baseline:
1. Logging._cleanup_after_logging(): Clears large payload fields (input,
original_response, raw_request_typed_dict) and streaming chunk lists from
the Logging object after all success/failure callbacks have consumed them.
This prevents per-request data (which can be kilobytes per request) from
being held in memory by async task references.
2. _periodic_memory_cleanup(): A scheduled background job (every 60s by
default, configurable via LITELLM_MEMORY_CLEANUP_INTERVAL) that runs
gc.collect() + malloc_trim(0) on Linux. Python's pymalloc allocator does
not return freed memory arenas to the OS by default; malloc_trim forces
glibc to release unused heap pages, allowing RSS to shrink after traffic
spikes subside.
Together these ensure that:
- Per-request memory is released promptly after logging completes
- The OS reclaims freed heap memory periodically, so RSS drops during idle
periods instead of staying at the peak level indefinitely
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: document x-litellm-tags header and request body tags parameter
- Add documentation for x-litellm-tags header (comma-separated or array)
- Add documentation for tags in request body
- Clarify that dynamic tags override config tags
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: consolidate tag documentation and improve cross-references
- Make request_tags.md the single source of truth for all tag options
- Add cross-reference from cost_tracking.md to request_tags.md
- Document both direct tags and metadata.tags formats
- Add key/team tag setup and custom header tracking to request_tags.md
- Reduce duplication and make navigation clearer
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: use generic examples instead of specific company names
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs: clarify x-litellm-tags header format is comma-separated string
HTTP headers are always strings, not arrays. Remove misleading
array format documentation for the header parameter.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* Update docs/my-website/docs/proxy/request_tags.md
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Replace literal Base64-encoded example value in docstring with a
placeholder to prevent GitGuardian/ggshield from flagging it as a
leaked Basic Authentication credential in container scans.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(proxy): add max_iterations limiter for agent session loops (#22058)
Adds a new proxy hook that enforces a per-session cap on the number of
LLM calls an agentic loop can make. Callers send a session_id with each
request, and the hook counts calls per session, returning 429 when the
configured max_iterations limit is exceeded.
- Uses Redis Lua script for atomic increment (multi-instance safe)
- Falls back to in-memory cache when Redis unavailable
- Follows parallel_request_limiter_v3 pattern
- Configurable via key metadata: {"max_iterations": 25}
- Session counters auto-expire via TTL (default 1hr)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add new code execution dataset
* feat(agent_endpoints/): allow giving agents keys
* fix: ui fixes
* feat: allow assigning mcp servers to agents
* fix: eliminate duplicate DB queries in MCP agent auth and N+1 in agent listing (#22110)
- Extract _get_agent_object_permission helper so _get_allowed_mcp_servers_for_agent
and _get_agent_tool_permissions_for_server share a single DB fetch instead of
each independently querying the same agent row (was 1+N queries per MCP request)
- Use include={"object_permission": True} on find_many in get_all_agents_from_db
to eagerly load permissions in one query instead of N+1
- Use include={"object_permission": True} on create/update/find_unique in all
agent CRUD operations, removing attach_object_permission_to_dict follow-up calls
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add new document explaining automatic credential usage tracking and tagging. When models use reusable credentials, LiteLLM automatically injects a Credential: <name> tag on requests, enabling credential-level spend tracking on the Usage page with no additional configuration.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* feat(ui): add user filtering to usage page
Adds "User Usage" as a new view option in the usage page dropdown,
allowing admins to view and filter usage data by individual users
via the existing /user/daily/activity backend endpoint.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat(ui/): working usage filtering
* fix(ui): use single-select for user filter and add tests
The user entity type's backend endpoint only accepts a single user_id,
so the filter now uses single-select mode instead of multi-select.
Added tests for the new user entity type in EntityUsage and
UsageViewSelect. Updated CLAUDE.md and AGENTS.md with guidance on
UI/backend contract consistency and test coverage for new entity types.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: remove unintended package-lock.json changes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* revert: restore package-lock.json to merge base state
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Added unit tests for all pricing calculator components with 64 passing tests across 5 test files:
- multi_export_utils.test.ts (16 tests for PDF/CSV export functions)
- use_multi_cost_estimate.test.ts (15 tests for cost estimation hook)
- multi_export_dropdown.test.tsx (8 tests for export dropdown component)
- multi_cost_results.test.tsx (15 tests for results display and UI states)
- index.test.tsx (10 tests for main calculator component)
Also fixed missing page description for tool-policies page in page_metadata.ts.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Duplicate import was causing UI build to fail:
- Line 4: import { Select, Switch, Tooltip } from 'antd'
- Line 5: import { Select, Tooltip } from 'antd' (duplicate)
Removed the duplicate line 5.
Add a `request_duration_ms` column to `LiteLLM_SpendLogs` to track request
duration. New rows are computed at write time. Legacy rows use a COALESCE
fallback in the `/spend/logs/ui` query to compute duration on the fly from
`endTime - startTime`. The field is also sortable in the UI endpoint.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Google's Interactions API spec changed the status enum:
- Values are now lowercase (was uppercase)
- 'UNSPECIFIED' value was removed
Updated test to match the current spec from:
https://ai.google.dev/static/api/interactions.openapi.json
