* feat(proxy): tool policies - auto-discover tools, manage policies, guardrail enforcement
- New LiteLLM_ToolTable in schema.prisma to store discovered tools
- Auto-discovery: tools seen in LLM responses get upserted via ToolDiscoveryQueue
(hooks into DBSpendUpdateWriter, same pipeline as spend tracking)
- Management endpoints: GET /v1/tool/list, GET /v1/tool/{name}, POST /v1/tool/policy
- ToolPolicyGuardrail: blocks tool_calls in responses based on policy setting
- UI: Tool Policies page under Guardrails section with policy selector,
filters by policy/team/key, live tail, sortable table
- Unit tests for queue, writer, endpoints, guardrail
* feat(tool-policies): track call_count + discover tools from request body and /messages API
- Add call_count column to LiteLLM_ToolTable; incremented on every flush
- Extract tools from request body too (not just response tool_calls):
- OpenAI /chat/completions: tools[].function.name
- Anthropic /messages pass-through: request_body.tools[].name
- Show call_count column in UI table (sortable)
- UI: drop dual_llm option, keep only trusted/blocked
* fix: address greptile review feedback
- Remove redundant @@index([tool_name]) from schema.prisma (tool_name has @unique which already creates an index)
- Replace gen_random_uuid()::text with str(uuid.uuid4()) for portability
- Rewrite test_tool_registry_writer.py to mock execute_raw/query_raw (actual implementation) instead of Prisma model methods
- Fix test patches in test_tool_management_endpoints.py to target source modules since imports are inside function bodies
- Add "Tool Policies" page title to ToolPolicies.tsx
* fix: address greptile review round 2
- Replace NOW() with Python datetime parameter in tool_registry_writer (SQLite portability)
- Fix cache key collision in tool_policy_guardrail: use null-byte separator instead of colon
- Remove type==function filter from request-side tool extraction to match response-side behavior
- Clear seen_tool_names on flush so call_count increments per batch cycle not per pod lifetime
* fix: address greptile review round 3
- Fix test_seen_names_persist_across_flushes to match actual per-flush-cycle behavior
- Update module docstring in tool_discovery_queue.py to accurately describe flush behavior
- Add created_at/updated_at to raw SQL INSERT in batch_upsert_tools and update_tool_policy
* fix: cache tool policies per tool name not per combination
Previously the cache key was built from the full set of tool names in a
request, so each unique combination of tools got its own cold cache entry
and triggered a separate DB query. With N distinct tools across requests
this was effectively a DB hit on every request.
Now each tool name is cached individually. Cache hits are checked per
tool, only missing tools are fetched from DB in a single batch query,
and each result is cached separately. Once a tool's policy is warm,
any subsequent request using that tool benefits from the cache regardless
of what other tools are in the request.
* Update ui/litellm-dashboard/src/components/ToolPolicies.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- Wait for strategy select (API data loaded) before clicking Save
- Assert specific payload content in setCallbacksCall
- Move NotificationsManager import to top of file
- Resolve merge conflict in pass_through_endpoints.py
- Add .copy() to proxy_server_request headers to prevent cache corruption
- Add test for request.state unavailable fallback path
- Add per-command error check in _pipeline_lpop_helper to match _pipeline_rpush_helper, preventing silent data loss on WRONGTYPE errors
- Fix pre-existing bug: org spend queue metric was using REDIS_DAILY_SPEND_UPDATE_QUEUE instead of REDIS_DAILY_ORG_SPEND_UPDATE_QUEUE
- Add test for per-command LPOP pipeline error propagation
- Remove re-raise in _store_transactions_in_redis so one Redis
push failure doesn't drop remaining transaction types
- Downgrade per-push success log from info to debug to reduce noise
- Fix misleading error message in update_database — entity spend
updates run as independent tasks and are not affected by this catch
* fix(router): preserve _hidden_params in FallbackStreamWrapper so x-litellm-overhead-duration-ms is emitted for streaming requests
* test(router): add regression test for FallbackStreamWrapper _hidden_params preservation
Set prometheus_emit_stream_label: true in litellm_settings to emit a
stream label (True/False/None) on litellm_proxy_total_requests_metric.
Opt-in to avoid breaking cardinality on existing deployments.
Add a Python script that detects duplicate issues using title similarity
(difflib.SequenceMatcher) and closes them via the gh CLI. Two-tier system:
- 0.6 threshold: informational comment via existing wow-actions step
- 0.85 threshold: auto-close with comment, label, and not_planned reason
Includes a workflow_dispatch workflow for one-time batch scans and
integrates auto-close into the existing check_duplicate_issues workflow
for newly opened issues.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Elevate silent debug-level and bare except:pass error paths to
warning/error so spend tracking failures are visible in production logs.
All new log messages are prefixed with "Spend tracking -" for easy
filtering. Changes cover the full request-to-DB lifecycle:
enqueue, in-memory flush, Redis buffer push/pop, DB commit,
cache updates, spend log writes, and pod lock management.
Also fixes a copy-paste bug in _update_team_cache that logged
"end user" instead of "team".
Replace 11 separate asyncio.create_task() calls per request with a
single batched task that runs all spend-update helpers sequentially.
This reduces task scheduling overhead at high RPS (11,000 -> 1,000
tasks/sec at 1K RPS) and cuts 5 copy.deepcopy(payload) calls to 1
shared copy.
Also fixes a mutation bug where the daily agent spend handler received
the raw payload without deepcopy, unlike all other daily helpers.
Verify that extra_headers are correctly forwarded to OpenAI's
images.generate() in both sync and async paths, and that they
are absent when not provided.
Add headers parameter to image_generation() and aimage_generation() methods
in OpenAI provider, and pass headers from images/main.py to ensure custom
headers like cf-aig-authorization are properly forwarded to the OpenAI API.
Aligns behavior with completion() method and Azure provider implementation.
* staged first pass
* black
* Update litellm/proxy/health_check.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* simpler
* restore cached logo
* fix tests for perform_health_check max_concurrency arg
* implement pr suggestion
* and the helm chart
* add configureable resources and probes to the deployment in the helm chart
* more helm chart unittests
* move some background healthcheck loggin to debug
---------
Co-authored-by: Sean Glover <sglover@athenahealth.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
Replace racy check-then-increment RPM logic with atomic increment-first
pattern to prevent concurrent requests from bypassing the rate limit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
