Greptile finding: tests/test_litellm/readme.md documents that directory as
mocked-tests-only, but the six new Lua script tests contacted a real
redis-server subprocess. Moved them verbatim to
tests/local_testing/test_tag_rate_limits_shared_redis.py, which already
hosts other real-Redis tests (test_redis_batch_optimizations.py). No
behavior change; tests/test_litellm/proxy/hooks/test_tag_rate_limits_shared.py
is mock-only again.
Greptile/Bugbot findings from the first review round:
- extract_key_hash/extract_key_alias only checked the top-level metadata
bucket, unlike order_tags_for_identity_resolution's _active_metadata_bucket
helper. By async_log_success_event/async_log_failure_event time, metadata
is only nested under litellm_params, so both silently returned None there,
diverging from the value read at admission time. Fixed by routing both
through _active_metadata_bucket, with regression tests for the nested case.
- Condensed the module's docstrings/comments to one concise line each,
keeping the non-obvious behavior they document, matching the trimming
already applied to litellm/types/router.py on the stacked PR #39900.
- Added real Redis execution coverage for both Lua scripts (admission,
rejection, ttl refresh semantics, decrement floor) against a throwaway
local redis-server, since fakeredis has no EVALSHA support without the
optional lupa dependency.
Adds tag_rate_limits_shared.py: Lua scripts for atomic bucket
admission/release, bucket-key hashing and policy-fingerprint primitives,
cache partitioning, and identity resolution / security-hardening helpers
both tag-scoped rate-limit hooks will share. Nothing consumes this module
yet; that starts in a later PR in this resplit.
PR 2 of a resplit of the tag-based rate limiting feature into much smaller
PRs. Stacked on #39900 (litellm_tag_rate_limit_types) for the
TagRateLimitScope/TagRateLimitEntry types this module depends on.
Greptile flagged this as P1 on the stacked #39902: both tag
identifiers accepted an empty string, and downstream identity lookup
builds a key off tag_id, so an empty value would silently search for
a bare colon prefix and never match instead of erroring at config
load time. Rejects it at construction, matching the existing
non-empty guard on TagRateLimitScope.values. Verified the new tests
fail without the fix and pass with it.
Codecov flagged 57% patch coverage on the new config types: only 3 of
TagRateLimitEntry's validators had tests, and TagRateLimitScope,
TagRateLimitGroup, TagRateLimits, and ModelInfo.tag_rate_limits had
none at all. Added a test for each remaining validator/normalizer
(NaN limit, non-positive period/ttl/cache-size, empty allowlists,
sort+dedup normalization, frozen scope) and for constructing the
group/aggregate types. Spot-checked kill power on the NaN check, the
sort+dedup normalizer, and the frozen guard by reverting each one and
confirming its test fails.
Greptile flagged the new TagRateLimitEntry/TagRateLimitScope docstrings
and the InMemoryCache ttl forwarding comment as excessive per the
repo's source-comment rule. Condensed each to one concise line while
keeping the non-obvious behavior they document.
Add TagRateLimitScope/TagRateLimitEntry/TagRateLimitGroup/TagRateLimits to
litellm/types/router.py and a ModelInfo.tag_rate_limits field. Pure config
types with no runtime behavior yet; a later PR wires up the enforcement
hooks that consume them.
Also fix InMemoryCache.async_increment_pipeline, which dropped each
pipeline operation's own ttl and always fell back to the cache's default
ttl, silently resetting long-window counters early. Independent bug fix,
found while building the tag rate limiting feature but unrelated to it.
First of a resplit of the tag-based rate limiting feature into much
smaller PRs, per maintainer feedback that ~250 LOC PRs review well and
the prior 3-way split (#38289, #38292, #38347, #36541) was still too
large; those four PRs are now closed in favor of this resplit.
The squash of #40512 onto a base that already carried #40514 left the Partial call on the metadata pre-read error path only, so a rejected /key/update again persisted the planned values into state and TestResourceKeyUpdateFailureKeepsPriorState fails on the default branch.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
get and post already take follow_redirects. put built the request and sent it
with the client default, so a caller uploading to a URL it did not choose had
no way to refuse a redirect. Same plumbing as the other two methods.
* feat(prometheus): bucket latency by input sequence length
* style: format startup resolver call
* fix(prometheus): handle unknown input lengths
* fix(prometheus): preserve disabled custom input length labels
* test(prometheus): seed startup snapshot in mocked logger test
* fix(proxy): preserve database setting types during startup
* fix(prometheus): distinguish missing usage and preserve config persistence
Keep quoted database-storage config values intact for legacy persistence readers while using a local boolean for early callback discovery. Distinguish absent provider usage from an explicitly reported zero when labeling latency metrics.
* fix(proxy): normalize input length flag from secret managers
* fix(prometheus): isolate input buckets and preserve missing usage
Keep built-in buckets on latency histograms, preserve unrelated custom labels, and classify raw incomplete usage and upstream total-only headers as unknown. Cover count conservation, failure callbacks, explicit zero, startup snapshots, and direct caller compatibility. Drop earlier branch budget changes.
* fix(proxy): defer Prometheus alerting until stored settings load
Reuse successful startup storage resolution and preserve callback deduplication across alerting reloads.
* test(prometheus): restore input length flag between tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* Revert "test(prometheus): restore input length flag between tests"
This reverts commit f302e0b7bd.
* refactor(prometheus): make input length flag config/env only
Drop the Admin UI General Settings row, the safe DB override entry, the startup reorder that loaded DB litellm_settings before Prometheus callbacks, and the alerting-only Prometheus path. The flag now behaves like prometheus_emit_stream_label: litellm_settings in config.yaml or an os.environ reference, applied on restart.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
RefreshableCredentials (IMDS roles) can refresh between the access key, secret and token reads SigV4 performs, producing a mixed-generation signature that S3 rejects with 403 and the log is dropped. Snapshot the credentials with get_frozen_credentials before signing, treat 403 like 500/503 in the upload retry loop, and fetch credentials plus sign again on every attempt in both the async and sync upload paths. Tests load a real botocore credential_process profile and fake only the HTTP boundary with httpx.MockTransport
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): bound concurrent key and spend-counter DB lookups to stop prisma pool thrash
A cache-miss burst fanned every get_key_object DB fallback and SpendCounterReseed
point lookup into the prisma query-engine httpx pool at once. httpcore's request
assignment is O(queued x connections) per event, so the event loop spent most of its
time in pool bookkeeping and the logging worker's 20s wait_for tripped. Callers now
wait on a small shared semaphore (PROXY_DB_LOOKUP_MAX_CONCURRENCY, default 25)
instead of queueing inside httpcore
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the in-flight counting prisma fake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop module docstring from db_lookup_gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): type the in-flight counting table fake
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(openai): drop tool schema regex patterns OpenAI's validator cannot compile
OpenAI validates function tool parameters with jsonschema's format checker,
which compiles every pattern with Python re. Claude Code's Artifact tool ships
an ECMA-262 pattern with \p{..} Unicode property escapes, so any OpenAI target
behind /v1/messages, /v1/responses or /v1/chat/completions 400s with
"Invalid schema for function 'Artifact': '...' is not a 'regex'" for every
model family. Drop only the patterns Python re rejects, keep the rest, at the
same seams that already flatten top-level combinators.
* fix(openai): walk only schema positions, iteratively, and drop regexes for every openai deployment
Review round: the regex sanitizer now walks JSON Schema applicator positions
only (properties, items, prefixItems, combinators, $defs, additionalProperties
and the rest), so a pattern key inside default, examples, const or a vendor
extension is data and stays. It also drops patternProperties keys Python re
cannot compile, which OpenAI checks the same way. The walk is level-order and
rebuilt deepest level first instead of recursive, so the code-quality recursion
gate passes and there is no depth cap below what a JSON parser admits. On the
chat wire an openai deployment with a custom api_base now drops such regexes
too, since that base is usually a proxy in front of the same validator, while
the lossier combinator flattening stays limited to api.openai.com hosts.
Encode complete non-Claude source names and include source_model in the
Claude Code listing. Preserve configured route and alias precedence,
normalize once before model policy checks, and select CLI models using
explicit source identity instead of name stripping or positional joins.
Resolves LIT-7360
Claude-Session: https://claude.ai/code/session_01WyqeRhfZGm26zAnHx9P3kq
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
* fix(proxy): surface runtime-registered callbacks in /get/config/callbacks
Config-file callbacks fire at runtime but never appear in the UI Logging
and Alerts page because /get/config/callbacks only reads the DB-merged
config. Append runtime-registered callbacks from LoggingCallbackManager
as read-only rows, deduplicated against configured rows via alias
normalization. UI hides edit/delete/test actions for read-only rows.
* fix: filter internal proxy hooks from runtime callbacks, update test
- Filter _PROXY*, ShadowEval, ServiceLogging, SkillsInjection, ResponsesID prefixes
- Update test to exclude read_only rows from count assertions
- Still allows deployment/guardrail callbacks to surface if configured
Note: comprehensive internal-hook filtering deferred, live-pr-risk will
observe real behavior on running proxy.
* fix: guard non-list config callbacks in get_config, use monkeypatch in tests
- Line-concat type error: normalize_callback now returns empty list for non-list types (dict/tuple/set) instead of passing through unchanged; prevents TypeError when config values are non-list
- Test quality TQ005: replace manual try/finally save-restore of litellm.callbacks with monkeypatch.setattr in test_get_config_callbacks_appends_runtime_only_callbacks and test_get_config_callbacks_redacts_runtime_only_row_secrets_for_view_only_admin
- Ruff format: wrap _internal_callback_prefixes tuple and isinstance check across multiple lines to respect 120-char limit
- All three new tests pass
* fix: rework runtime callback inventory filtering and dedup
- Filter internal proxy hooks by name: _PROXY_ prefix plus fixed internal names (cache, _ProxyDBLogger, deployment callbacks, service hooks)
- Hide guardrail instances and runtime instances of already configured callbacks via CustomLoggerRegistry class lookup
- Sort runtime rows and dedup per mode for stable output
- normalize_callback returns tuples for str/None/list config values and empty for any other type
- Tests mock get_callbacks_by_type explicitly and pin the exact row set; UI test covers read_only action hiding
* fix: list dict-shaped callback config values by their keys
Dict-valued success_callback/failure_callback/callbacks settings previously listed their keys as editable rows; keep that behavior instead of dropping them to read-only runtime rows. Adds a pin test for the dict shape.
* fix: mark dotted-path callbacks read-only to prevent duplicate display
Configured callbacks loaded from dotted Python paths (e.g. custom_callbacks.my_logger) are never matched against runtime instances by name because the registry uses short canonical names (e.g. langsmith, arize). Mark these rows read-only to prevent the UI from attempting delete operations that would fail at the endpoint level anyway.
* fix: dedupe dotted-path callbacks by instance module instead of marking them read-only
A dotted-path callback loaded from config registers as an object, so it
surfaces at runtime under its class name and never matched the configured
string, producing a second row. Marking the config row read_only hid the
duplicate but also hid delete, which does work for these rows.
Match the live instance back to its configured entry by module and drop it
from the runtime rows, so the callback stays a single editable row.
* test: cover dotted-path dedup across success, failure, and callbacks modes
* fix(proxy): filter runtime callback inventory by object identity and label read-only rows in the UI
Runtime-only rows were filtered by callback name, which missed initialized
CustomLogger instances, router and proxy hook methods, guardrails, and
user functions. The inventory now inspects the live callback objects
through a public LoggingCallbackManager.get_callback_objects accessor
and hides litellm-internal hooks, guardrails, and instances of already
configured callbacks. The dashboard shows a Read only label for
runtime-only rows instead of an empty action cell
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): keep configured-callback assertions minimal when runtime rows are present
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hide internal cache string callback from runtime callback inventory
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): hide auto-registered vector store hook from callback inventory
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep YAML OTel-family callbacks listed next to a configured one
arize, weave_otel and langfuse_otel all initialize OpenTelemetry subclasses, so hiding runtime
callbacks by configured class made one saved OTel callback swallow its YAML siblings. Match runtime
instances by their own callback_name and only fall back to class identity for bare OpenTelemetry
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): cover scalar and null YAML callback keys in callback inventory
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): drop docstrings that restate callback inventory helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep runtime-only s3 and sqs callbacks in UI Logging inventory
_is_litellm_internal_callback checked registry membership with the display alias (s3, sqs), which is not a registry key, so runtime-only S3Logger and SQSLogger instances were classified as internal and dropped from /get/config/callbacks. Check the registered name instead and cover both loggers in the internal-exclusion regression test
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
resourceKeyUpdate now calls d.Partial(true) on a failed /key/update so the SDK does not persist the rejected planned values into state, which made the next plan report no changes and silently dropped the update.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): send litellm_key model_max_budget as per-model BudgetConfig objects
The key resource typed model_max_budget as map(number) and forwarded the
bare numbers to /key/generate, which the proxy rejects with a 500
('int' object is not iterable) because each model entry must be a
BudgetConfig object. The attribute is now a JSON string of per-model
budget objects, matching litellm_user, litellm_budget and litellm_tag,
with a schema version 1 state upgrader that drops the old map value so
existing state keeps loading
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): reject litellm_key model_max_budget JSON that is not per-model budget objects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): reject unknown or empty per-model budget fields in litellm_key model_max_budget
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add an optional, computed, force-new team_id argument to the
litellm_team resource. When set, it is sent to /team/new as the
team_id; when omitted the provider keeps generating a UUID. Read
mirrors the resource id into state so import and plan stay clean.
Resolves LIT-6399
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): read team per-model rpm/tpm limits from metadata and clear them on removal
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(terraform): drop explanatory comments from team per-model limit helper and tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
