P1: Service.delete() fails for env-var-only deployments
settings.delete() raised LookupError (no DB row) before remove_job()
was reached — the scheduler kept running with no way to stop it via API.
Fix: deregister scheduler job first (independent of DB), then only
delete the DB row when credentials are NOT from env vars.
P1: GCS resumable session left open on _put_chunk failure
A mid-stream exception abandoned the session URI; GCS held it open for
up to 1 week, accumulating stale sessions on repeated transient failures.
Fix: wrap the chunk loop in try/except — on any exception, send DELETE
to the session URI (best-effort via contextlib.suppress) then re-raise.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Bug 1: float(df["spend"].sum()) raises TypeError when all spend values
are NULL — polars .sum() returns None for an all-null series.
Fix: float(df["spend"].sum() or 0.0)
Bug 2: df["completion_tokens"].sum() accessed without checking the column
exists — only "prompt_tokens" was guarded. Raises ColumnNotFoundError if
schema omits completion_tokens.
Fix: guard both columns together before summing.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
PUT /mavvrik/settings saved new credentials to DB but the running
Orchestrator kept its old Client with stale api_endpoint/connection_id —
directly contradicting "update credentials without restarting".
Fix: after saving merged credentials, reschedule the job with a new
Client/Uploader/Orchestrator built from the merged values using
replace_existing=True — same pattern as initialize(). The updated
credentials take effect on the next scheduler tick.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Importing 'from litellm.integrations.mavvrik import Service' at module level
caused the entire mavvrik package to load at proxy startup, which could
transitively import polars (optional [proxy] dep) for SDK-only users.
Fix: lazy import via _get_service() helper — Service (and the polars dep chain)
is only loaded when a Mavvrik endpoint is actually called, not at import time.
Note: cloudzero_endpoints.py and vantage_endpoints.py use the same
unconditional pattern; we've proactively fixed ours to be stricter.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Previous fix added api_key but the @@unique constraint on
LiteLLM_DailyUserSpend is (user_id, date, api_key, model,
custom_llm_provider, mcp_namespaced_tool_name, endpoint). Two rows
sharing (date, user_id, api_key, model) but differing in provider or
endpoint still have indeterminate order, causing OFFSET pagination to
duplicate or drop rows at page boundaries.
Fix: add dus.id (primary key, always unique) as the final tiebreaker.
Cleaner than enumerating all 7 constraint columns.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Service.export() and Service.dry_run() called exporter.export() which
silently returns an empty DataFrame when prisma_client is None, causing
the endpoint to return {"status": "success", "records_exported": 0} —
identical to a legitimate zero-traffic day. Admins had no way to tell
the difference.
Fix: call Settings._ensure_prisma_client() before reaching the exporter
in both methods. This raises an Exception with a clear message when the
DB is not connected, surfacing as a 500 response to the caller.
Also fixes the misleading exporter.py module docstring which claimed
"user-triggered endpoints surface the missing-DB error through
Settings._ensure_prisma_client() before reaching here" — that was wrong
until this commit.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Without dus.api_key in the ORDER BY, pagination is unstable: multiple rows
can share the same (date, user_id, model) with different api_key values.
When page boundaries fall inside such a group the DB can return those rows
in any order, causing duplicates or silent drops across pages.
Fix: add dus.api_key to the sort key, matching the table's @@unique
constraint: (user_id, date, api_key, model, custom_llm_provider, ...).
This guarantees a stable total order for all OFFSET-based pagination.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Finding 3 — proxy_server.py: remove inline `import os as _os`
Every other integration block in _initialize_spend_tracking_background_jobs
uses the module-level os import already in scope. Changed _os.getenv(...)
to os.getenv(...) and removed the mid-function import alias.
Finding 4 — custom_logger_registry.py: fix MavvrikLogger import order
MavvrikLogger was inserted between cloudzero and datadog imports.
Alphabetically mavvrik sorts after datadog/datadog_llm_obs/datadog_metrics
and before deepeval. Moved import to the correct alphabetical position.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
P1 — exporter.py: lazy polars import (SDK users)
polars is optional [proxy] dep. Importing at module level caused SDK users
to get ModuleNotFoundError via the import chain:
custom_logger_registry → mavvrik/__init__ → exporter → polars.
Now imported inside each method that uses it. TYPE_CHECKING guard for
type annotations only.
P1 — orchestrator.py + exporter.py: distinguish DB error vs zero-traffic day
_stream_pages now raises RuntimeError when DB is None instead of silently
returning. This propagates through _export → caught by _run_pipeline →
report_error, marker NOT advanced. Zero-traffic days yield nothing without
raising → advance IS called (correct).
P1 — uploader.py: accept 200 or 201 for GCS session initiation
Standard GCS returns 200 OK for resumable upload initiation, not 201.
P1 — test_e2e_upload.py: moved to tests/local_testing/test_mavvrik_e2e.py
BerriAI rule: tests/test_litellm/ must contain only mock-based tests.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
P1 — orchestrator: gate _advance on total_bytes > 0
When DB is unavailable, _export returns 0 bytes. Previously _advance was
called unconditionally, permanently skipping those dates (marker advances
past them). Now raises RuntimeError on 0 bytes so the single try/except
catches it, calls report_error, and leaves the marker unchanged — the
date is retried on the next scheduled run.
P1 — __init__.py: remove top-level `import polars as pl`
polars is an optional [proxy] dependency. `custom_logger_registry.py`
imports Logger from this package at module level, which triggered the
polars import for all SDK users. polars is not used directly in __init__.py
(Exporter uses it internally) so the import can be removed entirely.
P2 — uploader: add Content-Range header to _finalize_upload
GCS resumable upload spec requires Content-Range: bytes 0-{last}/{total}
on the final PUT. _finalize_upload was omitting it. Now consistent with
_put_chunk(final=True) which already sends Content-Range correctly.
CI — documentation_test_env_keys: env vars already in config_settings.md
(lines 607-611), failure was from an older commit.
CI — lint: factory.py Black format — pre-existing upstream issue, not our code.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
BerriAI enforce_async_clients check requires using the shared cached client
from get_async_httpx_client() instead of creating httpx.AsyncClient() per
request. Creating per-request clients adds +500ms latency overhead.
_http.py now calls get_async_httpx_client(LoggingCallback).client to get the
shared httpx.AsyncClient from LiteLLM's in-memory client cache, then calls
.request() on it directly (AsyncHTTPHandler only has verb-specific wrappers,
not a generic request() method).
Test mocks updated to patch get_async_httpx_client instead of httpx.AsyncClient.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Bedrock enforces non-increasing TTL ordering across cache_control blocks
(tools → system → messages). The tool cache_control TTL was being
unconditionally dropped to the default 5m, while system blocks preserved
the user-specified TTL for Claude 4.5+ models. This mismatch caused
"a ttl='1h' block must not come after a ttl='5m' block" errors when
users set ttl='1h' on both tools and system.
Converse path: add_cache_point_tool_block() now accepts a model param
and preserves TTL for Claude 4.5+, matching _get_cache_point_block().
Invoke path: _remove_ttl_from_cache_control() now also processes tools
(was only processing system and messages).
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Waiting to run
Two changes, both test-only:
- Configure the aiohttp session with TCPConnector(force_close=True) and an
explicit ClientTimeout(total=30, connect=10). Prevents reuse of idle TCP
connections that the proxy/kernel may have closed during the long window
between setup POSTs and the later poll loop, and surfaces a blocked proxy
event loop quickly instead of hanging on aiohttp's 5-minute default.
- In poll_key_spend_until, catch aiohttp.ClientError and asyncio.TimeoutError
around the single /key/info call. A transient transport hiccup now logs and
retries on the next tick instead of failing the entire polling loop.
Addresses the ConnectionTimeoutError observed on the first /key/info call
after the 20 chat completions.
The daily queue parameter types on _restore_spend_updates_to_in_memory_queues
were narrowed to specific subtypes (DailyUserSpendTransaction, etc), but
the caller passes Dict[str, BaseDailySpendTransaction] — the return type
of flush_and_get_aggregated_daily_spend_update_transactions. Widen the
parameters to the base type.
Also replace dynamic TypedDict key lookup (which returned object) with
explicit literal-keyed get() calls so mypy can type-narrow each field.
Replace recursive `_walk` helper with a stack-based traversal so the
recursive_detector CI check passes without adding to the ignore list,
and avoid Python recursion limits on deeply nested payloads.
Made-with: Cursor
store_in_memory_spend_updates_in_redis drained the in-memory queues
into local variables before the rpush pipeline. If rpush raised (cloud
Redis hiccup, timeout, connection blip), those already-drained
transactions were garbage-collected with the scheduler job, silently
losing all spend aggregated during that tick.
Wrap the rpush in try/except. On failure, re-enqueue the aggregated
transactions into their respective in-memory queues so the next
scheduler tick retries.
Add a unit test that seeds real queues, simulates an rpush failure,
and asserts the transactions land back in-memory.
Replace the calibration step (one request + 10-minute poll) with an
independent ground truth computed from response usage via
litellm.cost_per_token. All N requests are made up front, so a single
dropped Redis write no longer kills the test.
Add /health/readiness checks at test start and on poll timeout so the
failure message surfaces proxy state (db, cache) instead of "calibration
timed out".
Set PROXY_BATCH_WRITE_AT=2 in the spend tracking CI job to shorten the
scheduler flush window.
Principle: GHA handles work that doesn't need external API keys; CCI
stays for integration tests that hit real API endpoints.
Four CCI jobs moved to new or extended GHA workflows:
1. check_code_and_doc_quality (was 25 runs: ruff + import-safety +
21 code_coverage_tests + 3 documentation_tests + circular-imports).
- The 21 tests/code_coverage_tests/*.py scripts and the 3
tests/documentation_tests/*.py scripts run in the new
.github/workflows/test-code-quality.yml workflow.
- ruff, import-safety, and circular-imports were already run by
.github/workflows/test-linting.yml — no new migration needed.
- The 3 documentation_tests scripts read
docs/my-website/docs/proxy/config_settings.md. Since docs have
moved to BerriAI/litellm-docs, the GHA workflow checks out that
repo and symlinks docs/my-website -> the checkout so the
existing hardcoded paths resolve without touching the scripts.
The stale local docs/my-website/ copy in this repo will be
removed in a separate PR.
2. semgrep (custom-rule SAST against .semgrep/rules).
- New .github/workflows/test-semgrep.yml.
3. installing_litellm_on_python + installing_litellm_on_python_3_13
(pip install compat checks on Python 3.12 and 3.13).
- New .github/workflows/test-install-litellm.yml as a matrix job.
- 3.12 run also verifies litellm_enterprise import; 3.13 run
skips that check (matches previous CCI behavior).
- installing_litellm_on_python_v2_migration_resolver stays in CCI
because it requires a postgres service.
CCI .circleci/config.yml: -112 lines, 4 jobs and their workflow refs
removed.
Restore guardrail spend/UI event_type wiring, request_data on streaming
OUTPUT paths, and centralized match redaction after the upstream revert.
Made-with: Cursor
Follow-up cleanup after an independent review pass surfaced a few
loose ends:
- Delete a 6x-duplicated filter block in litellm_mapped_tests_proxy_part2
(same kind of copy-paste residue we fixed earlier in
langfuse_logging_unit_tests).
- Delete the empty "Install Semgrep" run step in the semgrep job — the
command body was empty because semgrep is installed on-demand via
uv tool run in the next step.
- Standardize machine-executor image: one job was on ubuntu-2204:2023.10.1
while build_docker_database_image was already on ubuntu-2204:2024.04.1.
Bumped everything to 2024.04.1.
- Remove the legacy "version: 2" inside the workflows: block — CircleCI
2.1 top-level already declares the version.
- Drop `{{ checksum ".circleci/config.yml" }}` from cache keys (13 sites).
It was busting the cache on every unrelated config edit; the uv.lock
checksum alone is the right dependency cache key.
- Add partial-restore fallbacks to every restore_cache with a single
templated key (10 sites). Jobs now fall back to the latest cache with
a matching prefix if the exact uv.lock hash isn't cached yet.
Net: -14 lines.
Docker-executor jobs:
- Consolidate base images on cimg/python:3.12. Jobs previously on
3.11 (26 jobs), 3.9 (1 historical: upload-coverage), and an
incidental 3.13.1 (litellm_assistants_api_testing) now use 3.12.
- installing_litellm_on_python_3_13 keeps cimg/python:3.13.1 as its
explicit "latest Python supported" install-check matrix job.
Machine-executor jobs:
- Delete the miniconda install step from 10 jobs. uv now manages
Python directly: uv sync --python 3.12 auto-downloads a
python-build-standalone interpreter if the ubuntu-2204 base
image's default python doesn't match.
- Remove 37 "if [ -f conda.sh ]; then conda activate myenv" wrappers
and 2 unconditional conda activate blocks left behind from the
conda days.
- proxy_build_from_pip_tests keeps its 3.13 target (it was
conda create -n myenv python=3.13) via uv sync --python 3.13.
Net: -301 lines.
Add a start_postgres command parameterized on db_name (default
circle_test) that runs the postgres-db container and waits for port
5432 to accept connections. Replace all 11 inline docker run /
wait_for_service blocks with a single - start_postgres call.
The helm chart test overrides db_name to litellm_test; everything
else uses the default.
One of the 11 sites previously used a bespoke pg_isready loop instead
of wait_for_service; it now goes through the same TCP-probe path
everyone else uses, which is sufficient for test ordering purposes.
Net: -112 lines.
Add a single install_uv command in the commands: section that encodes
the uv version (0.10.9) and its SHA256 in one place, then replace all
42 inline curl|sha256|install blocks across every job that needs uv.
setup_litellm_test_deps now calls install_uv too, so the shared
test-dep bootstrap goes through the same path.
Bumping uv version or SHA is now a one-line change instead of 43.
Net: -203 lines.
- Remove mypy_linting job (GHA test-linting.yml already runs this)
- Remove three redundant "Install curl" apt-get steps (curl is
already present on the ubuntu-2204 machine image and used
successfully earlier in each affected job)
- Dedupe langfuse_logging_unit_tests filter block (6x copy of the
same two branch filters collapsed to 1)
- Pin all docker image references by @sha256 digest so builds stay
reproducible when upstream tags are updated:
cimg/python:3.9, 3.11, 3.12, 3.12-browsers, 3.13.1, cimg/node:20.19,
cimg/postgres:16.0, and postgres:14 used via docker run
Net: -62 lines, 49 image references pinned.
* add moonshot/kimi-k2.6 to model registry
* add moonshot/kimi-k2.6 to backup model registry
* add tests for moonshot/kimi-k2.6 model registry
* fix moonshot/kimi-k2.6 pricing and add reasoning support
* fix moonshot/kimi-k2.6 pricing and add reasoning support in backup
* update kimi-k2.6 tests: fix pricing, add tool_choice and reasoning checks
* fix: load kimi-k2.6 registry tests from local backup instead of remote cost map
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Has been cancelled
Unit Tests: Proxy DB Operations / proxy-db (auth-checks, tests/proxy_unit_tests/test_auth_checks.py tests/proxy_unit_tests/test_user_api_key_auth.py, 20, 8) (push) Waiting to run
Unit Tests: Proxy DB Operations / proxy-db (remaining, tests/proxy_unit_tests --ignore=tests/proxy_unit_tests/test_key_generate_prisma.py --ignore=tests/proxy_unit_tests/test_auth_checks.py --ignore=tests/proxy_unit_tests/test_user_api_key_auth.py --ignore=tests/proxy_unit_tests/test_p… (push) Waiting to run
