A converted-stream request whose cache entry is a plain (non-stream) object is
replayed as that plain object, so nothing later fires the success callbacks.
Decide deferral from the replayed result's type instead of the request kwargs.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar
py-spy on the gateway showed the post-response _PROXY_track_cost_callback,
spend-log and DBSpendUpdateWriter work running on the inference workers'
event loop, so a DB or Redis stall backed up the request path.
When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact
typed SpendEvent per success and hands it to a SpendEventProducer that ships
it over a unix socket (default) or loopback-only TCP to a sidecar started as
`python -m gateway.spend_worker`. The sidecar runs the unchanged
_ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url).
When the sidecar is unreachable, the buffer is full, or the gateway shuts
down with events still queued or in flight, the producer applies
LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The
sidecar half-closes producers on SIGTERM and drains, the producer treats
EOF as unavailable, and the gateway flushes buffered spend counters on
shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its
own process log.
Helm: both charts gain an opt-in spend-worker sidecar container sharing an
emptyDir socket dir, and the componentized chart's HPA uses a
ContainerResource CPU metric scoped to the gateway container so sidecar
CPU does not drive inference scaling.
* feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks
Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): retrieve a completed batch in the in-process spend path test
The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(proxy): rename the spend worker sidecar to collector
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown
The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime
stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the
pod-local pgbouncer URL itself, replacing gateway/collector.py
The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it.
SpendEventProducer shields an in-progress fallback from the writer task cancellation so close()
no longer loses an event already handed to the in-process pipeline
Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs,
flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci(terraform): drop the gcp job duplicated by the aws/gcp matrix
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports
The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(proxy): format pooled_database_url after the pgbouncer rebase
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain
Cache hits updated the logging object after the early return, so the offloaded spend event carried
preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against
producer transports uvloop already closed so one dead connection cannot abort the drain
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(terraform): keep the gcp collector port off the metrics sidecar health port
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): collector connects to Postgres directly under IAM or Entra token auth
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop the cwd-relative sys.path.insert calls from the test suite
TQ003 stands at 1,077 across 1,058 files, and 1,015 of them are the same shape:
sys.path.insert(0, os.path.abspath("../..")) and its deeper siblings. The
argument resolves against the working directory rather than the file, so from
the repo root, where every job runs pytest, it inserts the directory two levels
above the checkout. It has never pointed at litellm. The package is installed
into the environment anyway, which is what actually makes the import work, and
what the rule's message has said all along.
Removing them leaves 1,634 imports of sys and os with no remaining reference,
and those go too, except where another test module imports the name back out of
the file. The rest of TQ003 is 62 call sites that resolve against __file__ or a
variable, which are a different question and are left alone.
Collection is identical either way: 45,871 tests and the same 51 pre-existing
collection errors before and after, and ruff reports no new undefined name.
* test: drop the duplicate imports the sys.path sweep exposed to F811
* test(pre-call-utils): restore the os import the new bedrock tests need
A name bound twice keeps only the second binding. In `tests/` that is nearly
always a repeated import, harmless but misleading, and the same rule is what
catches the cases that are not harmless: a local that shadows an import the
module still calls, and a second `def test_x` that quietly replaces the first.
311 of the 344 sites were repeated imports and came out with ruff's own fix.
The remaining 33 needed a decision. Four modules imported a name they never
used because a local definition below already shadowed it. Two comprehensions
bound `call` over `unittest.mock.call`, which those modules import and use.
One test rebound the two module handles its nested reload closure had captured.
One class attribute shadowed an unused `status` import.
The load-test fixtures move to a conftest, which is how pytest is meant to share
them, so the test module no longer imports three fixture names it never calls.
The nine `prisma_client` parameters keep a narrow `noqa`: pytest resolves that
fixture by name before the body runs, so the parameter never shadows anything.
Three process-lifetime retention points kept full request payloads
(messages included) alive after the request finished. Under bursts of
large-token traffic (~73K tokens/request mean) this presented as
stepwise RSS growth that never returned to baseline, ending in OOM:
1. Logging.pre_call/post_call stored their entire locals() (messages,
the Logging object, complete_input_dict) in the module-level
litellm.error_logs dict, pinning the most recent request's payload
per worker forever. Nothing reads that dict; the writes are removed.
2. LLMCachingHandler.request_kwargs kept litellm_logging_obj inside the
stored kwargs while the handler itself hangs off
logging_obj._llm_caching_handler, closing a reference cycle
(Logging -> LLMCachingHandler -> kwargs -> Logging). Cyclic payloads
are only reclaimed by generational GC, so megabytes of dead request
data lingered until a rare gen-2 pass, and the transient copies
fragment the allocator into a permanent RSS high-water mark. The
handler now drops litellm_logging_obj from its stored kwargs; the
caching layer never reads it.
3. The router stored every request's kwargs in the ITPM/OTPM contextvar
even when no deployment configures itpm/otpm. Pooled resources
created mid-request (e.g. redis connections) capture the asyncio
context, extending that pin far past the request. The slot is now
populated only for deployments with io token limits and overwritten
with None otherwise.
Live-proxy verification (bursts of 30 x ~300KB requests, PII guardrail
+ prometheus + redis cache): unfixed grows 16-29MB per burst without
release; fixed grows under 1MB per burst after warmup and flattens.
Resolves LIT-4434
* fix(caching): replay openai/responses bridge cache hits as chat streams
When chat completions route through openai/responses, cached ModelResponse
payloads under aresponses keys were deserialized as ResponsesAPIResponse
(500) or re-translated as responses events (empty streaming deltas). Deserialize
chat-shaped cache entries as acompletion and bypass the responses stream iterator
for cached CustomStreamWrapper replay.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(caching): map responses bridge call_type for sync vs async stream replay
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix: handle ModelResponse cache return in responses bridge and drop dead acompletion check
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(caching): detect chat cache hits via object field before choices fallback
Prefer chat.completion object type over the broad choices-key heuristic so
Responses API cached payloads are not misclassified if their schema changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* test(caching): cover responses bridge cache-hit paths in CI-tracked test suite
The new bridge cache replay logic in caching_handler.py and the
preformatted-stream guard in litellm_responses_transformation/handler.py
were exercised only by tests under tests/local_testing/, which the
responses-caching-types and misc shards do not run. Codecov flagged the
patch as 29.72% covered.
Add equivalent unit tests under tests/test_litellm/ so the responses,
caching, types, and misc shards execute them and ship their coverage
data to Codecov:
- _is_chat_completion_cached_dict happy/sad paths
- aresponses streaming bridge cache hit -> CustomStreamWrapper
- responses non-streaming bridge cache hit -> ModelResponse
- legacy ResponsesAPIResponse stream + non-stream replay
- _is_preformatted_cached_chat_stream true/false
- completion/acompletion early return on cached ModelResponse
- completion/acompletion skip rewrap on preformatted cached stream
* fix: add negative guard on object field in _is_chat_completion_cached_dict
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(vcr): treat corrupt cassette payloads as cache miss
* test: bump EOL'd NVIDIA rerank and OpenAI realtime models in CI
The NVIDIA hosted rerank endpoint for nvidia/llama-3_2-nv-rerankqa-1b-v2
reached end-of-life on 2026-05-18 and now returns HTTP 410 Gone, breaking
TestNvidiaNim::test_basic_rerank. Switch to nvidia/nv-rerankqa-mistral-4b-v3,
which is still hosted on the NVIDIA API catalog and is already listed in
model_prices_and_context_window.json.
OpenAI also retired the gpt-4o-realtime-preview-2024-12-17 model used by
test_realtime_guardrails_openai (now returns model_not_found). Switch the
realtime test URL to the GA gpt-realtime alias.
Unrelated to the responses-bridge cache fix in this PR, but committing
here to unblock CI per maintainer guidance.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(realtime): switch retired gpt-4o-realtime-preview to gpt-realtime
OpenAI removed gpt-4o-realtime-preview and all its date snapshots on
2026-05-18 (every variant now returns model_not_found), breaking the
live-WebSocket OpenAI realtime tests in CI:
- test_openai_realtime_direct_call_no_intent
- test_openai_realtime_direct_call_with_intent
- TestOpenAIRealtime.test_realtime_connection
- TestOpenAIRealtime.test_realtime_with_query_params
Point each of those to the current GA alias gpt-realtime (verified live).
Pure unit/mock tests that just assert the string value (e.g. in
test_realtime_query_params_construction and the
test_realtime_query_params_use_normalized_model_name mock) are left
alone since they do not depend on model availability.
Also relax the AI-response assertion in
test_text_message_blocked_by_guardrail_no_ai_response: gpt-realtime
occasionally produces a polite refusal ("I'm sorry, but I can't say
that") when the cancel arrives after the model has already started
generating, which is the expected outcome (no real AI content) but does
not contain the words 'blocked' or 'guardrail'. The primary guardrail
behaviour (guardrail_violation error event + transcript_delta block
message) is still asserted unchanged.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(nvidia_nim): mock rerank live API instead of hitting EOL'd endpoint
NVIDIA reached end-of-life for the hosted nvidia/llama-3.2-nv-rerankqa-1b-v2
rerank API on 2026-05-18 (returns HTTP 410 Gone), and the proposed
replacement nv-rerankqa-mistral-4b-v3 returns HTTP 404 for the CI account,
breaking TestNvidiaNim::test_basic_rerank.
Override test_basic_rerank to mock the HTTP transport (same pattern as
test_nvidia_nim_rerank_ranking_endpoint above) so the request/response
transformation and cost calculation stay covered without depending on
NVIDIA's hosted catalog rotation. The model identifier reverts to the
original llama-3.2-nv-rerankqa-1b-v2 since the request never leaves
the test process.
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(caching): preserve prompt_tokens_details through embedding cache round-trip
The embedding caching layer was dropping prompt_tokens_details (including
image_count) because CachedEmbedding had no field for usage metadata and
the cache retrieval code reconstructed Usage without it. This caused
inconsistent responses where the first call returned image_count but
cached responses did not, breaking cost tracking for multimodal embeddings.
Add prompt_tokens_details to CachedEmbedding, persist per-item details
during cache storage, aggregate them on retrieval, and merge them in
combine_usage() for partial cache hits.
* style: apply Black formatting to caching files
* fix(caching): address Greptile review — cyclic import, guarded construction, nested dict merge
Move PromptTokensDetailsWrapper to inline import to resolve CodeQL cyclic
import warning. Guard PromptTokensDetailsWrapper construction with
try/except to handle unexpected cached keys. Add recursive dict merging
in _merge_prompt_tokens_details for nested fields like
cache_creation_token_details.
* fix(helicone.py): add helicone api base support
Fixes https://github.com/BerriAI/litellm/issues/10825
* test: add unit test for cache hit response on embedding calls
* fix(caching_handler.py): fix handling cache hit on embedding when input is string
Fixes LIT-197
* docs(helicone_integration.md): document new helicone api base param
