Commit graph

85 commits

Author SHA1 Message Date
yuneng-jiang
24afa12253
chore(caching)!: remove the disk cache backend (#48114)
* chore(caching)!: remove the disk cache backend

Drop the disk cache type and its diskcache dependency, along with the
caching install extra that existed only to pull it in. diskcache has not
had a release since August 2023

Cache now raises a ValueError listing the supported types when it gets a
type it does not know. Before, an unknown type built a Cache with no
backend, which only failed on first use

BREAKING CHANGE: cache_params type "disk" and the litellm[caching] extra
are gone. A proxy configured with type disk now refuses to start. Use
local or redis instead

* test(proxy): expect the unsupported cache type error from _init_cache

The test pinned the old fallthrough, where an unknown type left the cache
without a backend and the proxy hit an AttributeError reading it

* chore(deps): drop the diskcache scan ignore main extended

diskcache is no longer in the lock, so the ignore matches nothing

* fix(caching): ignore a leftover disk_cache_dir instead of crashing redis and s3

Dropping the disk_cache_dir parameter sent the key into **kwargs, which
the redis and s3 branches hand to their clients, so a config that moved
from the disk cache to redis or s3 and kept the key failed with a
TypeError. Accept the key again, ignore it, and log a warning

The unsupported type test now checks the whole supported list, and two
proxy docstrings no longer name disk as a cache backend

* fix(caching): warn about a leftover disk_cache_dir only once the cache is built

The warning ran before the type check, so a config with type disk logged
that the key was ignored and then failed on the type anyway. It now logs
only after a supported backend is built. The test also checks that a
cache without the key logs nothing
2026-10-29 18:02:31 -07:00
devin-ai-integration[bot]
a0a8ca97d9
fix(cost): finalize synchronous response cost once for header, spend log and rollups (#48993)
* fix(cost): finalize synchronous response cost before accounting callbacks

* fix(cost): keep serving and billing when cost finalization fails

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): read the finalized cost only off real logging objects

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(guardrails): follow the renamed success dispatch and deferred sync callbacks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): keep cache hits free and their headers finalized when logging is deferred

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cost): preserve completed synchronous charges after hook failures

* fix(cost): retain completed usage across cancellation and log suppression

* fix(cost): reconcile completed failures and retain cached usage metadata

* fix(cost): retain finalized request tags in failure budget updates

* fix(cost): honor disabled spend tracking for completed failures

* fix(cost): preserve pass-through accounting across cancellation

---------

Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-29 11:46:49 -07:00
Mateo Wang
14193435b8
refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages (#48510)
* refactor(anthropic): rename llms/anthropic/pass_through to llms/anthropic/messages

Move the unified /v1/messages code out of a directory called pass_through, since it is not the
passthrough feature. The inner messages sub-package becomes native, and the Vertex AI partner
model experimental_pass_through package becomes messages. The old packages stay as import shims
that warn once and alias every submodule in sys.modules, so old deep imports still resolve

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): keep Final variables out of loops in the rename shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): match the warning text in the rename shim test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): resolve old pass_through imports lazily through a meta path finder

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): mark the alias loader spec restore for the type discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): format the renamed package shim

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(anthropic): keep the rebind-ok marker on the spec restore line

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(anthropic): carry the new package search path on alias specs

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): declare the shim test locals Final

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-27 10:08:25 -07:00
devin-ai-integration[bot]
8226b2be9a
fix(otel): report cache hits at zero cost with a cache-hit flag in V1 and V2 spans (#43604)
* fix(litellm): report zero cost and cache-hit attrs on cache-served spans

* fix(litellm): stamp cache-hit marker on HiddenParams carriers and restore probe side effects

gauntlet-mini on ace57d25f verified two live gaps. TextCompletionResponse and
ResponsesAPIResponse carry _hidden_params as a HiddenParams object, so the
dict-gated marker write skipped them and the served hit still stamped the full
price. The marker write is now container-agnostic, which it already was for the
partial-embedding stamp, and a regression test exercises the carrier end to end.
_uncached_response_cost also restores billed_token_rates, zero_cost_warned, and
the response_cost_failure_debug_information key alongside cost_breakdown, so the
probe leaves no pricing state on the call it reads.

* test(caching): cover cache-hit marker on HiddenParams carriers through the caching handler

* fix(logging): keep the uncached price probe's restores to the state it can leak

* fix(otel): emit V2 LLM-call span for cache hits with explicit cache attributes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): read cache payload fields defensively without casts

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(otel): clarify pyright suppression reason on cache payload reads

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): build the cache-hit view from typing_extensions.TypedDict so import works on Python 3.10 and 3.11

* fix(otel): report cache hits at zero cost everywhere and mark misses

Cache hits now carry a zero cost breakdown and a zero hidden response cost,
the /v1/messages hit cost header follows chat and responses hits, every span
carries litellm.cache_hit, V2 hit spans drop server.address, an async hit keeps
the provider-prefixed model name, and the V2 attribute budget counts preset keys

* fix(otel): send a 0 cost header on cache hits and keep hits out of deployment TPM

Cache hits now send x-litellm-response-cost: 0.0 on chat, responses and
/v1/messages, while a fresh call whose cost is 0 still sends no header.

The async hit path stamps the same model name as main again. Stamping the
provider-prefixed name made a hit's tokens count toward the deployment's
TPM limit, so repeated cached requests hit 429s under usage-based routing.

Drop comments that restated fields and annotate new test values as Final.

* fix(otel): keep the served pricing basis on a cache hit's cost breakdown

A hit's breakdown now keeps service_tier, data_residency, vertex_location, and the discount and margin rates the served call recorded, with every amount at 0, so savings reports still read the basis. The hit header test now drives the live non-streaming header path.

* fix(otel): send zero breakdown headers on a cache hit and give hit spans a fresh call's cost keys

A hit's x-litellm-response-cost-original, -input, and -output headers now read 0.0 instead of going missing or carrying the full price. A hit's cost breakdown now comes from pricing the response as fresh, so every route exports the same keys, pricing basis, and rates with every amount at 0.

* test(integration): assert zero cost and the cache-hit flag on cache-served OTel V1 and V2 spans

* test: wait for the cache entry before the cache-hit request and build cost headers without a route_request double

* test: annotate the new cache-hit unit test locals with Final

---------

Co-authored-by: shrey kharbanda <shreshth@berri.ai>
Co-authored-by: jesus <jesus@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-17 18:25:32 +00:00
yuneng-jiang
a4e0570bea
fix(logging): keep the async request's logging payload when the sync success thread runs late (#47019)
* fix(logging): keep the async request's logging payload when sync callbacks run late

Treat the "cache" success callback as internal when deciding whether to start
the sync success thread for async requests. It has no branch in the sync
handler, so with `cache: true` that thread did nothing except rebuild and
overwrite model_call_details["standard_logging_object"].

When the sync handler does run for an async request (a real sync-only logger
is configured), it now builds its own payload and hands it only to its own
callbacks. The async handler keeps sole ownership of the shared payload, so
edits made by async logging hooks (for example the v3 rate limiter's mirrored
x-ratelimit headers that Prometheus reads) are no longer discarded. Sync
requests keep their existing behavior.

* fix(logging): keep sync cache hits logged and isolate the async-request payload build

The sync completion cache-hit path reused handle_sync_success_callbacks_for_async_calls,
so treating "cache" as internal silently dropped sync callbacks on sync cache hits. It now
submits the sync handler through its own method, as it effectively always did.

For async requests the sync handler builds its payload from a copy of the call details, so
message redaction cannot rewrite the shared standard_logging_object, and when it builds no
payload its callbacks keep the shared one instead of getting None.

* test(caching): lock sync cache-hit logging for completion, embedding and text_completion

* test(logging): keep sync hook edits off a caller-supplied payload for async requests

* fix(logging): give sync hooks a copy of the async payload when the thread builds none

* test(logging): check every legacy sync logger gets the thread's own payload

* test(logging): cover traceloop and supabase sync logging for async requests
2026-10-17 06:39:35 +00:00
devin-ai-integration[bot]
86a91f35de
fix(responses): log and bill every round of non-stream MCP auto-execute (#43667)
* fix(responses): log and bill every round of non-stream MCP auto-execute

Both model rounds run as internal sub-calls on the request's logging object, and the outer call logs the response the client received with usage, cost, and cost breakdown summed across the rounds. The internal-call gate now also covers the sync success callbacks, so an inner round no longer takes their first-wins slot

* fix(caching): let the parent call log and bill an internal sub-call's cache hit

A round of non-stream MCP auto-execute that the response cache served fired the
cache-hit callbacks on the shared logging object, so the spend log kept round 1's
function_call row and dropped the billed row for the answer. Internal sub-call cache
hits now fire no callbacks and carry a zero cost, and the summed cost breakdown leaves
out rounds served from the cache

* fix(logging): keep a parent row unique when its response came from the response cache

A RAG query or an MCP turn whose final round is served from the cache hands back the cached response's id, which already belongs to the row that cached it, so SpendLogs dropped the new row as a duplicate. Such a row now gets the cache-hit id suffix, and it is logged as a cache hit when it cost nothing.

* fix(logging): skip the zero-cost warning for a response served from the cache at no charge

A non-stream MCP auto-execute request whose rounds the response cache served is priced
at $0 before the parent marks it a cache hit, so it logged a false 'priced at $0'
warning and counted in litellm_zero_cost_requests_total. The diagnostic now treats a
response the cache served at no charge the same as a marked cache hit

* fix(responses): bill the paid first MCP round when a later step of the request fails

* fix(responses): log each MCP tool call under its own id so a failed follow-up keeps its spend row

The non-stream auto-execute path passed the request's call id to the tool call, so the tool's spend row and the request's failure row shared one request_id and the failure row was dropped on insert, leaving the paid first round unbilled.

* fix(responses): bill round 1 on a client disconnect, keep MCP tool rows in the request's session, and charge a fixed margin once per request

* fix(responses): keep bridged MCP auto-execute in one session and trace and continue its conversation

The bridged follow-up no longer chains on round 1, so a thinking model keeps its reasoning and the request stays in one session. The tool call log continues the request trace without renaming it, the tools are fetched once, and a later turn finds the MCP turn under the id the client got, skipping the tool rows.

* fix(responses): log the non-stream MCP tool listing in the request's session and trace

* fix(responses): find a bridged MCP turn by the id the client holds and log its listing in the bridged trace

* fix(responses): derive the MCP tool row id from the request's call id

The tool call's spend row now uses <call id>_<tool call id>, so it stays
linked to the request without colliding with the request's failure row.
Also tighten the internal cache-hit test to a fixed clock and typed params.

* fix(responses): bill a retried MCP attempt for its own rounds

A router retry or fallback reuses the request's logging object, so a failed attempt's stashed round 1 usage made the next attempt's success row log stale tokens. Each non-stream MCP attempt now starts without that stash.

* fix(responses): bill every failed MCP attempt's paid first round on the request failure

A retry or fallback now sets a failed attempt's stashed round 1 aside instead of dropping it, and the proxy's failure row adds it to the last attempt's own, so a request whose later attempt fails on its first call still bills the round the provider charged. A cached final round also keeps the paid first round's cost breakdown.

* fix(responses): keep MCP tool catalog and results out of the bridged next-turn replay

* fix(responses): keep MCP tool spend, client body keys, and Langfuse traces working on auto-execute

Count the MCP tools an auto-executed Responses API call ran in the tool spend rollup, accept a client body key named previous_response_client_ids without a 500, and point MCP listing and tool spans at the trace Langfuse gives the parent request, including x-litellm-session-id requests

* fix(responses): join MCP spans to the Langfuse trace when the request sends body metadata

* test(integration): cover non-stream MCP auto-execute billing, failure rows, and bridged follow-ups

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-16 17:37:33 +00:00
Apeksha_2022
9423966f5d
fix(caching): report zero response_cost on a response-cache hit (#46471)
* fix(caching): report zero response_cost on a response-cache hit

* test(caching): cover response_cost on a cache hit

* fix(caching): pass cache_hit when stamping a cached response's metadata
2026-10-14 18:45:49 -07:00
Mateo Wang
d81a5719ad
fix(cache): keep cache hits free of unset fields, and follow intentional behavior changes in CircleCI suites (#46359)
* test(integration): follow count_tokens error surfacing and decisions pricing in CircleCI suites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): restore Responses input_file PDF pricing coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): keep integration AGENTS guide unchanged

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(cache): preserve provider field presence on cache hits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): expect credential display name in response

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): align OpenInference refusal field expectations

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): expect Akto request ingestion

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(mcp): expect missing tool error wording

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(integration): expect no refusal field in cursor chat wire

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-14 08:56:05 -07:00
Yassin Kortam
d36b66e19e
feat(github_copilot): per-user GitHub OAuth connections for Copilot credentials (#45241)
* feat(github_copilot): per-user GitHub OAuth connections for Copilot credentials

Related to LIT-9306

* fix(ui): make per-user GitHub Copilot OAuth credential-only in Add Model

* fix(proxy): render LiteLLM_UserProviderCredentials prisma spans

* style(ui): prettier-format add_model tests

* fix(lint): annotate connection endpoints and narrow per-user credential excepts

* fix(lint): satisfy type-discipline gate for per-user copilot code

* fix(types): clear basedpyright and strict-lint gate regressions

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): keep runtime guards on untyped responses input

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style: ruff format per-user Copilot files

* fix(lint): keep type-discipline suppressions on flagged lines

* fix(github_copilot): resolve team and deployment-id per-user routes, share token cache invalidation

* test(github_copilot): scope response-cache mutation via monkeypatch, assert route allowlist

* fix(github_copilot): fall back to the database on Redis errors and skip worker-local token caching

* fix(lint): suppress mutable return annotation after base merge

* style: shorten rebind-ok reason so the dispatch line stays formatted

* fix(github_copilot): harden per-user token cache, overrides, fallbacks, headers and device flow

* fix(lint): suppress MutableMapping param on session auth pinning

* test(github_copilot): narrow pytest.raises to HTTPException for service-key override

* fix(github_copilot): fail closed on revoke errors, strip caller auth headers, widen fallback discovery

* fix(github_copilot): verify revoke tombstone, bound fallback discovery, cover aliased fallbacks

* fix(github_copilot): make fallback discovery a router superset and guard credential override at deployment

* style(router): drop explanatory comment on per-user credential guard

* fix(github_copilot): gate fallback discovery on per-user credentials and bound aggregate targets

* fix(github_copilot): count every fallback mapping toward the discovery limit

* fix(github_copilot): suppress banned typing cast imports with per-line reasons

* test(github_copilot): cover per-user 401 cooldown skip through the router failure callback

* test(github_copilot): cover per-user 401 cooldown skip through router and fallback paths

* fix(github_copilot): cover auto-router discovery, per-user fallback cooldown and connect cache race

* fix(github_copilot): cover every strategy router kind and verify the connect cache write

* fix merge conflict marker remnant in credential endpoint tests

* fix(github_copilot): do not purge user connections on label-only credential updates

* fix(github_copilot): read user provider credentials from the writer engine

* test(github_copilot): move per-user auth type form test to the integration tier

* test(github_copilot): annotate new credential regression test locals as Final

* fix(github_copilot): type per-user OAuth JSON and DB boundaries

* fix(github_copilot): validate per-user OAuth boundaries and restore truncated comments

* style(github_copilot): complete cut-off suppression reasons on new casts

* refactor(github_copilot): annotate new boundary-typing locals as Final

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 15:19:07 -07:00
Yassin Kortam
fbf4af301c
feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange (#45158)
* feat(microsoft_365_copilot): add Microsoft 365 Copilot chat provider with OAuth token exchange

* fix(ui): create a credential for credential-only auth types in Add Model

* fix(microsoft_365_copilot): accept max_tokens and list the chat model

* fix(microsoft_365_copilot): register provider model set

* fix(ui): hide other auth types' fields when credential-only types are filtered

* fix(health): do not inherit stored auth settings when a connection test brings its own api_key

* docs(health): restore connection-test inheritance docstrings

* fix(lint): suppress justified provider boundary casts

* fix(lint): satisfy M365 type-discipline gate

* fix(types): eliminate type-check gate regressions

* style: shorten suppression reasons so ruff format leaves them on one line

* refactor(types): drop shared-file type widening unrelated to the Copilot provider

* fix(health): pass caller headers as secret_fields to connection-test probes

* fix(types): type connection-test probe and token counter locally

* fix(lint): drop cast import from connection-test header getter

* chore: remove explanatory comments flagged by review

* fix(proxy): gate OAuth client credentials behind proxy admin

* fix(router): skip cooldown on caller-scoped OAuth auth failures

* fix(router): skip fallback cooldown on caller-scoped OAuth auth failures

* fix(router): type caller-scoped cooldown lookups

* fix(ci): include Microsoft 365 Copilot tests in unit shard

* fix(proxy): keep saved endpoint when test_connection retests a deployment by id

* fix(m365): collapse doubled Graph replies

* fix(m365): preserve trailing system messages

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-09 15:19:07 -07:00
devin-ai-integration[bot]
56bafc29cf
feat(gemini): accept file content blocks with video_metadata on multimodal embeddings (#45305)
* feat(gemini): accept file content blocks with video_metadata on multimodal embeddings

* test(gemini): annotate the embedding mock helper and keep the type comment short

* fix(gemini-embeddings): keep file blocks through a partial embedding cache hit and validate video_metadata strictly

* test(caching): type the partial-hit cache test and pin its clock

* fix(gemini-embeddings): accept the Files API URI /v1/files returns as a file block's file_id

* fix(gemini-embeddings): 400 on bad input shapes, drop unknown block keys under drop_params, cache multi-block inputs

A bare object `input` answered 500 from both the caching handler and the
transformation; both now answer 400. An unresolved `files/` reference on
vertex_ai/ answered a ValueError 500; it now answers 400 naming the gemini/
provider. An empty `format` passed through to the provider; it now answers 400
naming file.format. Unknown block keys (`detail`, an unknown video_metadata
key, a top-level block key) are dropped under drop_params, global or
per-request, and still answer 400 without it. The embedding cache counted
file blocks against `max_messages`, so a request with 5 or more blocks was
never cached; file blocks no longer count.

* fix(caching): answer 400 for an object embedding input on the cache lookup

* test(gemini-embeddings): annotate the new embedding tests with return types and Final

* test(gemini-embeddings): annotate the transformation and files batch tests with return types and Final

* test(gemini-embeddings): audit file content blocks on the wire, in the cache, in batch uploads and under chaos

Integration cells for gemini/ and vertex_ai/ embedding file blocks with video_metadata: the batchEmbedContents and embedContent wire shapes, format overrides, nested and repeated inputs, every malformed block answering 400 before any provider call, drop_params at the request, deployment and YAML levels, the OpenAI SDK sync and async clients, Redis cache fills, hits, partial hits and the metadata in the key with zero-spend hit rows, chat, responses and messages controls, the vertex_ai/ batch JSONL upload, a mixed fast/slow/malformed/dropped burst and a worker kill. The wire peer now reads chunked request bodies, which the streamed GCS media upload sends

* test(integration): treat an upload the client abandons before its terminating chunk as a disconnect, never a stored request

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-08 13:20:03 -07:00
devin-ai-integration[bot]
cb138ba92f
refactor(types): replace Any with proven types in 16 files (#45029)
* refactor(types): replace Any with proven types in 16 files

* fix(types): import TypedDict from typing_extensions for pydantic on 3.10

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-07 04:31:51 -07:00
devin-ai-integration[bot]
d364d5e7ca
refactor: expose public hidden_params accessors (#44668)
* refactor: expose public hidden_params accessors

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: keep hidden params writes on duck-typed responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor: move hidden params helpers to core utils

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve hidden params in duck-typed cost responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover hidden params accessors across response types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve hidden params for dynamic responses

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: expose logs through backend component

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* Revert "fix: expose logs through backend component"

This reverts commit 621e5ff25e.

* test: cover dict hidden params helper path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve hidden params storage on item writes

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover hidden params accessors for OpenAI types

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: cover hidden params helper and response paths

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: complete hidden params accessor migration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): tighten hidden parameter validation

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): read HiddenParams model storage through get_hidden_params

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): transfer hidden params storage instead of the mapping view

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(litellm): expose only set HiddenParams keys through the mapping view

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* perf(litellm): read HiddenParams view keys without dumping the model

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(litellm): assert HiddenParams extras through model_extra

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(router): copy streaming hidden params into a plain dict

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-07 00:01:33 -07:00
devin-ai-integration[bot]
c3a23fe499
refactor: expose core private helpers under public names (#44871)
* refactor: expose core private symbols with compatibility aliases

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* types: narrow core migration diagnostics

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve runtime behavior in core symbol migration

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve core private usage migration behavior

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: preserve private value rebinding compatibility

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix: restore optional imports and cover public helpers

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: exempt router property from call coverage

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test: align recursive detector ignore names

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: mateo <mateo@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-07 00:05:27 +00:00
devin-ai-integration[bot]
abd3af422d
fix(caching): never store or serve a chat completion with no choices (#44709)
* fix(caching): never store or serve a chat completion with no choices

A provider response with empty choices was written to the response cache and served on every identical request until the TTL ended, with no provider call in between. The cache now skips storing such a response and treats an already stored one as a miss, so the next request goes back to the provider and its answer replaces the entry.

* fix(caching): skip responses with no output on the Responses API and Anthropic Messages too

* fix(caching): skip streams and stored entries that carry no output

A chat or text completion stream whose chunks carried no choice is closed
by the stream wrapper with one empty choice of its own, so the assembled
response passed the choices check and was cached. The assembled stream is
now judged on its content: a stream with no text, tool call, or other
output in any choice is never stored, on the async and sync writers alike.
The Responses API stream writer and the Anthropic Messages stream writer
apply the same no-output check before storing.

A stored entry with no output read through the worker memory tier is now
evicted from that tier on the miss, so the next read reaches Redis where
the refill lands; the text completion and messages writers only write to
Redis, and the memory copy otherwise kept missing until its own TTL.

* test(integration): response cache cells for answers without output

Deterministic cells for the response cache on every unified endpoint,
streamed and not, through the OpenAI and Anthropic SDKs and raw httpx,
plus the sync SDK paths, stale entries, malformed answers, per-request
TTLs, cache delete, and chaos (Redis stopped or paused mid burst, a
worker killed, in-memory cache mode). The scripted upstream counts only
POSTs as deployment calls, since the proxy's boot-time GET /v1/models
discovery of a config deployment is not one.

* test(caching): pin the stored entry timestamp in the worker-copy test

* test(integration): drop the restating comments from the chaos cells

* test(integration): close the breaker on the first call after the Redis restart

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-06 20:07:30 +00:00
devin-ai-integration[bot]
c2c0bb583e
perf(types): defer pydantic schema builds via shared LiteLLMBaseModel (#44720)
* perf(types): defer pydantic schema builds via shared LiteLLMBaseModel

Add LiteLLMBaseModel with defer_build driven by DEFER_PYDANTIC_BUILD (default true) and move litellm and enterprise pydantic models onto it

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(types): build deferred models created by a parent validator; keep lens worker models litellm-free

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(types): link pydantic issue on deferred-build rebuild hook

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: nate <nate@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-05 22:10:08 -07:00
devin-ai-integration[bot]
564d236985
fix(otel): nest cache spans under their operation and name service spans by purpose (#44150)
Response cache reads and writes open cache.get llm_response and cache.set llm_response phase spans with their Redis spans nested underneath, on the Python path and on the native Rust path, and deployment selection runs inside a route {model_group} phase so the cooldown, usage and model-id reads the router issues nest under it before chat {model}. The autorouter classifier call nests under that route phase as well and carries its typed internal origin on litellm.request.purpose, so it is told apart from the provider attempt. Service spans are named {service}.{verb} {target} from a low-cardinality key family the producer declares (llm_response, auth_objects, spend_counters, router_cooldowns, claude_code_session_router_binding, rate_limits, pod_lock, budget_reset, ...) instead of the raw method or a per-request pipeline length; a pipeline flush is targeted by the one family its ops share or by mixed with the sorted families on litellm.redis.families, a batch op keeps the family it was declared under whichever pipeline or standalone read settles it, and the ambient family labels Redis spans only, never the DB write-back a task spawned inside that context performs later. The raw method stays on litellm.service.call_type and on the Prometheus and Datadog labels. Caller attribution is carried across asyncio task boundaries on a ContextVar so forwarder-only chains no longer surface, the raw cache key is dropped from Redis span metadata, pipeline op counts land as an integer attribute, every call_type the Redis cache layer emits maps to a verb, and a scan over litellm/ and enterprise/ fails when a Redis producer, batch reservation included, declares no key family.

A V2 logger built for a key or team logging entry while the operator's V2 logger is already registered keeps only the exporters its own preset contributed, whether or not the operator holds credentials for that backend, so every chat span no longer reaches the operator's collector twice. A span the success callback has to open itself, with no pre-call carrier, starts at the provider handoff (api_call_start_time) instead of the logging object's creation.

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-10-03 09:20:27 -07:00
yuneng-jiang
bba85f0b6c
chore(lint): remove the LIT002 mutable-construction rule (#43971)
* chore(lint): remove the LIT002 mutable-construction rule

Drop LIT002 from scripts/check_type_discipline.py along with its helpers,
its budget entry, its unit tests, and the AGENTS.md and gate docstring
mentions. `# mutable-ok` now only suppresses LIT001, so the markers that
only existed to silence LIT002 became LIT013 stale suppressions and are
removed. The files whose layout depended on those trailing comments are
reformatted with ruff format.

Every other LIT rule count is unchanged and the ASTs of all touched
litellm/ files match main apart from one docstring.

* chore(lint): keep the leftover mutable-ok markers for a follow-up

Restore the ~1.4k `# mutable-ok` markers stripped in the previous commit
so this PR only touches the checker, its tests, the budget, and docs.
Those markers no longer suppress anything, so `# mutable-ok` is exempt
from LIT013 until a follow-up strips them.

* Revert "chore(lint): keep the leftover mutable-ok markers for a follow-up"

This reverts commit c35bc0b84e.

---------

Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
2026-10-01 12:24:02 -07:00
devin-ai-integration[bot]
24a7e89738
perf(responses): run aresponses through the async wrapper so the cache is read once (#43769)
* perf(responses): run aresponses through the async wrapper so the cache is read once

aresponses() sets kwargs["aresponses"] = True and runs the decorated sync
responses() on an executor, but _is_async_request() did not recognise that
flag, so the sync wrapper did a second cache lookup on the executor thread
with a differently ordered cache-key input. Every /v1/responses request paid
two cache GETs against two different keys. Recognising aresponses in
_is_async_request() leaves the async wrapper as the only cache reader and
writer for the async path, one GET per request, same key on read and write

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): let a responses cache entry cover aresponses so responses-only configs keep caching

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-29 15:09:22 -07:00
devin-ai-integration[bot]
69ad004015
refactor(anthropic): rename experimental_pass_through to pass_through (#43329)
* refactor(anthropic): rename experimental_pass_through to pass_through

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(anthropic): point compact patch targets at renamed pass_through path

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: kerry <kerry@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 13:00:50 -07:00
devin-ai-integration[bot]
d18fcb09d6
fix(otel): detach post-response service spans by request phase, name redis spans by operation (#43237)
* fix(otel): detach post-response service spans by request phase, name redis spans by operation

Service spans logged from the post-response phase (success callbacks, the response-cache write) now root their own trace linked to the request span even while the server span is still recording, instead of only when they happen to end after it. Redis service spans are named `redis <operation>`; the litellm call chain that issued them moves to the `litellm.service.caller` attribute via a typed `ServiceLoggerPayload.caller` field.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): keep the service caller on failure and legacy spans, test the production phase dispatch sites

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(otel): mark anthropic messages stream cache write as post-response phase

The /v1/messages streaming cache writer awaits async_add_cache inline
instead of going through create_cache_write_task, so its redis span
stayed parented under the request trace. Wrap the write in
post_response_phase so it detaches like the chat completions write.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(anthropic): write the Messages stream cache in a background task after handoff

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-26 10:10:12 -07:00
devin-ai-integration[bot]
320b40645c
fix(caching): stamp provider on sync cache-hit logs so responses spend logs record provider (#42830)
* fix(caching): stamp provider on sync cache-hit logs so responses spend logs record provider

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): tighten sync cache-hit provider regression docstring

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): drop redundant docstring on sync cache-hit provider test

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-23 20:49:23 -05:00
devin-ai-integration[bot]
a173657dfb
fix(caching): keep embedding cache hits aligned with request inputs (#42571)
* fix(caching): keep embedding cache hits aligned with request inputs

Partial hits now send only the uncached inputs to the provider and merge
fresh vectors back into their original positions. Responses whose item
count differs from the input count (one input scoring many documents)
are no longer written to the per-input cache, since a later hit would
return a single item.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): drop mutable collection builds flagged by the type discipline gate

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): bypass embedding cache entries written before the per input cardinality check

Embedding cache entries now carry format_version and readers treat entries without it as
misses, so entries that only hold the first row of a multi row response are refetched instead
of served until their TTL expires. The provider call also receives a copy of the request kwargs
with the uncached inputs rather than mutating the caller's mapping

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): assert a partial embedding cache hit becomes a full hit on repeat

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(caching): await pending embedding cache writes before asserting on cache hits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): validate cached embeddings without mutating responses or request kwargs

Validate cache rows through a frozen pydantic model so import does not depend on
TypeAdapter support for ReadOnly TypedDicts, accept string embeddings, build the
merged partial hit response instead of mutating the cached one, and hand the
provider request mapping to post call hooks

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(caching): keep cache_hit and response_ms on merged partial embedding hits

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-22 17:14:44 -05:00
mateo-berri
c62c187054 Merge remote-tracking branch 'origin/main' into litellm_decrease_anys_opus5_r5
# Conflicts:
#	litellm/integrations/otel/model/metadata.py
#	litellm/litellm_core_utils/coroutine_checker.py
#	litellm/litellm_core_utils/exception_mapping_utils.py
#	litellm/litellm_core_utils/llm_response_utils/response_metadata.py
#	litellm/llms/bedrock_mantle/responses/transformation.py
#	litellm/llms/openai/chat/guardrail_translation/handler.py
#	litellm/proxy/common_utils/reset_budget_job.py
#	litellm/proxy/db/db_spend_update_writer.py
#	litellm/proxy/guardrails/guardrail_hooks/llm_as_a_judge/__init__.py
#	litellm/proxy/openai_files_endpoints/storage_backend_service.py
#	litellm/proxy/pass_through_endpoints/llm_provider_handlers/vertex_passthrough_logging_handler.py
#	litellm/realtime_api/main.py
2026-09-21 12:57:38 -07:00
yucheng
496c2a5513 fix(caching): replay agentic loop follow-up cache hits as plain objects
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 08:35:13 +00:00
yucheng
ce45d6a09d style: drop explanatory docstrings from converted-stream helpers and tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 07:51:19 +00:00
yucheng
621db91d90 fix(caching): defer cache-hit callbacks by replayed result type, not request flags
A converted-stream request whose cache entry is a plain (non-stream) object is
replayed as that plain object, so nothing later fires the success callbacks.
Decide deferral from the replayed result's type instead of the request kwargs.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 07:51:19 +00:00
yucheng
8b86362703 fix(caching): replay cache hits for converted streams as streams
A deployment hook (Headroom, code interpreter, web search) can downgrade
kwargs["stream"] to False while the caller still expects to iterate the
result. The cache handler keyed stream replay and callback deferral off
the raw flag, so a cache hit returned a plain object to a caller that
iterates, and the Responses iterator never persisted the converted
stream in the first place. Key both off the conversion marker as well

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-15 07:51:19 +00:00
mateo-berri
da7d5fe128 refactor: replace Any with precise types across 54 modules
Narrow or remove reportAny / reportExplicitAny sites in provider
transformations, caching, guardrails, proxy endpoints and enterprise
batch-cost polling. Public parameters widen to Mapping/Sequence rather
than dict/list so no caller signature breaks, and runtime behavior is
unchanged.
2026-09-14 11:13:39 +00:00
devin-ai-integration[bot]
ae01882535
feat(proxy): offload spend tracking to a pod-local collector sidecar (#40545)
* feat(proxy): offload spend tracking to a pod-local spend worker sidecar

py-spy on the gateway showed the post-response _PROXY_track_cost_callback,
spend-log and DBSpendUpdateWriter work running on the inference workers'
event loop, so a DB or Redis stall backed up the request path.

When LITELLM_SPEND_WORKER_ENABLED=true, _ProxyDBLogger serializes one compact
typed SpendEvent per success and hands it to a SpendEventProducer that ships
it over a unix socket (default) or loopback-only TCP to a sidecar started as
`python -m gateway.spend_worker`. The sidecar runs the unchanged
_ProxyDBLogger pipeline against the pod's PgBouncer (pooled_database_url).
When the sidecar is unreachable, the buffer is full, or the gateway shuts
down with events still queued or in flight, the producer applies
LITELLM_SPEND_WORKER_ON_UNAVAILABLE (fallback in-process, or drop). The
sidecar half-closes producers on SIGTERM and drains, the producer treats
EOF as unavailable, and the gateway flushes buffered spend counters on
shutdown. The sidecar honors LITELLM_LOG so its writes are visible in its
own process log.

Helm: both charts gain an opt-in spend-worker sidecar container sharing an
emptyDir socket dir, and the componentized chart's HPA uses a
ContainerResource CPU metric scoped to the gateway container so sidecar
CPU does not drive inference scaling.

* feat(terraform): opt-in spend-worker sidecar for the AWS and GCP gateway stacks

Adds spend_worker_* inputs to both modules. On ECS Fargate the sidecar is a second, non-essential container in the gateway task; on Cloud Run it is a second container in the gateway service. Both listen on loopback TCP, share the gateway's DB/Redis/secret env, and set LITELLM_JOB_ROLE=spend_worker. Disabled by default. Plan-only tests cover both, and the terraform CI workflow now runs the gcp module too

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* test(proxy): retrieve a completed batch in the in-process spend path test

The base now defers cost tracking for batches that are still in flight, so an in_progress batch never reaches update_database

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* refactor(proxy): rename the spend worker sidecar to collector

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): run the collector from the installed litellm package and finish in-flight fallbacks on shutdown

The sidecar command becomes python -m litellm.proxy.collector so the classic image, whose runtime
stage copies only the installed package, can run it. The module now assembles DATABASE_URL and the
pod-local pgbouncer URL itself, replacing gateway/collector.py

The componentized collector sidecar inherits gateway.volumeMounts so custom CA mounts reach it.
SpendEventProducer shields an in-progress fallback from the writer task cancellation so close()
no longer loses an event already handed to the in-process pipeline

Helpers used across modules (address_argument, should_store_prompts_and_responses_in_spend_logs,
flush_spend_counters_on_shutdown) become public so the change adds no reportPrivateUsage errors

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* ci(terraform): drop the gcp job duplicated by the aws/gcp matrix

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(collector): keep metrics env off the classic sidecar and reject shared loopback ports

The classic chart no longer hands PROMETHEUS_METRICS_PORT and the billing metrics env to the collector container, and gives it the same /.npm scratch mount as the proxy on a read-only root. AWS and GCP now refuse a plan where the spend collector and the metrics sidecar bind the same loopback port. A regression test drives a sidecar crash mid-stream on asyncio and uvloop and checks no event is billed by both the sidecar and the in-process fallback; the producer docstring spells out why a failed drain() cannot double count

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* style(proxy): format pooled_database_url after the pgbouncer rebase

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): keep the cache-hit preset key and survive dead producers on collector drain

Cache hits updated the logging object after the early return, so the offloaded spend event carried
preset_cache_key=None and the collector re-hashed reconstructed kwargs. Also guard write_eof() against
producer transports uvloop already closed so one dead connection cannot abort the drain

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(terraform): keep the gcp collector port off the metrics sidecar health port

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): collector connects to Postgres directly under IAM or Entra token auth

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* fix(proxy): mark the collector's DATABASE_URL as pooled when it uses the pod's pgbouncer

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

---------

Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-09-10 17:14:13 -07:00
mateo-berri
362fb4cffe refactor(typing): replace Any with proven types in 42 more backend files 2026-09-02 15:35:01 +00:00
mateo-berri
cbb50bb37e fix(responses): flush streaming cache write cancelled at event loop shutdown 2026-08-26 12:05:11 -07:00
mateo-berri
1a696de40c fix(caching): flush async cache writes cancelled at event loop shutdown 2026-08-26 11:47:10 -07:00
mateo-berri
11552dbafc chore(typing): drop 1.3k basedpyright errors across 42 Any hotspot files
Replace implicit and explicit Any with real types across the highest-density
reportAny/reportExplicitAny files: module-private TypedDicts for dict payloads,
Protocols for duck-typed collaborators, and existing litellm/types models where
they already describe the shape

No new cast(), no # type: ignore, no # pyright: ignore, no # noqa, and no
new suppressions. Diagnostics that could not be resolved without one were left in
place rather than hidden
2026-08-19 01:12:41 +00:00
shivam
a09551a71a merge litellm_internal_staging into litellm_anthropic_messages_response_cache
Some checks failed
LiteLLM Rust / rustfmt, clippy, test (push) Has been cancelled
Terraform Provider / gofmt, vet, build, test (push) Has been cancelled
Terraform Provider / Provider endpoints vs proxy OpenAPI schema (push) Has been cancelled
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-08-11 20:48:41 +00:00
mateo-berri
338e411103 chore(lint): strip inert type: ignore comments and zero LIT009, LIT010, LIT011 headroom 2026-08-05 02:37:24 -07:00
mateo-berri
2708620d6a feat(lint): enforce Final on locals and freeze function parameters (LIT010, LIT011) 2026-08-04 12:54:39 -07:00
Classic298
c9887a1f94
perf: build log messages lazily so filtered-out log records cost nothing (#35703) 2026-08-04 04:34:52 +00:00
mateo-berri
b604e2b20c refactor(lint): apply every safe ruff autofix and zero 28 strict-rule budgets
About 35,000 fixes ruff marks safe across 32 rules (UP006/UP045/UP007
modern annotations, UP032 f-strings, SIM114/SIM118, RET501, and
friends), removal of the 1,296 typing imports the rewrite orphaned, and
hand fixes for what the fixers could not see: five star-import
freeloaders of typing names, two F823 late-import annotations, the
/get/config/list introspection crash on types.UnionType, redundant
function-local RoleMappings imports in ui_sso.py that shadowed the
module-level name once the annotation lost its quotes, and one FURB168
tautology.

B009/B010/PIE804/RUF019 are excluded on purpose: their safe fixes
rewrite getattr/setattr/**-splat/key-in-dict escape hatches into forms
basedpyright then rejects (283 new errors measured), so their budgets
stay at base values.

ruff-strict-budget.json drops by 39,579 this commit (39,968 across the
branch) with 28 rules at an actual 0 and 9 more sharply down.
type-discipline-budget.json ratchets LIT002/LIT006/LIT009 down; LIT001
moves to the now-honest total: the checker matches the spelling `set`
but not the alias `Set`, so the 160 typing.Set annotations rewritten to
set[...] were always mutable-set annotations and only now count.
2026-08-01 15:43:29 -07:00
mateo-berri
397e8e4918 refactor(imports): move collections.abc names out of typing and ratchet the lint budgets 2026-08-01 13:17:42 -07:00
milan
15c7d850e5 fix(caching): stamp provider on embedding cache-hit logs so spend logs record provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-30 22:11:01 +00:00
shivam
d2a5de2e04 refactor(caching): tighten anthropic messages cache types and drop comments
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-25 00:15:55 +00:00
shivam
ab997e04eb fix(caching): cache anthropic /v1/messages responses, including streaming
anthropic_messages was missing from the cache's supported call types, so every /v1/messages request went to the provider. Adding it alone is not enough: the cache key is built from the OpenAI-ish param set, which has no system, top_k or stop_sequences, so two requests differing only by system prompt shared an entry and the second got the first one's answer. The Anthropic Messages request shape now feeds the key set as well.

Streaming responses return to the caller before async_set_cache runs, so they are teed on the way out and the SSE events are stored verbatim once the stream reaches message_stop without a provider error. A hit replays those bytes and logs the request as a cache hit with zero cost.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-07-25 00:09:21 +00:00
Yassin Kortam
06e8013e6c
fix(logging): stop pinning large request payloads past request end (#33455)
Three process-lifetime retention points kept full request payloads
(messages included) alive after the request finished. Under bursts of
large-token traffic (~73K tokens/request mean) this presented as
stepwise RSS growth that never returned to baseline, ending in OOM:

1. Logging.pre_call/post_call stored their entire locals() (messages,
   the Logging object, complete_input_dict) in the module-level
   litellm.error_logs dict, pinning the most recent request's payload
   per worker forever. Nothing reads that dict; the writes are removed.

2. LLMCachingHandler.request_kwargs kept litellm_logging_obj inside the
   stored kwargs while the handler itself hangs off
   logging_obj._llm_caching_handler, closing a reference cycle
   (Logging -> LLMCachingHandler -> kwargs -> Logging). Cyclic payloads
   are only reclaimed by generational GC, so megabytes of dead request
   data lingered until a rare gen-2 pass, and the transient copies
   fragment the allocator into a permanent RSS high-water mark. The
   handler now drops litellm_logging_obj from its stored kwargs; the
   caching layer never reads it.

3. The router stored every request's kwargs in the ITPM/OTPM contextvar
   even when no deployment configures itpm/otpm. Pooled resources
   created mid-request (e.g. redis connections) capture the asyncio
   context, extending that pin far past the request. The slot is now
   populated only for deployments with io token limits and overwritten
   with None otherwise.

Live-proxy verification (bursts of 30 x ~300KB requests, PII guardrail
+ prometheus + redis cache): unfixed grows 16-29MB per burst without
release; fixed grows under 1MB per burst after warmup and flattens.

Resolves LIT-4434
2026-07-15 15:28:37 -07:00
Mateo Wang
48b5a5a0cc
style: unify ruff format width on 120 (#31518)
The repo linted at 120 (E501, isort) but ran ruff format at 88 via a
--line-length 88 override in the Makefile and CI, leaving the formatter
and the linter disagreeing on wrap width. Drop the override so ruff.toml's
line-length = 120 is the single source of truth and reformat the tree to
match.
2026-06-27 12:39:29 -07:00
Mateo Wang
17bfd415ae
chore: migrate Python formatter from black to ruff format (#31317) 2026-06-25 11:27:43 -07:00
ryan-crabbe-berri
5a62806fdc
chore(lint): remove PLR0915 too-many-statements ruff rule (#30574)
Drops PLR0915 from ruff's extend-select along with its per-file-ignores,
and strips the now-unused `# noqa: PLR0915` directives across the codebase
(RUF100 would otherwise flag them as unused). The C901 suppression that
shared a directive with PLR0915 in streaming_handler.py is preserved.
2026-06-16 16:52:49 -07:00
michelligabriele
2fe9feda71
fix(caching): restore stored prompt_tokens on embedding cache hits instead of recomputing (#30046) 2026-06-10 15:49:20 +05:30
Sameer Kankute
477b63c5ea
fix(caching): replay openai/responses bridge cache hits as chat streams (#28158)
* fix(caching): replay openai/responses bridge cache hits as chat streams

When chat completions route through openai/responses, cached ModelResponse
payloads under aresponses keys were deserialized as ResponsesAPIResponse
(500) or re-translated as responses events (empty streaming deltas). Deserialize
chat-shaped cache entries as acompletion and bypass the responses stream iterator
for cached CustomStreamWrapper replay.

Co-authored-by: Cursor <cursoragent@cursor.com>

* fix(caching): map responses bridge call_type for sync vs async stream replay

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix: handle ModelResponse cache return in responses bridge and drop dead acompletion check

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(caching): detect chat cache hits via object field before choices fallback

Prefer chat.completion object type over the broad choices-key heuristic so
Responses API cached payloads are not misclassified if their schema changes.

Co-authored-by: Cursor <cursoragent@cursor.com>

* test(caching): cover responses bridge cache-hit paths in CI-tracked test suite

The new bridge cache replay logic in caching_handler.py and the
preformatted-stream guard in litellm_responses_transformation/handler.py
were exercised only by tests under tests/local_testing/, which the
responses-caching-types and misc shards do not run. Codecov flagged the
patch as 29.72% covered.

Add equivalent unit tests under tests/test_litellm/ so the responses,
caching, types, and misc shards execute them and ship their coverage
data to Codecov:

- _is_chat_completion_cached_dict happy/sad paths
- aresponses streaming bridge cache hit -> CustomStreamWrapper
- responses non-streaming bridge cache hit -> ModelResponse
- legacy ResponsesAPIResponse stream + non-stream replay
- _is_preformatted_cached_chat_stream true/false
- completion/acompletion early return on cached ModelResponse
- completion/acompletion skip rewrap on preformatted cached stream

* fix: add negative guard on object field in _is_chat_completion_cached_dict

Co-authored-by: Yassin Kortam <yassin@berri.ai>

* fix(vcr): treat corrupt cassette payloads as cache miss

* test: bump EOL'd NVIDIA rerank and OpenAI realtime models in CI

The NVIDIA hosted rerank endpoint for nvidia/llama-3_2-nv-rerankqa-1b-v2
reached end-of-life on 2026-05-18 and now returns HTTP 410 Gone, breaking
TestNvidiaNim::test_basic_rerank. Switch to nvidia/nv-rerankqa-mistral-4b-v3,
which is still hosted on the NVIDIA API catalog and is already listed in
model_prices_and_context_window.json.

OpenAI also retired the gpt-4o-realtime-preview-2024-12-17 model used by
test_realtime_guardrails_openai (now returns model_not_found). Switch the
realtime test URL to the GA gpt-realtime alias.

Unrelated to the responses-bridge cache fix in this PR, but committing
here to unblock CI per maintainer guidance.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(realtime): switch retired gpt-4o-realtime-preview to gpt-realtime

OpenAI removed gpt-4o-realtime-preview and all its date snapshots on
2026-05-18 (every variant now returns model_not_found), breaking the
live-WebSocket OpenAI realtime tests in CI:

  - test_openai_realtime_direct_call_no_intent
  - test_openai_realtime_direct_call_with_intent
  - TestOpenAIRealtime.test_realtime_connection
  - TestOpenAIRealtime.test_realtime_with_query_params

Point each of those to the current GA alias gpt-realtime (verified live).
Pure unit/mock tests that just assert the string value (e.g. in
test_realtime_query_params_construction and the
test_realtime_query_params_use_normalized_model_name mock) are left
alone since they do not depend on model availability.

Also relax the AI-response assertion in
test_text_message_blocked_by_guardrail_no_ai_response: gpt-realtime
occasionally produces a polite refusal ("I'm sorry, but I can't say
that") when the cancel arrives after the model has already started
generating, which is the expected outcome (no real AI content) but does
not contain the words 'blocked' or 'guardrail'. The primary guardrail
behaviour (guardrail_violation error event + transcript_delta block
message) is still asserted unchanged.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(nvidia_nim): mock rerank live API instead of hitting EOL'd endpoint

NVIDIA reached end-of-life for the hosted nvidia/llama-3.2-nv-rerankqa-1b-v2
rerank API on 2026-05-18 (returns HTTP 410 Gone), and the proposed
replacement nv-rerankqa-mistral-4b-v3 returns HTTP 404 for the CI account,
breaking TestNvidiaNim::test_basic_rerank.

Override test_basic_rerank to mock the HTTP transport (same pattern as
test_nvidia_nim_rerank_ranking_endpoint above) so the request/response
transformation and cost calculation stay covered without depending on
NVIDIA's hosted catalog rotation. The model identifier reverts to the
original llama-3.2-nv-rerankqa-1b-v2 since the request never leaves
the test process.

---------

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
2026-05-18 16:27:06 -07:00
Sameer Kankute
e0398cade7
fix(caching): defer streaming cache-hit callbacks for all stream=True
Success handlers already run when CustomStreamWrapper or
CachedResponsesAPIStreamingIterator finishes replay. Logging at
cache-hit time for acompletion/completion streaming duplicated spend
and callbacks. Align tests with deferred behavior.

Made-with: Cursor
2026-05-01 17:03:32 +05:30