The allowlist recorded eight files in tests/local_testing, 118 tests, that
every job globbing that directory then deselects: local_testing_part1 and
part2 carry `-k "... and not caching and not cache"`, and the other three keep
one unrelated keyword each. They counted as covered while running nowhere.
Five of the eight need nothing. Measured with no provider credentials and no
Redis: test_cache_preset_key, test_caching_handler, test_prompt_caching,
test_responses_stream_cache_keys and test_unit_test_caching pass, 45 tests
together, and they now run as a caching-local shard. The other three stay
allowlisted with what they actually need recorded rather than a question:
test_caching wants Redis and a provider key for 37 of its 65, disk-cache wants
OPENAI_API_KEY for 2 of 4, gcs-cache wants GCS credentials for all 4.
Taking them off the allowlist exposed a gap in the slice guard itself: it
reasoned only about CircleCI `-k` expressions, so a file every slice drops read
as unrun even when a workflow names it outright. It now credits workflow
test-paths the way the census already does, and only workflows, so a tree only
CircleCI globs is still reported.
* feat(ci): gate patching of SDK internals in tests as TQ008
TQ002 catches the narrowest symptom of the suite's dominant mocking idiom,
patch X then assert only that X was called. The idiom itself is wider: tests
reach for litellm's own functions instead of faking the wire, so they pin how
the code is wired rather than what it does, and a test that patches internals
but makes weak real assertions trips nothing today.
TQ008 counts patch targets rooted at `litellm`, both the dotted string form and
the attribute chain handed to patch.object, and ratchets like every other rule.
Mocking anything outside the SDK is untouched: respx, httpx transports and
third-party clients do not trip it, which is the point, since those are the
patterns this is meant to move the suite toward.
Seeded at 9,643, in line with the ~9.4k patch sites an independent grep found
in the mirror. The burn-down horizon is long; the value here is stopping the
flow rather than clearing the stock.
Five existing rule tests patched `litellm.completion` incidentally and now
report TQ008 alongside what they were pinning. Their expected values are
updated to the accurate pair rather than loosened, so they keep failing on a
regression in either rule.
* test: add TQ008 to the shipped-budget rule canary
* fix(ci): resolve imported SDK names in TQ008
patch.object(handler.OpenAIChatCompletion, ...) after a from-import reaches the
same internal as the dotted string form, but the rule only saw the bare local
name and let it through. Import bindings are now resolved to the path they
stand for, so the aliased, renamed and from-imported forms all read alike and
the reported target is the real one.
That is 1,496 patches the ratchet could not see, so the TQ008 limit moves from
9,643 to 11,139. Third-party names and locals with no SDK import behind them
stay unflagged.
* perf(ci): fan the budget checkers out across cores
check_type_discipline.py and check_test_quality.py each walk a few thousand
files and parse every one, single-threaded. In the lint job those two steps
measure 2.3 and 1.6 minutes, second and third behind dependency install, and
lint is the slowest required check on 9 of the last 10 merged staging PRs.
check_file is already pure per-file work, so the walk fans out over a process
pool with no change to what either rule reports. Callers sort, which is what
keeps output order stable when results land out of order. Runs below
PARALLEL_MIN_PATHS stay serial rather than pay for process startup, and the
worker count is capped so a large runner does not oversubscribe.
Measured locally over the same trees, output byte-identical both times:
type-discipline 17.8s -> 3.0s over litellm/ (78,768 report lines), test-quality
14.4s -> 2.3s over tests/ (6,321 report lines), per-rule counts unchanged.
* test(ci): type the fan-out helpers and skip the comparison on one core
The 20% promotion that runs to 2027-01-31 covers every gemini model, not
just the 2.5 pair, so a constant naming two of them implied the other four
were exempt. Six covered entries live in the registry: two store the
discounted rate and four store list, which is a pre-existing overcharge
this branch does not touch, since it only adds cache fields and derives
them from each entry's own input rate. Name both groups for what they
store, pin the expiry, and tighten the tolerance to 2e-4.
The two gemini 2.5 entries price a factor of 1.25 under the published DBU
table because the published figures exclude a 20% promotion that runs to
2027-01-31. The previous constant name and test called them an older vintage
awaiting a refresh, which would have led a future reader to scale them up and
overcharge. Pin the discount and the cache relationship instead.
`queued` is the Interactions API's not-started-yet status. It was in neither
the pollable set nor the terminal one, so a create returning it got no poll
task, counted as a response with nothing to charge for, and released its
budget reservation: billed nowhere, alerting nobody.
Poll it alongside `in_progress`, and pin the union of the pollable and
terminal sets against the generated spec enum so a status Google adds later
fails CI rather than shipping another unbilled path. A give-up on a status in
neither set now names the status and logs at error, instead of the warning
that reads as an interaction merely still running.
Also drop `model_id` and `litellm_model_name` from the settled body next to
the foreign `response_cost` already dropped there. All three come from the
poll's own throwaway client call, and left in place the two identity fields
overwrite the create's real deployment in the payload every logging
integration reads.
Rewrites the callback's per-status test to assert the observable outcome
(reservation held vs released) across all eight statuses rather than
comparing the gate to the function it delegates to, and pins the shipped
5-10-20-40-60 poll backoff and its timeout cutoff.
A parenthesised VALUES list ended the search for an insert's row source
only when no group followed it, so a RETURNING or an ON CONFLICT DO
UPDATE carrying a subquery was read as the rows the insert copies. A
writable CTE bounded by its own VALUES list was handed the query the
statement ends with for the same reason: the WITH branch read the whole
statement rather than the part holding the insert.
A CREATE FUNCTION or CREATE PROCEDURE body was scanned as if it ran at
boot, but defining a routine only stores it. The body is now read when
the same migration names the routine somewhere else, so a migration that
defines a backfill and then runs it is still caught, and one whose name
needed quoting is read either way since quoting is blanked at the call
sites too.
main() had no test, so neither its exit codes nor the branch the CI gate
reads were pinned; a mutant returning 0 on a violation passed the whole
suite. Its four outcomes now have tests, along with both directions of
each fix above.
databricks-claude-fable-5 was the only fable-5 entry in the registry
declaring supports_vision false, and the only one of the five new
entries to do so.
Only the five new models were pinned against the published DBU table,
so the 26 cache literals added to pre-existing entries were checked by
nothing independent. Extend the table to all 33 entries carrying cache
rates and assert both cache fields against it for the 31 that take the
published rates, leaving the two older-vintage gemini-2-5 entries to
their existing guard.
Also widen the cache-declaration guard to both cache fields, and
replace the single-model equals-input assertion with one that covers
all 14 entries publishing no cache rates.
Cache rates were derived as ratios of the dollar input rate (1.25x write,
0.1x read) while input and output derive from the published DBU table
times $0.070. Databricks publishes cache write and cache read DBU per
model, and those are not exact multiples of the input DBU, so the two
rules disagreed by up to 0.1 percent.
Rewrites 43 cache literals across 31 entries to published_cache_DBU x
$0.070. Skips databricks-gemini-2-5-pro and databricks-gemini-2-5-flash,
whose input and output rates predate the current table by a 1.25x
increase; their cache rates stay tied to their own input rate so each
entry remains internally consistent.
Replaces the ratio assertions with a test pinning the absolute published
DBU figures for the five new models, and adds a test pinning the
older-vintage exception. Corrects the metadata note on the five new
entries, which claimed the reference-only *_dbu_cost_per_token fields
drive cost calculation.
Silencing the cost-tracking alert for every usage-less interaction response
went one status too far. An interaction that stopped at failed, cancelled,
incomplete or budget_exceeded genuinely has nothing to charge for, so alerting
on it is noise. completed and requires_action are different: both mean the
model produced output, so a usage block is always expected, and one arriving
without it means the charge for real work was lost. That is precisely the case
failed_tracking_alert exists to surface, and swallowing it would let an
operator's interactions bill nothing with no signal that anything was wrong.
The status knowledge lives next to the other status predicates rather than in
the proxy callback. The reservation is still released on both paths, since
suppressing the alert was never what freed it.
The poll fetches the terminal interaction through its own client call, which
stamps a response cost computed by a throwaway logging object holding none of
the original request's deployment context: no model_info, no router model_id,
no deployment litellm_params. Carrying that cost into the settlement event
billed custom-priced deployments at the wrong rate, and it also satisfied the
"already calculated" shortcut in _response_cost_calculator's caller, so the
settlement never repriced and never built a cost breakdown. The zeros stamped
by the usage-less create survived into the spend log row and the OTEL span.
Dropping the imported cost before re-emitting makes the settlement price the
settled body itself, against the deployment that served the create.
The tightened gate correctly stopped deferring the reservation release for
InteractionsAPIResponses the scheduler will not poll (terminal status, or
in_progress without an id), but the response then fell through into the
generic 'Cost tracking failed' raise and the failed_tracking_alert path.
A create returning failed, cancelled, requires_action, incomplete,
budget_exceeded, or an id-less in_progress without usage therefore released
its reservation as intended and, in the same breath, alerted operators for a
legitimate no-usage response, both creating noise and masking real
cost-tracking failures.
The two gates are now nested under a single 'unbilled interaction response'
outer check, so any InteractionsAPIResponse with no usage takes either the
defer path (pollable, polling on) or the release-and-return path, and none
of them fall through to the generic failure raise. The two regression tests
also now assert failed_tracking_alert is not called, closing the observation
gap the report flagged.
The new supports_prompt_caching assertion reads a capability this branch
adds to the registry, so it only holds against the bundled map. CI leaves
LITELLM_LOCAL_MODEL_COST_MAP unset and fetches main's copy, which lags the
branch until merge, so the test failed there while passing locally.
Use the local_model_cost_map fixture the repo already provides, matching
what the other two test files in this change do.
The shared cost calculator treats a missing cache rate as free, so routing
Databricks through it billed cached tokens at zero on the 14 entries that
publish no cache pricing. On a 10,000 token prompt with 8,000 cache reads
that is $0.0010000 against the correct $0.0050001, a fivefold undercharge.
Those entries now declare cache rates equal to their input rate, which is
what a model with no caching discount should charge, and a test pins every
priced Databricks entry to declaring cache rates so no future entry can
regress into it.
Also repoints the provider-neutral generalization test off an id the new
Opus 5 entry now shadows, adds backup-to-main parity tests for the five new
entries, pins that Databricks Claude is never auto-injected with cache
control despite reporting caching support, and trims the Sonnet 5 pricing
note, which is served on an unauthenticated route.
Both leave a background interaction's pre-call reservation open, so the
serving process keeps refusing traffic on the key at the estimated cost
while its recorded spend stays near zero.
A raise from the completion event propagated out with the settlement gate
already claimed, and nothing retries a claim that is set, so the reservation
was never released. Billing now releases it on the way out.
`requires_action` was missing from the terminal set. It is terminal for the
interaction it names: the API has no operation that resumes one, and a caller
answers a tool request by creating a new interaction whose
`previous_interaction_id` points at it. A function-calling background create
that stopped there was polled until the 3600s timeout, losing the tokens it
had already spent producing the tool request and holding its reservation open
for that whole window.
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
Taking the last group at the statement's outermost level assumed the row
source was written there, and an insert is allowed to carry more after it:
`(SELECT ...) ON CONFLICT ("id") DO NOTHING` ends on the conflict target
and `... RETURNING ("id")` on the returning list, so the query supplying
the rows was never reached and a full table copy passed the gate.
Each group is now read on its own terms and the first to name a row source
is the answer, since the others are the column list and the clauses an
insert may carry, none of which names one.
The introductory DBU rates run through 2026-08-31 and pricing carries no
expiry date, so a static introductory entry would undercharge by a third
from September 1 and let spend outrun enforced budgets. Ship the standard
rates, which match Sonnet 4.5 and 4.6, and keep the introductory numbers
in the entry notes.
Also give the new cost calculator tests full type annotations.
An `INSERT` whose `VALUES` list holds a scalar subquery was reported as a
rewrite whenever that list was not the plain top-level one: joined to
another term by `UNION`, `INTERSECT` or `EXCEPT`, or written inside
parentheses, which Postgres accepts. Both shapes insert a fixed handful of
rows, so the gate was rejecting migrations that do nothing wrong.
A set operation is now split into its terms and each is read on its own,
since the insert is a rewrite when any one term is a query. A row source
kept in parentheses is read on its own terms too. The operators are found
outside every parenthesis, so a set operation written inside a `VALUES`
list does not cut the list in half.
Databricks cost calculation multiplied every prompt token by the input rate, so
a cache read cost the same as an uncached token. Route it through
generic_cost_per_token, which already understands cache reads and cache writes,
and add the cache rates the registry was missing.
Adds Claude Opus 4.7, Opus 4.8, Opus 5, Sonnet 5 and Fable 5 on Databricks.
An empty content list, or one holding only opaque blocks, still lets the
provider-bound branch replay the summary text. The inspection path treated
any non-None content as final, so that replayed text stayed invisible to
guardrails and token counting.
The field is declared optional on OpenAIFileObject and its own docstring says it
is absent on every upload guardrails did not touch, but the /v1/files routes have
no response_model, so FastAPI falls through to jsonable_encoder with exclude_none
off and serialises the unset default as an explicit null. Every create and
retrieve response on a proxy with no guardrails configured at all picked up a
litellm_batch_guardrail: null it never had before, and so did every row of a file
list, since those rows are the same object.
A wrap serializer drops the key only when nothing set it, so the populated report
still reaches the wire intact, including a record whose guardrail is null. The
managed-files list route spreads a stored file_object blob rather than the model,
so rows persisted before this lands keep their null until it is dropped there too.
A `JOIN ... USING` inside a subquery that helps build an EXECUTE's command
was taken for the start of its bind values, so anything written after it
went unscanned and a rewrite there was never reported. Only a `USING` with
the parentheses closed can be the bind-values clause.
A reasoning input item that carries only summary text is replayed to the
provider as reasoning_content, so inspection-only callers must see that
text too. They used to fall through to the generic content branch, which
reads content and drops a summary-only item, leaving guardrails and token
counters blind to text the model still receives.
A reasoning item id is not an Anthropic signature. Passing it off as one got the
block replayed to Anthropic and Bedrock as if it were real, and every backend that
verifies signatures rejected the turn. Thinking blocks now come back unsigned, and
the streaming path no longer emits a signature_delta for them.
Azure AI Foundry, Fireworks, and vLLM reject unknown message fields, so they now
strip reasoning_content alongside thinking_blocks the way Mistral already did.
The thinking-block helpers take ChatCompletionThinkingBlock and
ChatCompletionRedactedThinkingBlock instead of loose mappings.
litellm_request_total_latency_metric's start_time is set inside
common_processing_pre_call_logic, which only runs after user_api_key_auth
has already succeeded, so the metric silently excluded authentication and
pre-call setup time despite being documented as total request latency. The
sibling litellm_request_queue_time_seconds metric had the same problem:
its arrival_time was captured after auth too, despite its own comment
claiming to track when the request arrived at the proxy.
request.state.litellm_received_at is now stamped unconditionally at the
very first line of user_api_key_auth (previously only when OTEL was
configured), giving a timestamp that precedes all auth work. Both metrics
now derive from it: queue_time_seconds genuinely spans arrival through the
start of pre-call processing, and the total-latency metric adds that
queue time on top of its existing start/end window so it becomes true
end-to-end latency.
queue_time_seconds ends exactly at start_time rather than a separately
captured timestamp, so its window and the total-latency window share a
boundary instead of overlapping and double-counting a few lines of setup
work on every request.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
The Admin UI mints every session key against the UI_TEAM_ID sentinel,
which by design never has a team row, so every UI request hit the new
refusal with no override. Exempt UI_TEAM_ID explicitly so it keeps
reconstructing from the token unconditionally, matching how the MCP
handler and agent_permission_handler already special-case it.
Resolves LIT-5522
* fix(auth): resolve team object_permission independently in the unresolvable-team fallback
When get_team_object fails for a token's team_id, _user_api_key_auth_builder
reconstructs a LiteLLM_TeamTableCachedObj from the token's own cached fields,
carrying team_object_permission_id but leaving object_permission unset. That
silently dropped any vector-store or MCP restriction the team carried,
granting more access than the token's own object_permission_id vouches for.
Resolve the object permission by its id directly via get_object_permission,
independent of the unreadable team row, matching how every other consumer of
a team's object_permission (vector store access checks, MCP tool/server
resolvers) already treats an unresolvable team as "no restriction at this
level" and re-resolves on its own.
* fix(auth): trim ticket references and narrative docstrings per Greptile review
Drop the LIT-5539 ticket id from test names and fixture strings, and shorten
both the new helper's docstring and the regression test docstrings to their
contracts rather than restating the fix's history.
The team roster update, the user.teams update, the team membership
delete, and the team-scoped verification token delete ran as four
sequential writes with no transaction around them, so a failure
between any two left the removal half applied. Thread a single
prisma transaction through all four writes, following the same
tx.<table> pattern /team/member_add and /team/member_update already
use, so either all four land or none do.
The Presidio guardrail masks messages in place inside pre_call_hook, but three
paths independently persisted or emitted the raw pre-guardrail data: the
SpendLogs proxy_server_request body snapshot (taken before the hook runs),
a verbose_proxy_logger.debug dump of the raw request, and logging_only mode's
async_logging_hook, which never masked the model's response before it reached
external logging callbacks.
Resolves LIT-6015
A `FOR ... LOOP` header carries no semicolon of its own, so the first
statement of the loop body is written into the same semicolon-delimited
run. Reading the pair as one statement let the header's row source stand
in as the keyword for both, which hid whatever the loop repeats: a plain
`UPDATE` in a query-driven loop went unreported, and so did an `EXECUTE`
of one. That is the shape a row-by-row backfill takes, and it is the
shape this gate exists to stop.
The session lookup reads spend logs straight out of the database, so a
follow-up sent seconds after the turn it chains off found nothing while the
row was still queued in the worker that served it, and the conversation was
dropped without an error. Responses calls now ask the spend-log writer to
flush on its next pass instead of waiting out its poll interval, and the
lookup gives a just-finished turn a short second chance.
Replaying a session also accepted `input` only as a string or a single dict,
so the standard list shape dropped every user turn and left the model with
assistant messages alone.
