Extract a shared _valid_max_results predicate that rejects bools (an int
subclass) and non-positive values, and reuse it from both the connection-mode
request count and the response-side cap so both paths honor the same contract.
- send a caller api_key via the Azure api-key header instead of Authorization: Bearer
- cap web_search results to the requested max_results (the tool has no count knob)
- surface a Foundry failed/incomplete response status as a 502 error
- zero the per-query cost in web_search mode; keep the map price for connection mode
- trim the example config to terse env-var pointers
* fix(anthropic): reconcile enum with declared type in output_format schema
Anthropic cross-validates `enum` against `type` in structured outputs: every
enum value must match a single declared type. A union `type` array, or an enum
value whose JSON type differs from a scalar `type`, is rejected with
"Invalid schema: Enum value 'low' does not match declared type '['string','null']'"
filter_anthropic_output_schema had no enum/type reconciliation, so both keys
reached Anthropic untouched. Drop the conflicting `type` -- `enum` is the
tighter constraint, and an enum with no `type` is accepted
The drop is conditional: `type` is only removed when it is a union array, or
when some enum value does not match the scalar type. A matching enum plus
scalar `type` is left exactly as-is, so existing behaviour is unchanged
Pydantic emits the failing shape for Optional[SomeEnum], so this affects any
caller with a nullable enum field on the native output_format path. vertex_ai
is unaffected because it is forced onto the permissive tool-use path
Fixes#37881
* refactor(anthropic): make enum/type reconciliation immutable and precisely typed
Address review: the predicate registry was a mutable `dict[str, Any]`, and the
reconciliation removed `type` by mutating the built result with `pop`
- registry is now `Final[Mapping[str, Callable[[Any], bool]]]` wrapped in
`MappingProxyType`, so predicate signatures are statically checked and the
table cannot be mutated
- the conflict decision moves into a pure helper evaluated once against the
input schema, and the conflicting `type` key is skipped at build time in the
existing loop instead of being popped afterwards, so nothing is mutated
Behaviour is unchanged; all 27 tests in the schema-filter suite still pass
Resolves the test-file conflict by keeping both sides, extends the
finish-reason gate to trace-bearing metadata events so guardrail trace
chunks keep their pre-regression delta shape, parametrizes the
regression test over tool-call, mixed, and reasoning streams, and
repairs the one ant-design icon usage the lucide-react migration left
behind in skill_detail.tsx (semantic conflict on the base branch)
`exception_type` decides the class and status a caller sees for every provider
failure, across 190 raise sites, and the tests for it were written one incident
at a time. Nothing said what a plain 401 from any given provider should be, so
mutating a raise site went unnoticed: swapping the class at each of the 190 in
turn, the mapped test file caught 15.
Adds two tables asserted end to end through `exception_type`: 25 providers by
the 9 upstream statuses, and the three error shapes the router branches on
(a full context window, a content policy block, a timeout). The same 190
mutants now fail 97 of them.
The tables record today's behavior, uneven where it is uneven. cloudflare,
ollama and vllm map no status at all, so every failure reaches the caller as a
500. A full context window is recognised by 15 of the 25, and a content policy
block by 11, which bounds where `context_window_fallbacks` and the content
policy retry policy can fire.
The allowlist recorded eight files in tests/local_testing, 118 tests, that
every job globbing that directory then deselects: local_testing_part1 and
part2 carry `-k "... and not caching and not cache"`, and the other three keep
one unrelated keyword each. They counted as covered while running nowhere.
Five of the eight need nothing. Measured with no provider credentials and no
Redis: test_cache_preset_key, test_caching_handler, test_prompt_caching,
test_responses_stream_cache_keys and test_unit_test_caching pass, 45 tests
together, and they now run as a caching-local shard. The other three stay
allowlisted with what they actually need recorded rather than a question:
test_caching wants Redis and a provider key for 37 of its 65, disk-cache wants
OPENAI_API_KEY for 2 of 4, gcs-cache wants GCS credentials for all 4.
Taking them off the allowlist exposed a gap in the slice guard itself: it
reasoned only about CircleCI `-k` expressions, so a file every slice drops read
as unrun even when a workflow names it outright. It now credits workflow
test-paths the way the census already does, and only workflows, so a tree only
CircleCI globs is still reported.
* feat(ci): gate patching of SDK internals in tests as TQ008
TQ002 catches the narrowest symptom of the suite's dominant mocking idiom,
patch X then assert only that X was called. The idiom itself is wider: tests
reach for litellm's own functions instead of faking the wire, so they pin how
the code is wired rather than what it does, and a test that patches internals
but makes weak real assertions trips nothing today.
TQ008 counts patch targets rooted at `litellm`, both the dotted string form and
the attribute chain handed to patch.object, and ratchets like every other rule.
Mocking anything outside the SDK is untouched: respx, httpx transports and
third-party clients do not trip it, which is the point, since those are the
patterns this is meant to move the suite toward.
Seeded at 9,643, in line with the ~9.4k patch sites an independent grep found
in the mirror. The burn-down horizon is long; the value here is stopping the
flow rather than clearing the stock.
Five existing rule tests patched `litellm.completion` incidentally and now
report TQ008 alongside what they were pinning. Their expected values are
updated to the accurate pair rather than loosened, so they keep failing on a
regression in either rule.
* test: add TQ008 to the shipped-budget rule canary
* fix(ci): resolve imported SDK names in TQ008
patch.object(handler.OpenAIChatCompletion, ...) after a from-import reaches the
same internal as the dotted string form, but the rule only saw the bare local
name and let it through. Import bindings are now resolved to the path they
stand for, so the aliased, renamed and from-imported forms all read alike and
the reported target is the real one.
That is 1,496 patches the ratchet could not see, so the TQ008 limit moves from
9,643 to 11,139. Third-party names and locals with no SDK import behind them
stay unflagged.
* perf(ci): fan the budget checkers out across cores
check_type_discipline.py and check_test_quality.py each walk a few thousand
files and parse every one, single-threaded. In the lint job those two steps
measure 2.3 and 1.6 minutes, second and third behind dependency install, and
lint is the slowest required check on 9 of the last 10 merged staging PRs.
check_file is already pure per-file work, so the walk fans out over a process
pool with no change to what either rule reports. Callers sort, which is what
keeps output order stable when results land out of order. Runs below
PARALLEL_MIN_PATHS stay serial rather than pay for process startup, and the
worker count is capped so a large runner does not oversubscribe.
Measured locally over the same trees, output byte-identical both times:
type-discipline 17.8s -> 3.0s over litellm/ (78,768 report lines), test-quality
14.4s -> 2.3s over tests/ (6,321 report lines), per-rule counts unchanged.
* test(ci): type the fan-out helpers and skip the comparison on one core
The 20% promotion that runs to 2027-01-31 covers every gemini model, not
just the 2.5 pair, so a constant naming two of them implied the other four
were exempt. Six covered entries live in the registry: two store the
discounted rate and four store list, which is a pre-existing overcharge
this branch does not touch, since it only adds cache fields and derives
them from each entry's own input rate. Name both groups for what they
store, pin the expiry, and tighten the tolerance to 2e-4.
The two gemini 2.5 entries price a factor of 1.25 under the published DBU
table because the published figures exclude a 20% promotion that runs to
2027-01-31. The previous constant name and test called them an older vintage
awaiting a refresh, which would have led a future reader to scale them up and
overcharge. Pin the discount and the cache relationship instead.
`queued` is the Interactions API's not-started-yet status. It was in neither
the pollable set nor the terminal one, so a create returning it got no poll
task, counted as a response with nothing to charge for, and released its
budget reservation: billed nowhere, alerting nobody.
Poll it alongside `in_progress`, and pin the union of the pollable and
terminal sets against the generated spec enum so a status Google adds later
fails CI rather than shipping another unbilled path. A give-up on a status in
neither set now names the status and logs at error, instead of the warning
that reads as an interaction merely still running.
Also drop `model_id` and `litellm_model_name` from the settled body next to
the foreign `response_cost` already dropped there. All three come from the
poll's own throwaway client call, and left in place the two identity fields
overwrite the create's real deployment in the payload every logging
integration reads.
Rewrites the callback's per-status test to assert the observable outcome
(reservation held vs released) across all eight statuses rather than
comparing the gate to the function it delegates to, and pins the shipped
5-10-20-40-60 poll backoff and its timeout cutoff.
databricks-claude-fable-5 was the only fable-5 entry in the registry
declaring supports_vision false, and the only one of the five new
entries to do so.
Only the five new models were pinned against the published DBU table,
so the 26 cache literals added to pre-existing entries were checked by
nothing independent. Extend the table to all 33 entries carrying cache
rates and assert both cache fields against it for the 31 that take the
published rates, leaving the two older-vintage gemini-2-5 entries to
their existing guard.
Also widen the cache-declaration guard to both cache fields, and
replace the single-model equals-input assertion with one that covers
all 14 entries publishing no cache rates.
Cache rates were derived as ratios of the dollar input rate (1.25x write,
0.1x read) while input and output derive from the published DBU table
times $0.070. Databricks publishes cache write and cache read DBU per
model, and those are not exact multiples of the input DBU, so the two
rules disagreed by up to 0.1 percent.
Rewrites 43 cache literals across 31 entries to published_cache_DBU x
$0.070. Skips databricks-gemini-2-5-pro and databricks-gemini-2-5-flash,
whose input and output rates predate the current table by a 1.25x
increase; their cache rates stay tied to their own input rate so each
entry remains internally consistent.
Replaces the ratio assertions with a test pinning the absolute published
DBU figures for the five new models, and adds a test pinning the
older-vintage exception. Corrects the metadata note on the five new
entries, which claimed the reference-only *_dbu_cost_per_token fields
drive cost calculation.
Silencing the cost-tracking alert for every usage-less interaction response
went one status too far. An interaction that stopped at failed, cancelled,
incomplete or budget_exceeded genuinely has nothing to charge for, so alerting
on it is noise. completed and requires_action are different: both mean the
model produced output, so a usage block is always expected, and one arriving
without it means the charge for real work was lost. That is precisely the case
failed_tracking_alert exists to surface, and swallowing it would let an
operator's interactions bill nothing with no signal that anything was wrong.
The status knowledge lives next to the other status predicates rather than in
the proxy callback. The reservation is still released on both paths, since
suppressing the alert was never what freed it.
The poll fetches the terminal interaction through its own client call, which
stamps a response cost computed by a throwaway logging object holding none of
the original request's deployment context: no model_info, no router model_id,
no deployment litellm_params. Carrying that cost into the settlement event
billed custom-priced deployments at the wrong rate, and it also satisfied the
"already calculated" shortcut in _response_cost_calculator's caller, so the
settlement never repriced and never built a cost breakdown. The zeros stamped
by the usage-less create survived into the spend log row and the OTEL span.
Dropping the imported cost before re-emitting makes the settlement price the
settled body itself, against the deployment that served the create.
The tightened gate correctly stopped deferring the reservation release for
InteractionsAPIResponses the scheduler will not poll (terminal status, or
in_progress without an id), but the response then fell through into the
generic 'Cost tracking failed' raise and the failed_tracking_alert path.
A create returning failed, cancelled, requires_action, incomplete,
budget_exceeded, or an id-less in_progress without usage therefore released
its reservation as intended and, in the same breath, alerted operators for a
legitimate no-usage response, both creating noise and masking real
cost-tracking failures.
The two gates are now nested under a single 'unbilled interaction response'
outer check, so any InteractionsAPIResponse with no usage takes either the
defer path (pollable, polling on) or the release-and-return path, and none
of them fall through to the generic failure raise. The two regression tests
also now assert failed_tracking_alert is not called, closing the observation
gap the report flagged.
The new supports_prompt_caching assertion reads a capability this branch
adds to the registry, so it only holds against the bundled map. CI leaves
LITELLM_LOCAL_MODEL_COST_MAP unset and fetches main's copy, which lags the
branch until merge, so the test failed there while passing locally.
Use the local_model_cost_map fixture the repo already provides, matching
what the other two test files in this change do.
The shared cost calculator treats a missing cache rate as free, so routing
Databricks through it billed cached tokens at zero on the 14 entries that
publish no cache pricing. On a 10,000 token prompt with 8,000 cache reads
that is $0.0010000 against the correct $0.0050001, a fivefold undercharge.
Those entries now declare cache rates equal to their input rate, which is
what a model with no caching discount should charge, and a test pins every
priced Databricks entry to declaring cache rates so no future entry can
regress into it.
Also repoints the provider-neutral generalization test off an id the new
Opus 5 entry now shadows, adds backup-to-main parity tests for the five new
entries, pins that Databricks Claude is never auto-injected with cache
control despite reporting caching support, and trims the Sonnet 5 pricing
note, which is served on an unauthenticated route.
Both leave a background interaction's pre-call reservation open, so the
serving process keeps refusing traffic on the key at the estimated cost
while its recorded spend stays near zero.
A raise from the completion event propagated out with the settlement gate
already claimed, and nothing retries a claim that is set, so the reservation
was never released. Billing now releases it on the way out.
`requires_action` was missing from the terminal set. It is terminal for the
interaction it names: the API has no operation that resumes one, and a caller
answers a tool request by creating a new interaction whose
`previous_interaction_id` points at it. A function-calling background create
that stopped there was polled until the 3600s timeout, losing the tokens it
had already spent producing the tool request and holding its reservation open
for that whole window.
* test: add regression coverage for twelve closed issues
Adds targeted regression tests for behavior that was fixed but left ungated,
so the fixes cannot silently regress:
- #33772 openai cache_write_tokens cost
- #34309 Responses API cache cost_breakdown
- #35363 /v1/responses batch spend
- #36619 auto-router api_base/api_key leak on a shared model name
- #35359 batch fallbacks within the owning model group
- #36523 passthrough streamed Responses spend log
- #36646 passthrough embeddings spend log
- #37147 non-object metadata on create_batch is a 400
- #35362 unscoped list files reads the managed-file store
- #33221 gpt-5.6 bridges to Responses on function tools alone
- #34487 LLM complexity classifier runs for every caller metadata shape
- #35124 streamed /v1/messages emits success logging on both bridges
Cost assertions read rates from litellm.model_cost rather than hardcoding
dollar amounts, so they do not drift on repricing.
* fix: stop the new regression tests polluting and tripping over shared global state
Two shard failures, both from global state the new tests share with their
neighbours rather than from the behaviour under test.
test_main.py's local_cost_map pinned litellm.model_cost but left the
get_model_info lru_cache warm, so completion_cost billed at whatever prices
were cached earlier in the process while the assertions read the pinned map.
Clear the cache on both sides of the fixture, matching the local_model_cost_map
fixture in tests/test_litellm/conftest.py.
The anthropic messages streaming tests called GLOBAL_LOGGING_WORKER.flush()
on whatever queue happened to be around. A queue left non-empty by an earlier
test is still bound to that test's loop, so join() either hangs or raises
"bound to a different event loop". Rebind to the running loop before the call
and wait for the captured payload instead of a fixed sleep.
The introductory DBU rates run through 2026-08-31 and pricing carries no
expiry date, so a static introductory entry would undercharge by a third
from September 1 and let spend outrun enforced budgets. Ship the standard
rates, which match Sonnet 4.5 and 4.6, and keep the introductory numbers
in the entry notes.
Also give the new cost calculator tests full type annotations.
Databricks cost calculation multiplied every prompt token by the input rate, so
a cache read cost the same as an uncached token. Route it through
generic_cost_per_token, which already understands cache reads and cache writes,
and add the cache rates the registry was missing.
Adds Claude Opus 4.7, Opus 4.8, Opus 5, Sonnet 5 and Fable 5 on Databricks.
An empty content list, or one holding only opaque blocks, still lets the
provider-bound branch replay the summary text. The inspection path treated
any non-None content as final, so that replayed text stayed invisible to
guardrails and token counting.
The field is declared optional on OpenAIFileObject and its own docstring says it
is absent on every upload guardrails did not touch, but the /v1/files routes have
no response_model, so FastAPI falls through to jsonable_encoder with exclude_none
off and serialises the unset default as an explicit null. Every create and
retrieve response on a proxy with no guardrails configured at all picked up a
litellm_batch_guardrail: null it never had before, and so did every row of a file
list, since those rows are the same object.
A wrap serializer drops the key only when nothing set it, so the populated report
still reaches the wire intact, including a record whose guardrail is null. The
managed-files list route spreads a stored file_object blob rather than the model,
so rows persisted before this lands keep their null until it is dropped there too.
A reasoning input item that carries only summary text is replayed to the
provider as reasoning_content, so inspection-only callers must see that
text too. They used to fall through to the generic content branch, which
reads content and drops a summary-only item, leaving guardrails and token
counters blind to text the model still receives.
A reasoning item id is not an Anthropic signature. Passing it off as one got the
block replayed to Anthropic and Bedrock as if it were real, and every backend that
verifies signatures rejected the turn. Thinking blocks now come back unsigned, and
the streaming path no longer emits a signature_delta for them.
Azure AI Foundry, Fireworks, and vLLM reject unknown message fields, so they now
strip reasoning_content alongside thinking_blocks the way Mistral already did.
The thinking-block helpers take ChatCompletionThinkingBlock and
ChatCompletionRedactedThinkingBlock instead of loose mappings.
litellm_request_total_latency_metric's start_time is set inside
common_processing_pre_call_logic, which only runs after user_api_key_auth
has already succeeded, so the metric silently excluded authentication and
pre-call setup time despite being documented as total request latency. The
sibling litellm_request_queue_time_seconds metric had the same problem:
its arrival_time was captured after auth too, despite its own comment
claiming to track when the request arrived at the proxy.
request.state.litellm_received_at is now stamped unconditionally at the
very first line of user_api_key_auth (previously only when OTEL was
configured), giving a timestamp that precedes all auth work. Both metrics
now derive from it: queue_time_seconds genuinely spans arrival through the
start of pre-call processing, and the total-latency metric adds that
queue time on top of its existing start/end window so it becomes true
end-to-end latency.
queue_time_seconds ends exactly at start_time rather than a separately
captured timestamp, so its window and the total-latency window share a
boundary instead of overlapping and double-counting a few lines of setup
work on every request.
When get_team_object fails, the centralized auth gate rebuilds the team
from the token's own fields. A token whose team row was missing when the
key was read carries team_models=[] and team_blocked=False, and the
model-access check reads an empty model list as every model, so the
rebuilt team grants more than the real team ever did.
get_team_object reported a deleted team and a database that would not
answer as the same 404, so the fallback could not tell a definitive
answer from a degraded read. Raise a TeamNotFoundError subclass, still a
404 with the same detail so every other caller is unaffected, only when
the database answers and the row is absent.
A team that is provably gone now refuses, and no setting overrides that.
Otherwise the grant is merely unknown: a token carrying one may vouch,
since replaying a recorded grant cannot widen it, and a token carrying
none may not. allow_requests_on_db_unavailable still opts back out there,
and is only consulted once the failure is known to be a degraded read.
The Admin UI mints every session key against the UI_TEAM_ID sentinel,
which by design never has a team row, so every UI request hit the new
refusal with no override. Exempt UI_TEAM_ID explicitly so it keeps
reconstructing from the token unconditionally, matching how the MCP
handler and agent_permission_handler already special-case it.
Resolves LIT-5522
