* fix(guardrails): exclude litellm_metadata from pass-through guardrail scan text
The pass-through guardrail translation json-dumps the request body as the
scan text when no request_fields are configured. The unified guardrail hook
injects litellm_metadata into that body first, so the dump carried the
caller's key metadata. Exclude it alongside metadata and litellm_logging_obj;
vendors still receive it through request_data.
* test(guardrails): type the pass-through scan-text tests per AGENTS.md
* fix(otel): decide per-team Arize sampling from the trace id so every pod agrees
* test(otel): cover per-team sampling across pods, workers, endpoints and restarts
* fix(otel): remember only a failed tree's sampling verdict so a later failed call answers to the error rate
A successful tree's verdict follows from the trace id alone, so remembering it only made a later failed call of the same trace follow the success rate on the pod that served both, while other pods judged it at the error rate
* test(otel): flush every worker before the restart cell kills it and cover a failed call after a dropped one
* fix(otel): decide team sampling per request, not per trace
Calls sharing a caller's trace id are separate requests, keyed by their root span.
Each is decided by the trace hash at the rate for its own outcome, and only its own
late spans follow its verdict, so no pod applies an earlier call's verdict to a later
call of the trace
* fix(otel): forget a span's request root when it ends, not when newer spans start
The root of each open span was kept in a first-in-first-out table of 16384 entries,
so a span two levels below the root that stayed open while the worker started more
spans than that lost its entry and ended under its parent's id. Its request then
never closed and was held from every sampled destination until shutdown.
Each open span now keeps its entry until it ends, and a span starting under a parent
that has already ended reads the root from the parent span itself
* fix(otel): decide a request as its spans end once it falls out of the open span table
A request whose span was pushed out of the table of open spans, because the worker
held more than 16384 open at once, ended that span under another key and was held
from every sampled destination until shutdown. Pushing a span out now drops its
request's open count too, so the request is decided as each of its spans ends, the
way one pushed out of the open request table already is
* test(otel): annotate the team sampling test locals as Final
* test(otel): build the caller's remote parent in one helper instead of rebinding it per trace
* fix(otel): keep each open span's request until it ends, bounded by the open requests
The span table had its own bound of 16384, so a worker holding more spans open than
that pushed out the oldest, and a failure in such a span no longer reached its
request. Each request now holds the ids of its open spans, and a span's entry in the
lookup goes when the span ends or when its request is pushed out of the open request
table, the same bound main already applies. A failure anywhere in a tracked request
reaches its verdict however many spans are open
---------
Co-authored-by: yucheng-berri <293926326+yucheng-berri@users.noreply.github.com>
* fix(streaming): stop re-raising and re-logging a failed stream on every next()
A stream that raised keeps its provider iterator alive, so a consumer that iterates past the error re-detected, re-raised and re-logged the same failure on every call, flooding failure callbacks for one request. Latch the failure and re-raise it without logging again.
* fix(streaming): drop the retained traceback before re-raising a failed stream
* test(streaming): count failures through a real logging callback and require a clean stop on later reads
* test(streaming): require the stuck stream to close early and release its socket
The provider stream now records a GeneratorExit and caps its chunks, so draining it or detecting the loop late fails. Two wire cases check a sync and an async stuck stream release their TCP session.
* test(streaming): avoid failure recorder name collision
---------
Co-authored-by: Devin AI <devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
Preserve oversized JSON integers in proxy requests and model rewrites
Parse large bodies in a worker thread to avoid blocking request handling
Fixes#47947
Co-authored-by: Jeevan Babu Gotru <114808955+jeevan1098@users.noreply.github.com>
Co-authored-by: moe-berri <moe@berri.ai>
* fix(deepseek): pass reasoning_effort through to the API
DeepSeekChatConfig.map_openai_params mapped reasoning_effort to a binary
thinking {type: enabled/disabled} toggle and discarded the effort value, so
DeepSeek always ran default high effort. High-effort reasoning can run away
and exhaust max_tokens during reasoning, returning empty content. Preserve
the reasoning_effort value into the request body when thinking is enabled so
callers can bound effort (e.g. reasoning_effort="low"). Backward-compatible.
See https://api-docs.deepseek.com/guides/thinking_mode
* test(deepseek): reasoning_effort passthrough coverage
* fix(deepseek): exclude reasoning_effort=none sentinel from passthrough
"none" is the thinking-OFF sentinel, not a real effort value. Passing it
alongside explicit thinking:enabled would send contradictory instructions to
DeepSeek. Only forward real effort values (low/medium/high/xhigh/max).
Addresses review feedback on the reasoning_effort passthrough change.
* refactor(deepseek): name supported effort values and type mapped params
* fix(deepseek): keep mapping types local to the adapter
* fix(deepseek): remove cast and type mapped return
---------
Co-authored-by: moe-berri <moe@berri.ai>
* fix(proxy): reconcile team member total_spend with current cycle spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(integration): read /team/list as a list and drain the scripted provider
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(repositories): compose the membership reset writer and drop SQL text asserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: type the cascade repair test and flatten memberships without a stacked comprehension
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: keep the reset job test file's existing typing untouched
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(cost): parse 12-hour cache write rates for batch and ultrafast tiers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): bill 12-hour batch cache writes once when only the 12hr batch key exists
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(teams): show caller member spend and budget on Your Teams
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): use member-linked reset dates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep type-discipline suppressions on cast lines
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): preserve empty page list response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): keep the member reset date inside the spend column
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(teams): match email-only roster members for caller membership
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(ui): show team member budget reset date in UTC
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(teams): address review on caller membership enrichment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(ui): label the caller's team spend line My Spend
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): bill OpenCode Go requests from the reported list_cost
OpenCode returns usage.opencode.list_cost on every response. Read it into the provider-reported cost the cost calculator checks before the price map, for chat and Responses, streamed and not. Keep the static rows as the fallback, and stop the generic usage.cost path from billing Go requests at 0 (usage.cost is what hit the prepaid balance, 0 inside the plan).
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(core): resolve reported cost breakdowns without recursion
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(opencode): cover reported list_cost spend through the proxy
Replaces the recursive _reported_cost_value with a flat _cost_number helper
so the code-quality gate passes, and adds an integration test that runs the
proxy against a scripted OpenCode upstream for a non-stream and a stream
chat completion, asserting the response cost header and the spend log row
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): bill native /v1/messages from the reported list_cost
The native Anthropic Messages route priced OpenCode Go bodies from the price
rows twice over: the non-stream body is a TypedDict the response-cost header
path priced as-is, and the stream assembled its ModelResponse with the rows
cost already stamped. Both now read usage.opencode.list_cost, the non-stream
body through billable_result and the stream from the final message_delta.
The integration test now covers chat, responses and messages, stream and
non-stream, through the proxy against a scripted OpenCode upstream
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): keep the stream cost stamp inside the lint gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(opencode): read list_cost off dict chunks and trust it only from OpenCode on /v1/messages
Stream builders receive chunks both as mappings and as ModelResponseStream
objects, so the list_cost scan now reads usage through chunk_usage. The
native Messages route only trusts a reported cost from opencode_go, every
other upstream keeps the price map, which also keeps a zero usage.cost from
being billed as free. Served bodies without stop_sequence still price, and
the new code stays inside the basedpyright gate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(proxy): add disable_pre_routing_error_logs to skip spend rows for pre-routing rejections
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): decide pre-routing rejections by pipeline stage, not exception type
Guardrails such as AIM raise ProxyException(400), the same shape as an unparsable
body, so the status code list skipped their spend rows. The pipeline stamps
litellm_call_id only after auth, body parsing and model resolution, so a failure
reaching the hook without one died before any provider was considered.
_handle_logging_proxy_only_error flags those requests when it mints the call id,
and is_pre_routing_rejection keys off that flag plus ProxyModelNotFoundError.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark pre-routing rejections with a server-owned sentinel
request_data is the client body, so a boolean marker or a litellm_call_id in it
can be sent by the client to hide a guardrail block or force a spend row. The
failure handler now stores an object() sentinel whenever it finds no logging
object, and is_pre_routing_rejection checks identity, which JSON cannot forge.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): skip declared text-only deployments for image requests without enable_pre_call_checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): run the vision filter after the cooldown filter so a cooled vision leg falls back to the text leg
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): apply the vision filter after pre-call checks and tidy the new tests
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): mark the remaining locals in the vision filter tests Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* feat(guardrails): add native Reco guardrail provider (pre_call)
Registers guardrail: reco as a first-class vendor integration, built on
the existing generic guardrail API wire contract rather than reimplementing
the HTTP client and payload mapping. Maps reco_tenant_id to the
X-Reco-Tenant-Id header and hardcodes fail_open/fail_on_error since those
should not be customer-configurable. v1 only supports mode: pre_call;
post_call and streaming redaction already work against the existing
generic pilot and are deferred.
Also overrides apply_guardrail to re-raise blocked-request errors with the
guardrail's own configured name: the base class hardcodes a module-level
name in that exception, which would otherwise mislabel every blocked Reco
request in logs.
* feat(guardrails): configurable Reco fail modes
Expose unreachable_fallback and fail_on_error on the Reco guardrail
and keep fail-open as the default. Values under optional_params win.
Top-level litellm_params values apply only when set explicitly, since
their defaults are fail-closed and would otherwise flip the default
Also type RecoGuardrail's forwarded kwargs with an Unpack'd TypedDict
instead of silencing ANN003
Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
* chore(guardrails): align Reco with main's CI gates
Upstream moved unit tests from tests/test_litellm to tests/unit, so
test_reco.py was invoked by no CI job. Move it next to the other
guardrail tests
Replace RecoGuardrail's forwarded kwargs with explicit keyword
params, read optional_params through a pydantic adapter instead of
untyped dict/getattr access, freeze RecoOptionalParams, and drop
mutable-ok markers that suppress nothing, to stay within main's
type-discipline and basedpyright budgets
Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(guardrails): fail closed by default in the Reco guardrail
Matches GenericGuardrailAPI and every other guardrail here: a Reco error or outage now blocks the request unless the admin sets fail_on_error: false. Tests cover the default, explicit fail-open, and top-level vs optional_params precedence
Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): honor explicit fail-open on Reco when optional_params is a model
model_dump(exclude_unset=True) so pydantic defaults in optional_params no longer override an explicit top-level fail_on_error. Also drop the duplicate log_guardrail_information decorator, call the parent by keyword, freeze RecoConfigModel, and type the tests
Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): keep Reco config model assignable and the hook decorator the gate requires
frozen=True on RecoConfigModel froze the shared LitellmParams mixin, which the guardrail registry assigns resolved secrets on, so every guardrail failed to load. Dropped it and added a test that LitellmParams stays assignable. Restored log_guardrail_information on the Reco override since tests/code_coverage_tests/check_guardrail_apply_decorator.py requires it on every hook. Test params typed as Mapping for LIT001
Co-authored-by: RecoLabs-YaronBratspiess <208601538+RecoLabs-YaronBratspiess@users.noreply.github.com>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style(guardrails): format the Reco test file
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Signed-off-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Yaron Bratspiess <yaronb@reco.ai>
Co-authored-by: MickeyShnaiderman-RecoLabs <mickeys@reco.ai>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: MickeyShnaiderman-RecoLabs <77006830+MickeyShnaiderman-RecoLabs@users.noreply.github.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): price claude_platform calls from the anthropic cost map rows
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: avoid recursive Claude Platform lookups
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(bedrock): price claude_platform calls from their own cost map rows
Replace the claude_platform -> anthropic lookup redirect with
bedrock/claude_platform/* rows, and apply the inference_geo multiplier
in the bedrock cost path so US-only inference is priced at 1.1x
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* test(bedrock): exempt claude_platform rows from the Converse routing guard
The claude_platform/ route is explicit in the row key, so these rows are
not accidental non-Converse Bedrock chat rows
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(anthropic): stop injecting empty-content placeholder into tool-call-only assistant turns
_sanitize_empty_text_content replaced empty or whitespace-only text with
"[System: Empty message content sanitised to satisfy protocol]" for every user and
assistant turn. anthropic_messages_pt calls it unconditionally, independent of
modify_params, because an empty text block always makes Anthropic 400.
An assistant turn carrying tool_calls and no text is the normal shape an OpenAI-format
agent emits, and Anthropic accepts an assistant turn whose content is just tool_use, so
there was nothing to substitute there. Substituting anyway put a protocol diagnostic in
the content field, which is the assistant's utterance as far as the model and the client
are concerned, so it got persisted into conversation history and echoed back on later
turns.
Empty text is now dropped rather than replaced when the assistant turn already carries
substantive content (tool_calls, or non-text blocks such as tool_use). Turns that would
otherwise be left with no content at all keep the placeholder, since Anthropic rejects
those too.
Fixes#24498
* fix(anthropic): only drop empty text when a sibling block survives conversion
_carries_non_text_content treated any non-text block as substantive, but the assistant
list-content converter only keeps thinking (when signable), server_tool_use and
*_tool_result blocks. For an assistant turn holding empty text next to a block it
discards, such as an image_url or a signature-less thinking block, dropping the empty
text left nothing behind and the whole turn disappeared from the request.
The predicate now mirrors what the conversion actually keeps, so those turns retain the
placeholder and survive. This also improves on the previous behaviour, where such a turn
was dropped outright.
* test(anthropic): cover malformed content blocks in the survivor predicate
_survives_assistant_conversion had a separate non-dict guard that no test reached,
since _is_empty_text_block already rejects non-dict blocks before the predicate runs
on them in practice. Folding the check into the type lookup keeps the same behaviour
for bare strings and blocks without a type key, with no unreachable branch.
Extends the sibling-block test with a block missing its type key.
* test(anthropic): set modify_params through monkeypatch in sanitization tests
The test-quality gate flags direct writes to litellm module globals. Swap the
class setup/teardown for an autouse monkeypatch fixture, drop writes that just
repeat its default, and route the False cases through monkeypatch
---------
Co-authored-by: tbartik <22518863+tbartik@users.noreply.github.com>
Co-authored-by: ryan-crabbe-berri <ryan@berri.ai>
* fix(vector_stores): put azure vector store id before api-version query
Azure OpenAI's get_complete_url returns a url ending in ?api-version=...,
and the vector store handlers appended the id with an f-string, producing
.../openai/vector_stores?api-version=X/vs_abc and a 404 from Azure.
Add BaseVectorStoreConfig.get_vector_store_url, defaulting to the previous
behaviour so other providers are unchanged, and override it in
AzureOpenAIVectorStoreConfig to keep the query string last. The six
retrieve/update/delete handlers now call it.
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(vector_stores): trim comments on get_vector_store_url
Drops the inline comment in the Azure override and shortens the base docstring, per AGENTS.md
Co-authored-by: Keith-Hardaway <276775292+hardawaykwork@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(vector_stores): capture the Azure retrieve request without a mutable url list
Greptile on #48224: the handler test now follows the file's received list pattern with Final bindings
Co-authored-by: Keith-Hardaway <276775292+hardawaykwork@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Keith-Hardaway <keith.hardaway@docusign.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Keith-Hardaway <276775292+hardawaykwork@users.noreply.github.com>
* fix(router): keep latency routing alive when latency samples are unusable
latency-based-routing ranks deployments by the average of the cached samples
and keeps everything within lowest_latency_buffer times the lowest. A negative
or NaN sample breaks that invariant, because the minimum no longer satisfies
its own bound. The candidate list then comes back empty and random.choice
raises IndexError, which turns every request to the group into a 500 until the
cache entry expires.
Samples come from completion_start_time - start_time, and that difference is
negative whenever the two timestamps invert. Production recorded samples
around -600s from successful calls.
Drop non-finite and negative samples before ranking, and fall back to every
candidate if the window is empty anyway, for example with a negative buffer.
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
* test(router): seed latency_normalized in the unusable TTFT fallback test
Main reads latency_normalized since #47727, so seeding latency left both deployments ranked at zero and the test picked at random.
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): assert ranked latency after dropping unusable TTFT samples
Greptile on #48216: the new tests passed with the sample filter removed because the empty-window fallback alone saved them. They now assert _latency_per_deployment so unfiltered negative or NaN samples fail deterministically. Also adds Final to ttft_samples and drops a comment that restated the code
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): drop Final on a loop-local in latency ranking
basedpyright rejects Final on a variable assigned inside a loop, and the type_check_gate counts it as a new reportGeneralTypeIssues error
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): type the request kwargs in the unusable TTFT tests
Greptile on #48216: bare dict annotations. The tests now hold the metadata dict in a typed local and assert on it directly
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Rick <26716961+Bytechoreographer@users.noreply.github.com>
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: AaronHowell <111272993+AaronHowell@users.noreply.github.com>
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(typing): replace pass-through Any with object in 13 annotations
* docs(client): say dict[str, object] in the models get docstring
* refactor(typing): drop the unused Span aliases in base_cache and levo
* refactor(typing): take a Mapping in delete_nested_value and cover the fresh copy
* fix(typing): deep-copy a read-only Mapping in delete_nested_value through a dict
* fix(proxy): buffer tool discoveries in redis and let the leader write LiteLLM_ToolTable in one statement
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): declare the redis family on the tool discovery buffer and drop the blind except and cast
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): satisfy LIT010 and LIT014 in the tool discovery buffer
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): mark the aggregation fold Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): requeue drained tool rows to redis when the write provably did not apply
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): drain tool discoveries on their own lock and scheduler job, not the spend commit
* chore(proxy): suppress the untyped apscheduler add_job on the tool discovery job
* refactor(proxy): move the LiteLLM_ToolTable bulk upsert into its own module next to the other bulk upserts
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): name the LiteLLM_ToolTable bulk write span so the prisma span check resolves it
* refactor(proxy): move the tool discovery drain out of DBSpendUpdateWriter into tool_registry_writer
* refactor(proxy): share db_write_failure_is_requeue_safe from exception_handler so the spend writer stays untouched
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Add OpenCode Go model discovery, protocol routing, session affinity, and published pricing metadata. Cover Chat Completions, Responses, Messages, and Gemini through the existing provider adapters.
Fixes#31568
* feat(ui): configure the default Usage time range
* fix(ui): retain the selected calendar preset during input sync
* fix(ui): preserve fallback dates when applying across DST
* fix(router): route decisions calls only to deployments whose provider supports decisions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): account for indirect route helper coverage
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): pin the chat leg so the mixed-group check is deterministic
Weights pick the Bedrock leg for chat instead of relying on simple-shuffle landing on both legs. The route filter now passes the candidate list through unchanged in type, so the pre-existing selector lines lose their pyright ignores
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(typing): narrow Any sources across providers, proxy and guardrails (wave 1)
* refactor(typing): narrow Any sources across hubs, providers and proxy (wave 2)
* refactor(typing): validate RunwayML task JSON and let dotprompt loaders accept exported prompts
* fix: restore pass-through behavior on paths the typing pass tightened
MCP client IP extraction trusts X-Forwarded-For only when use_x_forwarded_for is True, dotprompt skips a malformed JSON prompt entry instead of raising, RunwayML task payloads pass through unvalidated, and a malformed provider body raises the same AttributeError as before at the CompactifAI, Gemini, and mock transport parse sites
* fix(runwayml): keep a malformed 2xx task body on the 500 path
* fix(edenai,cli): keep a non-object body in the error and fail on a malformed migration report
EdenAI's embedding and image generation parse dropped a non-object body from the error
they raised, so the message no longer named what the provider sent. A helper now raises
with the body in the message, and both paths have a regression test.
`lite encryption migrate` exited 0 with no message when the proxy's report was not an
object or its residual_legacy was not a number, where base crashed. It now fails with a
message naming the value.
The CompactifAI and Gemini image-edit non-object body tests accept TypeError or
AttributeError, which both base and this branch raise, so they guard against a
ValidationError instead of pinning one class. RunwayML's async parse gets the same
non-object body test as the sync one.
* fix: keep base's error classes, duck-typed checks and fail-closed guardrails the typing pass changed
* test: annotate the new regression tests' locals with Final
* test: annotate the remaining new test locals with Final
* feat: add OpenAI 12-hour cache write pricing
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix: omit empty cache-write response fields
* style: format cache-write fields
* fix(cost): bill openai 30m and 12h cache writes from the usage split, add gpt-5.6 rates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): apply the 12h long-context write rate only above the token threshold
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): key cache write rates by TTL instead of growing the rate tuple
_get_token_base_cost returned a positional 6-tuple and every cache lifetime added a slot to it, to the off-peak helper and to every caller. It now returns a TokenBaseCosts dataclass whose cache write rates live in one mapping keyed by TTL. The TTLs come from the ephemeral_*_input_tokens fields on CacheCreationTokenDetails, so supporting a new lifetime only means adding that usage field and its rate key
A lifetime with no rate of its own now always falls back to the cache creation rate after off-peak pricing is applied. Before this the non-tiered 12h rate and the tiered 1h rate fell back to the rate from before off-peak, so the same request billed differently depending on the pricing path
* refactor(cost): hardcode the cache write TTLs as a Literal instead of parsing usage field names
Parsing ephemeral_*_input_tokens names saved nothing, since a new TTL still needs a usage field and provider parsing code. The TTLs are now a CacheWriteTTL Literal with an explicit rate key per TTL
* refactor(cost): resolve cache write lifetime rates to floats at lookup
Decide the 1h and 12h cost map key in the threshold loop, then resolve each
rate once after off-peak with the base write rate as the default, so no None
flows through the cost code. Add an integration case for a 12h write on a
row without a 12h rate
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(cost): name every cache write TTL, marking 5m and 30m as base rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): list every cache write TTL in one rate key table, None meaning the base write rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): drop base rate entries from the cache write rate table and rely on fallback
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): key cache write rates by plain string TTL
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): fill missing cache write TTL rates with the base write rate after off-peak
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): default missing cache write TTL rates to the base write rate inline
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): rename cache write rate key table to make clear it only covers long TTLs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): list every cache write TTL and bill writes of an unknown TTL at the base write rate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): resolve tiered cache write TTL rates after off-peak instead of passing None
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): carry cache write TTL rates on TokenRates and drop TokenBaseCosts
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* refactor(cost): define TokenRates before its first use to drop the string annotation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(cost): build the TokenRates TTL default with default_factory and type tiers as Mapping
MappingProxyType({}) as a dataclass default is rejected on Python 3.11, where
mappingproxy is unhashable, so import litellm failed there. The tier type
moves from dict[str, object] to Mapping[str, object] (LIT001), with tier_rate
accepting a Mapping to match.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(cost): type select_tier_for_input tiers as Mapping so tier_rate callers stay typed
The tier returned as a bare dict made every tier_rate call site a partially
unknown argument once tier_rate took Mapping[str, object]. The range bounds are
now validated with a TypeAdapter instead of indexing an unknown list.
---------
Co-authored-by: devin-ai-integration[bot] <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* fix(responses): honor per-call custom pricing on the native Responses API path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): narrow the custom pricing regression test to the type gates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* refactor(main): expose register_custom_pricing_for_request for the responses path
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(responses): mark custom pricing test locals Final
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): price per-call custom rates request-locally instead of registering them globally
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): bind per-call rates on the request logging object instead of inferring them in Logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): keep Router deployments on the cost-map pricing path
Router inherits cache rates onto the deployment's cost-map entry, so a routed
Responses call binds no request-local price and keeps billing cached tokens at
the inherited rate. Moves the deployment-id lookup to router_utils so the
Responses and completion paths share it.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(responses): gate request-local pricing on the typed deployment model_info
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep image requests off deployments declared supports_vision false in pre-call checks
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): type the vision pre-call helpers so the basedpyright gate holds
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): keep the pre-call index set typed without a mutable annotation
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover the vision pre-call helpers directly for the router coverage check
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): type the vision pre-call test helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(router): cover the no-params and unresolvable-provider vision branches
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): await the per-request rollup enqueues and report their failures
The daily and entity spend rollups ran in an unreferenced asyncio task that
nothing awaited or drained, so a shutdown after the spend log was enqueued
could drop every rollup for that request, and a malformed metadata payload
dropped the request from all daily tables at debug level. The rollups now run
inline in update_database, each helper failure is logged through
spend_log_error with the request id, and unreadable metadata only costs the
row its metadata-derived fields
Resolves LIT-6027
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): type the rollup failure helpers for the lint gates
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): track the rollup enqueue task so shutdown drains it
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(spend): shutdown drain waits for an in-flight rollup enqueue
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(spend): register the rollup task without blocking the cost callback
Awaiting the rollups inline made a blocked queue put stall increment_spend_counters, so budget checks could run on stale counters. The task is now only registered via spawn_logging_task; drain_logging_tasks still waits for it on shutdown
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: joshua <joshua@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
