An out-of-range cursorless page returns nothing, and reading its total off the
offset reported more sessions than exist (page 4 of 100 sessions at page size 50
claimed 150). Only a page that holds rows, or the first page, ends the list;
anything past it falls back to the bounded count.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
A cursorless page that comes back without its lookahead row is the end of the
list, so the total is offset + len(page) and the bounded grouped COUNT over the
whole spend-log table is skipped. First pages on small deployments and every
offset last page now cost one query less.
Moves the count into _count_grouped_sessions and reworks the query-optimization
test that asserted the count always runs second onto a full page, where it does.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
The streaming iterator hook timed the whole provider stream and logged that as the
guardrail duration, so PrometheusLogger added LLM generation time to
litellm_overhead_with_guardrails_latency_metric. The hook now accumulates the time
spent inside _filter_single_text per chunk and logs that sum, keeping start_time and
end_time as the wall-clock window.
Resolves LIT-7589
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A page size that does not divide SPEND_LOGS_PAGINATION_COUNT_CAP left the last
page starting inside the capped window and reading past it, so the rows
disagreed with the total reported next to them. The page limit now stops at the
end of that window, and has_more plus next_session_cursor still hand back a
cursor for walking further.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
A page starting at or past SPEND_LOGS_PAGINATION_COUNT_CAP lies outside the
total the client is given, so it now returns no rows without running the page
query and the grouped top-N sort bound stays capped.
Rewrites the offset test to page a fake session store instead of asserting on
the generated SQL, and covers the last page inside the cap next to the first
page past it.
Claude-Session: https://claude.ai/code/session_01ESi9JwaXDww1vP3Qsrr4Mz
LiteLLM sends the Bedrock Mantle GPT rows through Bedrock's Responses endpoint, which refuses minimal on gpt-5.4 and gpt-5.5 like every other Bedrock GPT row. The earlier commit measured the raw chat endpoint, which accepts it, and dropped the flag by mistake. The ladder test now matches what the proxy path returns
Live calls to Bedrock Mantle and Converse on 2026-09-11: the gpt-5.6 luna, sol, and terra rows and gpt-6-astra return 200 on reasoning_effort=max, gpt-6-astra returns 400 on none, and Mantle gpt-5.4 and gpt-5.5 return 200 on minimal. The commercial Bedrock rows now carry exactly those flags, and the schema test asserts the measured ladder per row instead of a blanket mirror of the direct OpenAI rows
* fix(datadog_llm_obs): keep tool call and result structure under redaction and emit tool output tokens
Under datadog_llm_observability_params.turn_off_message_logging the span kept only one role plus "redacted-by-litellm" per message, so Datadog showed Tool Call 0, Tool Result 0 and no tool output token data. The shared CustomLogger hook collapsed the messages before the callback ran, and the Datadog redaction then dropped tool_calls and tool_results.
The Datadog callback now opts out of the shared message collapse (redacts_messages_itself) and redacts its own normalized messages, keeping roles, tool names, ids and types while replacing content, arguments and results. Tool result tokens are counted with litellm.token_counter before redaction and shipped as the tool_output_tokens metric. Other callbacks keep the inherited behavior.
Resolves LIT-7545
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(datadog_llm_obs): drop explanatory docstrings from the redaction change
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore(ui): regenerate schema.d.ts for the classifier descriptions changed in #40655
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): write failure spend log for guardrail-blocked /mcp-rest/tools/call
call_tool_rest_api only translated exceptions to HTTP responses, so a pre_mcp_call
guardrail block never reached failure_handler / async_failure_handler /
post_call_failure_hook and no LiteLLM_SpendLogs failure row was written. Extract
the failure logging from call_mcp_tool into _fire_mcp_tool_call_failure_logging
and run it in the REST route for anything raised between
common_processing_pre_call_logic and execute_mcp_tool
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): keep the original REST tool error when failure logging raises
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(mcp): log virtual mcp_tool_call failures and keep REST success latency scoped to tool execution
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep call_type and request start time on failed-request spend logs
post_call_failure_hook pops litellm_logging_obj before the failure callbacks
run, so the spend row built from request_data had a blank call_type and used
datetime.now() as the start time. A guardrail-blocked MCP tool call therefore
showed up in the Logs page as an LLM row with no call type and a 0s duration.
Lift call_type and start_time off the logging object alongside the fields
already lifted, and have the DB failure hook prefer the lifted start time.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(proxy): inject the spend writer into _ProxyDBLogger instead of patching a module global
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): fall back from unhealthy auto-router tier
Co-Authored-By: Claude Code <noreply@anthropic.com>
(cherry picked from commit 00c7fd8376)
* fix(router): treat budget and tag exhaustion as a no-capacity verdict
The eligibility probe only read typed router errors as "nothing here can
serve this". Provider and deployment budget exhaustion, and tag routing
with no matching deployment, report it as a bare ValueError carrying a
RouterErrors marker, so the probe read a spent tier as live, skipped the
peer and default recovery, and failed the request.
---------
Co-authored-by: Tin Chi Lo <tin@berri.ai>
Co-authored-by: Claude Code <noreply@anthropic.com>
* fix(proxy): authorize every Responses API id, not only the ones the proxy issued
The ownership check on the Responses API only ran when the id arrived in the
proxy's own encrypted format. An id in any other shape skipped the check and
was forwarded upstream, so a key that did not own the response could retrieve,
cancel, delete, or chain off it.
Every addressed id now goes through one authorization step shared by retrieve,
cancel, delete, list-input-items, and create's previous_response_id. An id the
proxy did not issue is refused with 403 unless the deployment opts in with
general_settings.allow_unmanaged_response_ids, has responses id security
disabled, has no signing key configured, or the caller is a proxy admin.
* fix(proxy): re-authorize the retained responses id instead of trusting it
The docstring cited session_affinity_ttl_seconds as the keepalive bound,
but the Router-level knob feeding ttl_seconds is
deployment_affinity_ttl_seconds; session_affinity_ttl_seconds is the
separate per-request PreRoutingHookResponse override. Anyone grepping
the docstring's name to shrink the Router default finds only the
override. Name both, scoped correctly.
Starlette scans the route table in registration order, so a request pays one
regex match per route registered ahead of its own. The proxy registers several
hundred routes and left the liveness probe near position 280 and the lazy
loaded /v1/messages at the very end. Move /health/liveliness, /health/liveness,
/v1/chat/completions, /chat/completions and /v1/messages to the front of the
route table after startup registration and again after a lazy router loads.
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
