error_status_code only read status_code, so a ProxyException raised
before routing (which stores its status as the string code) answered
500 with its 4xx type through the rerank, images, realtime, files, and
pass-through tails. It now falls back to a decimal code. A 408 maps to
timeout_error instead of invalid_request_error.
Tail regressions for rerank, images, realtime calls, and the chat
pass-through fail at the merge base with ('None', 'None'); the new
files-test helpers are fully typed.
Fireworks renders a Responses request through a chat template that only
accepts a system message at the very beginning, so a request carrying
`instructions`, a developer item, and a replayed reasoning item (the shape
Codex CLI sends from its second prompt on) came back 400 with "System
message must be at the beginning".
The leading system or developer items, and any developer item later in
the conversation, now fold their text into top-level `instructions`,
joined with blank lines, and leave `input`. A developer item that closes
the conversation right after an assistant turn stays where it is as a
system item, as does any system or developer item with an image or file
part, so those parts still reach Fireworks. Mid-conversation system items
stay untouched. Non-string `instructions` pass through unchanged.
Folding into `instructions` rather than a leading system item keeps
`previous_response_id` chaining working, since Fireworks prepends the
stored history to `input` and a leading system item would land after it.
This supersedes the leading system item approach from deaadc21d3 and
4807630c2a on this branch. The leading and closing block rules match the
chat path change in #39852.
Agent harnesses send a lot of operational turns that relay or reformat tool
output rather than reason about it, and the cheapest built-in tier was SIMPLE.
NON_REASONING adds a rung below it, behind enable_non_reasoning_tier so an
already-deployed router cannot move.
The toggle is what keeps it safe. The tier set feeds the classifier rubric, the
response-format enum, the escalation ladder and the savings baseline, so a
default-on fifth tier would have changed what every existing router sends and
where its traffic lands. Off, the ladder, rubric, wire labels and baseline are
byte-identical to before. On, the rung is added at index 0, escalation walks up
out of it, and it can never win the savings baseline.
It requires an llm or custom classifier and a model of its own: the v1 score
ladder has no rung below simple_medium and the v2 artifact is trained on four
classes, so the heuristic scorers cannot produce the tier and a router that
enabled it there would pay for a bullet nothing reaches.
The dashboard follows the same flag, and the edit modal now reads the tier back
from the stored config rather than assuming four keys, since it rewrites tiers
wholesale on save and would otherwise delete a hand-written tier on any edit.
A post_call pipeline step whose guardrail only implements the older
async_post_call_success_hook used to skip the stream entirely: PR #38721
fails that shape open with a warning. The streaming step now assembles the
buffered stream into the response the hook expects, runs the hook, ends the
stream with the hook's exception when it raises, and delivers the hook's
rewrite through the same event write-back the unified guardrails use on
chat, Responses, and Messages streams (Messages gets the Anthropic shape).
A stream a pipeline manages no longer runs the same hook again after the
stream ends. A guardrail with neither the unified interface nor a post-call
hook keeps the fail-open, as does a rewrite the buffer cannot be patched
with.
CLI session tokens are in-memory only and never get a LiteLLM_VerificationToken
row, so the usage APIs could not resolve key_alias, team_id, or user_email for
their spend rows: the exact join and the reverse-hash recovery both miss. The
owner is written to LiteLLM_SpendLogs.metadata at request time under the same
hashed api_key, so read it back from there for keys still unresolved after the
token-table passes.
The lookup is sha256-gated like the existing reverse-hash recovery and bounded
to the records' startTime window (min date minus one day, max date plus two) so
it stays on the startTime index. No migration.
Also guard the window parser against the date=None rollup rows GROUPING SETS
aggregation emits, which raised TypeError from strptime and turned the
aggregated usage endpoints into HTTP 500s.
A POST /v1/responses with background: true returns a queued response, so the
post_call pipelines attached at submit time had nothing to inspect. They now
defer on queued and in_progress responses and run on GET /v1/responses/{id}
instead: the retrieval resolves the response id back to its deployment,
re-attaches the policies that governed the original model, and reports them
in the x-litellm-applied-* headers of the retrieval response.
SigV4 signing resolves AWS credentials, and botocore refreshes expiring
credentials inside that signing with a blocking HTTP call. Every async
Bedrock path that still signed on the event loop (/v1/messages, Converse,
count tokens, the agent-runtime and Comprehend Medical pass-throughs,
async-invoke status polling, realtime, AgentCore, SQS, S3) now signs on a
worker thread, so one Bedrock request no longer stalls the whole worker.
Fixes#40165
* fix(guardrails): keep guardrail telemetry when a policy pipeline blocks or modifies the response
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(guardrails): count every raw-snapshot guardrail evaluation and type the telemetry carry helpers
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test(policy_engine): type the recording guardrail hooks and telemetry test parameters
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Only the leading run of system messages (before the first non-system
message) is joined into the Responses `instructions` field. A system
message that arrives after a user, assistant, or tool turn now becomes a
system input item at its position, whether its content is a string or a
list, so a client that re-sends the same reminder as a string on the
next request produces byte-identical input and `instructions` stays
stable. Claude Code >= 2.1.237 appends such reminders after every user
turn, and folding them into `instructions` made Azure treat every
request as a cold prompt (cached_tokens 0 on every request of a
session).
Fixes#40198
The response.completed envelope carries its function_call output items as
SDK objects without a get shim, so the write-back skipped them and the
envelope still showed the original arguments after every stream event had
been rewritten. Write the item whenever one is present, and cover the typed
event shape the live proxy carries in the handler test.
Wrap the context-management editors, the end-of-stream chunk builder,
acount_tokens, the compression interception hook, the passthrough
interrupted-stream recovery, the A2A usage counters, and the semantic
cache embedding truncation in asyncify so a multi-megabyte payload no
longer stalls the worker's event loop while it is tokenized
The pass-through suite now drains the process-global logging worker
from an autouse conftest fixture so work queued on one test's loop
cannot fire against the next test's callbacks
Resolves LIT-7190
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
A post_call pipeline guardrail that rewrites a streamed tool call (its
arguments or its name) now has that rewrite written back across the buffered
chunks on chat, Responses, and Messages streams, so the client receives the
rewritten tool call instead of the original. The chat handler rewrites the
first fragment of each tool-call index and blanks the rest, the Responses
handler syncs the function_call output items and their argument events, and
the Messages handler rewrites the tool_use content_block_start and
input_json_delta events in both dict and SSE-bytes chunks.
The delivers_ended_stream_text_rewrites flag becomes
delivers_ended_stream_rewrites, since the write-back now covers both text and
tool calls, and the executor only discards a tool-call rewrite on translations
without write-back or on a shape the translation refuses.
Cooldown entries rode the router-wide DualCache, which re-reads a key that is
missing from memory at most once every 10s. A deployment benched on one replica
therefore kept taking traffic on its siblings for up to 10 seconds, and the same
shared in-memory tier could evict a live cooldown once 200 unrelated router keys
crowded it out, which sent even the benching replica back to the dead deployment.
CooldownCache now owns a DualCache over the router's Redis with a 1s read
interval and an in-memory tier that only holds cooldown keys. Redis is attached
lazily because the router builds the cooldown cache before it wires Redis up.
* fix(proxy): load db credentials inside the model reconcile so a worker never serves a model before its credential
* fix(proxy): load db credentials in the model read-through so a request miss never adds a model before its credential
* fix(proxy): read credentials from the writer db before the router update and look a credential up once
* test(proxy): assert the credential is loaded when db models reach the router instead of the call order
* fix(proxy): kill the whole prisma process group when a boot migration command times out
* fix(proxy): keep boot alive when the installed proxy extras has no prisma runner
The no-tools skip path in async_pre_call_hook now logs at DEBUG and the
denied-by-rule messages in async_pre_call_hook and _evaluate_tool_calls
log at INFO. The malformed tool arguments warning is unchanged. Adds
regression tests pinning each level
Co-authored-by: yucheng <yucheng@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(router): rank streaming latency routing by raw TTFT, not TTFT per token
Latency-based routing divided time-to-first-token by completion_tokens
before storing it, so a deployment that streamed a long answer looked
faster to first token than one that answered briefly. TTFT is now stored
as plain seconds (first token time minus request start) in both the sync
and async success handlers, which is what the routing decision compares.
Non-streaming latency normalization per output token is unchanged.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* fix(router): store streaming TTFT under a seconds-only cache key
Workers on the previous release keep writing seconds-per-token samples
under "time_to_first_token" in the shared router cache during a rolling
deploy, so mixing the new raw-seconds samples into the same list averaged
incompatible units. Raw TTFT now lives under "time_to_first_token_seconds"
and routing reads only that key.
Also fix the regression test's token counts: with 50 tokens on the fast
deployment and 500 on the slow one the old per-token formula picks the
slow deployment, so the routing assertion now catches the bug.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
* test(router): cover the TTFT sliding window from the unit-test shard
Move the TTFT list trimming checks from the CircleCI-only suite into the
mapped unit test file as one sync/async parametrized test, so the changed
lines in lowest_latency.py are exercised by the GitHub unit-test shard
that reports patch coverage.
Claude-Session: https://claude.ai/code/session_01Ttd5Q9ZhRPB4ch5guos3rj
