Batch creation snapshotted the team's organization with a direct
litellm_teamtable query on every create. Go through get_team_object
instead, which serves the team auth already cached and only falls back
to the database when the team was never cached.
Suppression state moves out of request metadata into a request-scoped ContextVar.
refresh_proxy_server_request_body_snapshot copies metadata into
proxy_server_request.body, which deployments persist to spend logs, so the marker
naming each suppressed guardrail was readable by the caller whose request produced
it. Recovering it was enough to replay {token}:{name} for any CustomGuardrail and
switch off a PII or content-filter guardrail, since the check never verified the
named guardrail was a compression one. Nothing is read from metadata now, so there
is no marker to forge and the per-process token is no longer needed.
Routing-side compression reads the live messages instead of a pre-guardrail copy.
arm_pre_call runs before the pre-call hook, so its snapshot held the prompt as it
was before any masking guardrail rewrote it, and messages_for_routing handed that
to a compression guardrail which POSTs it to an external service. Masked content
left the proxy anyway. The cost is one combination: when the model hop compressed
and the hops differ, routing now classifies on the compressed text, since no
uncompressed copy survives that a masking guardrail has already seen.
policy_for_model no longer falls back to a marker scoped to tags the request does
not carry, which applied an 'eu' policy to a 'us' request on config order alone.
Each fix carries a regression test; all three fail when the fix is reverted.
client_ack_messages classified a websockets ConnectionClosed raised by
the client socket as the backend closing, so bidirectional_forward kept
waiting on the upstream instead of ending the session. Starlette clients
raise WebSocketDisconnect, but the realtime test client in
tests/llm_translation/realtime raises websockets.exceptions.ConnectionClosed,
which hung test_openai_realtime_simple.py until the run was killed.
Only the receive_text call now maps every exception to
CLIENT_DISCONNECTED; the loop body keeps ConnectionClosed as
BACKEND_CLOSED, since the backend socket is the only websockets socket
touched there.
The detail endpoint now returns untracked_usage_units_by_team and
untracked_usage_units_by_key next to the cost breakdowns, and the By team and
By key tables show them in an Unpriced Units column, so a row that pairs its
total units with a partial cost says how many units that cost leaves out.
The overview comparator no longer treats a missing cost as zero: guardrails
with no known cost sort last in both directions instead of mixing in with
genuinely free ones.
Refs LIT-5652
The overview table gains Usage Units and Cost columns plus a Guardrail Cost
card, and the detail page gains a Usage & Cost section that breaks units and
cost down by counter, team and key. Units the cost map could not price are
called out next to the cost they are left out of.
Both pages now read /guardrails/usage/* through $api.useQuery so the rows are
typed from schema.d.ts; the hand-written PerformanceRow and the untyped fetch
helpers are gone. fetchClient resolves fetch per request so integration tests
that stub the global see typed-client calls too.
Refs LIT-5652
When the upstream closes while the proxy is forwarding a client message,
the client loop ends before the backend relay sees the close, and the
relay skipped closing the client because it read the client loop's exit
as the client hanging up. The client loop now reports why it stopped, so
a close observed on the backend send still reaches the client with the
error event and the upstream close code
When the provider closes the realtime websocket (for example Vertex Live
refusing the session with 1008 "Publisher model ... was not found"), the
proxy swallowed the close and kept waiting on the client, so the client
sat on an open socket with nothing coming back and the session was logged
as a $0 success
The backend relay now returns the upstream close, and bidirectional_forward
sends the client an OpenAI-style error event naming the upstream code and
reason, then closes the client socket with the same code (or 1011 when the
upstream code is one a server may not send). A session the upstream refused
before sending any frame is logged through the failure handlers instead of
as a success
OpenAI and Azure realtime usage reports output_tokens == text_tokens + audio_tokens
with reasoning_tokens counted inside text_tokens, so generic_cost_per_token billed
the reasoning share twice. When the output token details sum past completion_tokens,
the nested reasoning overlap is now subtracted from text_tokens before pricing;
shapes where text_tokens already excludes reasoning are unchanged.
The classifier scores extracted text, so a turn whose complexity lives in
its image is invisible to it: a screenshot of a stack trace classifies on
its caption, and an image-only turn flattens to empty text and never
reaches the classifier at all.
classifier_llm_config.vision opts in, off by default, with max_images
bounding what one turn can add. Images are still dropped when the
classifier model is declared supports_vision false. Anthropic and
Responses image parts are rewritten into chat-completions dialect before
they reach the classifier call, since /v1/messages hands the pre-routing
hook its own dialect untranslated.
The local scorer no longer short-circuits heuristic_first or hybrid on a
turn carrying forwarded images, because it reads text alone and its
confidence describes a request it has only partly seen.
The completed-batch early return skipped both the cancel and the list
assertion while the lifecycle's covers markers still credited both cells.
List does not depend on the batch being cancellable, so it now runs either
way; cancel on a completed batch stays a documented vacuous pass
Following S3 continuation tokens let GET /v1/files walk an entire managed prefix however large it grew, and the follow-up page fetches dropped the caller's timeout. The handler now stops once MAX_FILE_LIST_LIMIT files are collected (10,000, the most OpenAI returns per list call), slicing the last page to fit, and hands the request timeout to the first and every later page fetch. MAX_FILE_LIST_LIMIT moves to litellm.constants so the proxy's limit validation and the handler share one number
