* fix(rerank): emit latency and cost headers on /rerank
Thread the logging object into the rerank httpx calls and pass hidden_params through to get_custom_headers, so x-litellm-overhead-duration-ms, x-litellm-response-duration-ms, x-litellm-response-cost, x-litellm-call-id and the LITELLM_DETAILED_TIMING x-litellm-timing-* headers show up on rerank like they do on chat completions
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(rerank): keep zero response cost in the /rerank cost header
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* ci: assign the new rerank endpoint tests to the proxy-endpoints shard
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: suppress TQ008 on the rerank header tests with reasons
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: milan <milan@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Co-authored-by: yassin <yassin@berri.ai>
* fix(proxy): make /team/member_delete's four cleanups atomic
The team roster update, the user.teams update, the team membership
delete, and the team-scoped verification token delete ran as four
sequential writes with no transaction around them, so a failure
between any two left the removal half applied. Thread a single
prisma transaction through all four writes, following the same
tx.<table> pattern /team/member_add and /team/member_update already
use, so either all four land or none do.
* fix(team): serialize member_add, member_delete, and delete under the team's advisory lock
/team/member_add validated a team exists and then wrote the user's teams array and
a membership row without holding anything across that gap, so a /team/delete could
commit its reference sweeps in between and leave a member pointing at a team id that
no longer exists. The write path already re-read members_with_roles under a row lock
before this change, but SELECT ... FOR UPDATE can deadlock with the access-group
endpoints, which lock an access group and then a team.
member_add now takes pg_advisory_xact_lock(hashtext(team_id)) before re-reading the
team and only writes if it is still there, so a delete that already committed is
visible before any write happens. delete_team takes the same lock around its own
row delete and reference sweep, so the two requests can never interleave: whichever
acquires the lock first runs to completion before the other's read can proceed.
Dropping the row lock from member_add's read also dropped the incidental protection
it gave against a concurrent member_delete, which still wrote from the snapshot it
validated against, unlocked, and could silently overwrite whatever member_add had
just committed. member_delete now takes the same advisory lock and re-reads the
roster under it before computing its own write, so it can never resurrect a member
by overwriting from stale data.
Resolves LIT-5544
* fix(team): run member writes on the advisory lock's transaction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team): keep member writes on the lock holder's connection after merge
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team): keep the transactional member create an upsert on user_id
The transaction path was creating the email-identified user row outright, where the
regular client path upserts on user_id. Share one upsert helper between both member
paths so the create stays idempotent on the lock holder's connection.
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(team): read member_delete's user and key rows on the lock-holding transaction
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Tools passed for Together models with no supports_function_calling entry in
model_prices_and_context_window.json were rejected with UnsupportedParamsError
by default and silently dropped under drop_params, which made models emit tool
calls as plain text. Fail open instead: pass tool params through with a warning
and let Together validate. Models the registry explicitly marks as not
supporting function calling keep the loud contract: raise by default, drop with
a warning under drop_params.
anthropic_messages goes through _ageneric_api_call_with_fallbacks rather
than _acompletion, so its returned streaming iterator was never wrapped
by the chat-completions fallback handler. A retriable SSE event: error
frame (overloaded_error, internal_server_error) from a native
Anthropic/Bedrock passthrough passed through to the client unchanged,
and a MidStreamFallbackError raised by the completion-bridge path's
CustomStreamWrapper propagated unhandled.
Add _aanthropic_messages_streaming_iterator, mirroring
_acompletion_streaming_iterator: it detects a retriable SSE error event
via the new parse_anthropic_error_event helper, raises
MidStreamFallbackError once real generated content (a content_block_delta
frame) has not yet reached the caller, and re-enters the Router's
fallback chain. A MidStreamFallbackError raised directly by the source
iterator (the completion-bridge path) is gated the same way via its own
is_pre_first_chunk flag. The raised MidStreamFallbackError carries a
status-coded original_exception built from the parsed error type, so
status_code/cooldown logic sees the real 429/500/503/etc. instead of a
hardcoded 503.
Lifecycle/bookkeeping frames (message_start, content_block_start, ping,
...) never disqualify a fallback attempt by themselves, since Anthropic
routinely sends message_start before an overload error - but they are
buffered rather than forwarded immediately, since forwarding one and
then appending a fallback attempt's own message_start would produce two
overlapping message lifecycles on one SSE stream. Buffered frames flush,
in order, once real content arrives or the stream ends without error.
Once real content has streamed, or the error is a non-retriable 4xx, the
chunk (or exception) is forwarded as-is rather than starting a second
lifecycle. Content and error coalesced into a single physical read are
handled the same way: once the client has genuinely received the content
(bundled in that same forwarded chunk), no fallback is attempted. A
`ping` keepalive is dropped outright before any real content arrives
(it recurs indefinitely on a slow-starting connection and carries
nothing worth buffering), and the pre-content lifecycle buffer is capped
at MAX_BUFFERED_PRE_CONTENT_ANTHROPIC_CHUNKS, forcing an early commit to
the primary stream so a hostile or pathological upstream can't grow it
without bound. is_anthropic_ping_chunk only matches a chunk whose every
event: line is event: ping, so a ping coalesced with real content or a
retriable error into one physical transport chunk is never dropped.
The fallback request kwargs also deep-copy nested litellm_metadata/metadata
(matching the Responses API path) so the primary attempt's
deployment-specific fields never leak into the fallback request, and the
fallback deployment's own provider headers are merged onto the wrapper's
_hidden_params so they still reach the client/logging pipeline. A
fallback that resolves to a non-streaming response (e.g. an agentic
tool-use interception loop) is synthesized into a real Anthropic SSE
event sequence via the new anthropic_messages_response_as_sse_events
helper, instead of yielding a raw dict into the byte stream - including
a trailing signature_delta for a thinking block, and a message_start
whose stop_reason/stop_sequence/output_tokens stay null/zero the way a
real stream's does instead of leaking the completed response's final
state.
Resolves#24004
* fix(scim): return user_id as Group members[].value on transformed group responses
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* style: format scim transformation tests per ruff
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* test: drop redundant assertion comment
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: retrigger ci
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(dashboard): don't show a stale provider prompt-cache chip on a response-cache hit
The playground's non-streaming chat completion and responses paths replayed a cache hit's original usage payload verbatim, so ResponseMetrics kept rendering the provider's prompt-cache-write/read chips using token counts from the original request. Detect the hit via the x-litellm-cache-key response header and render a Response Cache indicator instead.
* fix(dashboard): expose x-litellm-cache-key through CORS for the playground cache-hit indicator
* fix(proxy): stop expected 4xx responses from saturating worker CPU on failure logging
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* fix(proxy): keep threaded sync failure handler so CustomLogger sync callbacks still run
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
* chore: remove redundant comments per review
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
---------
Co-authored-by: yassin <yassin@berri.ai>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
The edit form's Authorize & Fetch Token button built its temporary OAuth
session payload without issuer, authorization_url, token_url, or
registration_url, unlike the create form's equivalent payload builder. The
backend's temporary-session endpoint builds its ephemeral server purely from
that payload, so any admin-configured OAuth endpoints on an existing server
were silently dropped, endpoint discovery fell back to (and failed against)
the plain server url, and Authorize & Fetch Token 400'd with "authorization
url is not configured" even though the saved server had those fields filled
in. Add the four missing fields to the edit form's temporary payload builder,
mirroring the create form.
