mirror of
https://github.com/BerriAI/litellm.git
synced 2026-10-10 03:28:53 +00:00
221 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
yuneng-jiang
|
f8a2ea7378
|
Merge pull request #31426 from BerriAI/litellm_/cranky-hamilton-21b5d0
fix(ui): stop Request Logs page from overflowing horizontally and size its columns |
||
|
Yuneng Jiang
|
256b5aadfb
|
fix(ui): revert Request ID width to default, tighten Session ID
Drop the explicit size on Request ID so it falls back to the default width like the other reverted columns. Narrow Session ID from 160px to 120px since its truncated value needs less room |
||
|
Yuneng Jiang
|
84d7a32020
|
fix(ui): revert Duration and TTFT column widths to default
The explicit 90px/80px sizes were too narrow for the Duration (s) and TTFT (s) headers once the sort arrows were factored in, cramping the header labels. Dropping the size lets these two columns fall back to the default width like before |
||
|
Yuneng Jiang
|
76be4461ca
|
feat(ui): give Request Logs columns explicit widths and tighten the dense ones
Now that the page-overflow bug is fixed by letting the main pane shrink, bring back per-column sizing purely to control widths. Columns declare explicit pixel sizes and the table derives its min-width from getCenterTotalSize(), so it stretches to fill a wide card but scrolls once the columns no longer fit. The shared DataTable applies this only when columns declare sizes, leaving the other consumers on their existing fluid layout Trim the columns that were eating horizontal space without earning it: Request ID and Key Hash drop ~30% (Key Hash now narrower than Key Alias, which is the more useful of the two), and Duration and TTFT shrink to fit their short numeric values |
||
|
Yuneng Jiang
|
014754be94
|
fix(ui): let dashboard main pane shrink so wide tables scroll instead of overflowing
The Request Logs page pushed the whole page past the viewport horizontally. The cause was the app shell flex layout: <main className="flex-1"> is a flex item, and flex items default to min-width: auto, so they refuse to shrink below their content's intrinsic width. The logs table is intrinsically ~2300px across its 16 nowrap columns, so main grew to that width and dragged the page with it; the table's own overflow-x-auto wrapper never got the chance to scroll Add min-w-0 to main so it can shrink to the available width, at which point the existing overflow-x-auto wrapper engages and the table scrolls inside its card. This applies to every dashboard page, not just logs Also drop the dead max-w-screen class on the logs container (not a real Tailwind utility, so it was a no-op), and revert the earlier column-sizing attempt which targeted table-layout rather than the actual containment problem |
||
|
Sameer Kankute
|
133da06aa3
|
chore: litellm oss staging (#31185)
* fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped
The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.
Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
switch the requests chart to the shared valueFormatter so it uses the
same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
every formatted label at most 7 chars.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* docs(readme): add Deploy on AWS/GCP with Terraform section
Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.
Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): add 1-click deploy buttons for AWS + GCP
GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.
AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* docs(readme): move AWS + GCP deploy buttons next to Render button
* docs(readme): unify deploy button sizes and badge styles
* docs(readme): bump deploy button height to 48 to match Render/Railway
* docs(readme): bump AWS/GCP badge height to compensate for SVG padding
* docs(readme): bump AWS/GCP badge height to 72
* docs(readme): bump AWS/GCP badge height to 84
* fix(readme): make deploy buttons same height (48px)
https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc
* docs(readme): flag GCP project ID substitution in image_registry
* docs(readme): equalize deploy button heights and fix Cloud Shell button font
GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.
Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.
* docs(readme): collapse Railway deploy anchor to a single line
The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.
* Add Claude Fable 5 cost map entries as a data-only hotfix
Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.
https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm
* fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano
Three bugs in model_prices_and_context_window.json:
1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens
were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K
max output, but the values were set as max_input=128000,
max_tokens=272000. This caused token limit errors when sending
prompts over 128K tokens to GPT-5 Pro.
2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was
272000, but GPT-5.4 Mini shares the same 1,050,000 token context
window as GPT-5.4. This was inconsistent with the azure/ variants
which already correctly had 1,050,000.
3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini,
max_input_tokens was 272000 instead of 1,050,000.
Source: OpenAI model documentation and contextwindows.dev which
aggregates official context window sizes.
Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini
should be 400K; their 272K values are correct per OpenAI docs)
* fix: also correct max_output_tokens for gpt-5-pro (272000→128000)
Per reviewer feedback, max_output_tokens was left at 272000 while
max_tokens was corrected to 128000, causing an internal inconsistency.
Both should be 128000 per OpenAI docs.
* fix(cost): price gpt-image generated output tokens as image tokens (#31147)
The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return
usage with no output token breakdown — litellm's `ImageUsage` has no
`output_tokens_details` field — so generated-image OUTPUT tokens were priced at
the text rate (`output_cost_per_token`) instead of the image rate
(`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x
undercount on the dominant cost component (image output is ~74% of spend). This
also affects azure gpt-image, which shares this calculator.
The OpenAI gpt-image cost calculator re-implemented usage handling instead of
reusing `calculate_image_response_cost_from_usage`, the shared helper that
azure_ai/gemini/vertex_ai already use. That helper classifies generated output
tokens as image tokens when the provider does not itemize output, and splits
text/image when it does.
Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage`
(pre-transformed chat Usage objects are still costed directly). Adds a regression
test for the no-breakdown ImageUsage case (gpt-image-2).
* fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098)
A bare application-inference-profile ARN passed as bedrock/arn:... fell
through to the invoke route, which cannot derive a provider from the
opaque profile id and raised 'Unknown provider=None'. The converse route
needs no provider, so detect these ARNs in get_bedrock_route and route
them to converse, matching the behavior of the already-documented
bedrock/converse/arn:... workaround.
Explicit invoke/ prefixes still win, and they remain a dead end for these
ARNs by design (no provider derivable). System-defined inference-profile
ARNs that embed a known model, and other opaque ARN types
(provisioned-model, imported-model, custom-model-deployment) that are
frequently invoke-only, are deliberately left on their current routes;
tests guard both boundaries.
* fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060)
_add_tool_choice_required_message appended the "select a tool" prompt to
the caller's messages list in place, so transform_request corrupted the
caller's conversation history and appended a duplicate prompt on every
retry. Build and return a new list instead so the call stays idempotent.
Adds a regression test asserting the input messages list is unchanged
across repeated transform_request calls.
Co-authored-by: Wassbdr <wassim.badraoui07@gmail.com>
* fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996)
gpt-4o-transcribe and compatible ASR backends return a diarized_json
response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8.
TranscriptionUsageDurationObject typed seconds as int, so parsing the
response raised a pydantic ValidationError (int_from_float). That error
surfaces as an APIConnectionError which the router treats as retryable, so
it keeps re-calling the upstream (200 every time) until the upstream
rate-limits and returns 429 to the caller.
OpenAI specs this field as a float (see openai SDK UsageDuration.seconds),
so widen seconds to float. With the parse succeeding there is no exception
left to retry, which removes the loop.
Co-authored-by: Neimar Avila <19142978+neimaravila@users.noreply.github.com>
* fix(deepseek): drop non-function tools before chat completions call (#30910)
* fix(deepseek): drop non-function tools before chat completions call
DeepSeek's /chat/completions only accepts tools of type "function".
Requests bridged from /v1/responses can carry responses-API-native tool
types, for example a Codex CLI tool typed "namespace", which DeepSeek
rejects with "unknown variant 'namespace', expected 'function'" so the
whole request fails (issue #30722).
Filter unsupported tool types in the DeepSeek request transform so the
function tools still go through; when nothing callable remains, also drop
the now-dangling tool_choice and parallel_tool_calls
Fixes #30722
* test(deepseek): cover async tool filtering and document tool_choice assumption
Add an async_transform_request regression test so the sync and async tool
filtering paths cannot silently diverge, and document in _drop_unsupported_tools
that only non-function tools are dropped, so a function-named tool_choice always
references a surviving tool
* feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840)
* feat(ui): surface team budget on key overview when key has no own budget (#30801)
* feat(ui): surface team budget on key overview when key has no own budget
* fix(ui): replace IIFE with derived variable and use find() for team budget display
* fix(anthropic): emit replayable streaming thinking blocks (#31022)
* feat(proxy): read cold-storage prompts back in the logs detail view (#30364)
* feat(proxy): read cold-storage prompts back in the logs detail view
When a deployment offloads prompts and responses to cold storage instead of
Postgres, the spend-log row holds only "{}" placeholders plus a
metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed
nothing. The detail endpoint only read the placeholder columns and never
fetched the object back.
Resolve the payload per row based on actual content, not a config flag: if
Postgres has content, return it; otherwise read the exact stored object key and
fetch from the configured cold storage backend through ColdStorageHandler.
Reading the persisted key is a single GET. The key embeds a microsecond
timestamp that cannot be reconstructed from the millisecond-precision startTime
column, and listing the day's prefix to match on request_id would be too
expensive for this per-open path.
Also teach the detail drawer's pretty-view parser to accept a bare messages
array. The cold storage payload carries the prompt as a top-level messages list
with no proxy_server_request, so without this the output rendered while the
input stayed blank.
ColdStorageHandler gains an optional injected logger so the resolver can be unit
tested without monkeypatching. Postgres-stored prompts are unaffected: the fast
path returns the existing columns and the request-body object still renders the
same way.
* Update litellm/proxy/spend_tracking/spend_management_endpoints.py
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure
Add unit tests for ColdStorageHandler (injected logger, graceful None when no
logger is configured, and resolution of a configured logger from the callback
registry) and a regression test asserting a cold storage backend exception
degrades to the Postgres values instead of surfacing a 500.
---------
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068)
* fix(mavvrik): advance metricsMarker after upload + fix scheduler startup
Two bugs fixed:
1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a
successful GCS upload, so metricsMarker stayed at 0 and every daily run
re-exported the same dates in an infinite catch-up loop.
Fix: add _update_metrics_marker(date_epoch) called at the end of deliver()
after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS
file is already committed). A 410 raises consistent with the rest of the
destination.
2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call
has triggered lazy instantiation of MavvrikFocusLogger, so it found no
logger instance and silently skipped registering the daily export job.
Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call
_init_custom_logger_compatible_class to force instantiation before
the APScheduler job is registered.
* fix(mavvrik): catch up from earliest window when metricsMarker=0
When the connector is freshly registered, metricsMarker=0 parses to None.
The catch-up block was guarded by `if last_ingested and ...` which skipped
it entirely for None, so only yesterday was exported instead of the full
_MAX_CATCHUP_DAYS window.
Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup).
The existing > 7 day warning only fires for non-None markers that are old.
* fix(mavvrik): use now as end_time for yesterday's export window
LiteLLM_DailyUserSpend rows for a given date get their updated_at
bumped by the spend flush job throughout the next morning. The core
database query filters on updated_at, so capping end_time at midnight
(yesterday + 1 day) missed any spend rows flushed after midnight.
Fix: pass now (cron fire time) as end_time for the daily "yesterday"
window so all fully-settled rows are captured regardless of when the
flush job ran.
Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per
row in the exported FOCUS CSV.
* fix(mavvrik): also use now as end_time for catch-up windows
* fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class
Calling it with only logging_integration raised TypeError at proxy startup
because internal_usage_cache and llm_router have no defaults. Also fix test
name to reflect the actual status code (5xx not 4xx) used in the mock.
* ci: retrigger CI run
* feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757)
* Add optional `instruction` passthrough to the rerank API
vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction`
field (folded into the model's chat_template_kwargs and consumed by the
chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently
dropped it: RerankRequest / OptionalRerankParams had no such field, so the
outgoing body was rebuilt without it.
Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(),
get_optional_rerank_params, and the hosted_vllm transformation into the
request body, only when non-None. When callers omit it, model_dump(exclude_none)
drops the field and the outgoing request is byte-for-byte unchanged — fully
backward-compatible. (DeepInfra already forwards `instruction` via
non_default_params; this formalizes the field in the shared types.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review: thread `instruction` as a typed param + cover rerank_utils
Per PR review (greptile P2 + codecov):
- Make `instruction` a typed, named argument on the rerank provider interface
instead of recovering it from the opaque `non_default_params` blob. Adds
`instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params`
and every provider override, and forwards it explicitly from
`get_optional_rerank_params`. hosted_vllm now reads the named param directly.
It is still also surfaced in `non_default_params` so providers that read it
there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction`
as a named param rather than leaving it in **kwargs.
- Add get_optional_rerank_params unit tests (present + absent) to cover the
previously-uncovered threading line flagged by codecov.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: scan rerank `instruction` through request guardrails
The rerank guardrail translation (CohereRerankHandler.process_input_messages)
only scanned `query`, so the newly added `instruction` field reached the
backend model unscanned. Since instruction-aware rerankers (hosted vLLM /
Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller
could place content there to bypass configured rerank request guardrails.
Generalize the handler to scan every user-controlled text field (`query` and
`instruction`) in one apply_guardrail call and write each sanitized value back
by index. Query-only requests are unchanged (single-element list at index 0);
non-string fields are left untouched. Adds tests covering instruction
scanning, PII masking write-back, and the non-string case.
Addresses the Veria AI security review on PR #30757.
* test: narrow Optional results before len() to satisfy basedpyright budget
The lint gate (basedpyright delta-vs-base budget) flagged one new
reportArgumentType: len(result.results) where results is
List[RerankResponseResult] | None. Assert results is not None first to
narrow the type before len()/indexing.
* fix: read rerank `instruction` from kwargs to satisfy basedpyright budget
The basedpyright delta-vs-base gate flagged one new reportArgumentType: the
Router forwards rerank calls via an untyped `**kwargs` unpack
(`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a
typed named param on the public `rerank`/`arerank` entrypoints made pyright
check that key against `str | None`, adding an error at router.py with no real
safety gain. Read `instruction` from kwargs in `rerank` instead.
It remains fully typed where it matters - threaded as a typed argument through
`get_optional_rerank_params` and each provider's `map_cohere_rerank_params`
(the original Greptile P2 ask). Whole-repo reportArgumentType is back to the
base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(github_copilot): synthesize empty choices at the provider seam (#30929)
Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with
choices=[], either carrying Anthropic-native content blocks or, for the
max_tokens=1 probe Claude Code sends, no content at all. github_copilot
is dispatched through the OpenAI SDK handler, which calls
convert_to_model_response_object directly and never invokes
GithubCopilotConfig.transform_response, so the empty-choices guard there
surfaced as a 500
Instead of synthesizing choices inside the shared
convert_to_model_response_object (which would silently turn empty choices
into a fabricated success for every provider), add a no-op
transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig
overrides it to synthesize choices from Anthropic-native content, reusing
its existing parsing, and the OpenAI SDK handler routes its parsed
response through the hook before generic conversion. The core utility
keeps treating empty choices as an error for all other providers
Fixes: https://github.com/BerriAI/litellm/issues/30927
Signed-off-by: David J. M. Karlsen <david@davidkarlsen.com>
* fix(router): stop fallback lookups from mutating the router fallbacks config (#30624)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)
* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens
* test: scope local cost map env var with monkeypatch to avoid test pollution
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)
* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold
_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.
mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.
* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers
Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.
Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.
* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview
MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.
* fix(mcp_debug): mask short auth values in debug headers instead of echoing them
Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.
* test(mcp_debug): assert masked short value preserves length
* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)
Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.
Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:
- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
ProviderConfigManager.get_provider_audio_transcription_config() in
litellm/utils.py; update the stale comment in
get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
get_supported_openai_params() in
litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
model_prices_and_context_window.json and
litellm/model_prices_and_context_window_backup.json (both had
mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
imports from tests/llm_translation/test_fireworks_ai_translation.py
No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.
* feat: add darkbloom provider (#30876)
* feat: add darkbloom provider
* fix: document darkbloom provider endpoints
* fix: address darkbloom review feedback
* fix: update darkbloom tool metadata
* fix: fail fast for non-Postgres database URLs (#30883)
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup
LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.
Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.
Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.
Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.
Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.
* fix: resolve CI failures and proxy DB URL typing issue
* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging
* Validate DIRECT_URL alongside DATABASE_URL startup guards
* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)
* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)
* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)
* style(bedrock): black-format stream-error helper (#24608)
* fix(mcp): re-land native tool preservation with typed annotations (#30645)
* fix(mcp): preserve native tools in semantic filter hook with typed annotations
* fix(mcp): tighten _is_mcp_tool Chat Completions shape check
* fix(sambanova): return embeddings supported params instead of dropping them (#30937)
* fix(router): send fallback metadata when streaming (#30914)
When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:
1. The response now correctly populates the fallback headers
(`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
to the client (opt-in) by passing `include_fallback_errors: true` in
the request.
The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.
* fix(mistral): drop output-only reasoning fields from input messages (#30884)
LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.
Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)
* fix(perplexity): bill search queries at the per-request price, not 1/1000
The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").
The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.
Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.
* test(perplexity): update integration test search-cost expectations to per-request
The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.
* test(perplexity): drop unused mock imports flagged by ruff
* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)
* fix(fireworks_ai): return None for transcription in get_supported_openai_params
Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.
* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting
Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.
Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.
* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test
The operator gate added in
|
||
|
Yuneng Jiang
|
4a6f0dbd8c
|
fix(ui): size Request Logs table columns so it scrolls instead of overflowing
Tremor's Table forwards className to a wrapper div rather than the inner table element, so the table-fixed class never reached the table and it stayed table-layout: auto. Across 16 whitespace-nowrap columns that expanded the table far past the viewport Give each spend-logs column an explicit pixel size and drive the table width from getCenterTotalSize(), matching the Virtual Keys table. The shared DataTable applies this only when columns declare sizes, so the other consumers keep their existing fluid layout |
||
|
ryan-crabbe-berri
|
fa307fe9e5
|
fix(ui): render logos under a custom server_root_path (#31156)
The App Router migration moved pages to deeper path segments and the proxy can be mounted under a sub-path (e.g. /litellm behind a reverse proxy). Local logo asset paths were emitted without the server root prefix, so they resolved off the origin root and 404'd. Route every local logo src through a single resolver that prefixes the live server root path and leaves external URLs untouched, fixing provider, guardrail, vector store, callback, MCP and audit-log logos at any route depth and root path. |
||
|
ryan-crabbe-berri
|
ee5b2a367d
|
fix(ui): label request logs column "Key Alias" to match filter (#31037)
The request logs table column displayed "Key Name" while its accessor (metadata.user_api_key_alias) and the corresponding filter both use the "Key Alias" label; this aligns the column header with that naming. |
||
|
ryan-crabbe-berri
|
1fab5c4ca6
|
chore(ui): remove dead UI components unreferenced by any page (#30340)
These components are not reachable from any page: WebRTCTester, the agents agent_card/agent_card_grid pair (the agents route renders a Table directly), and view_logs ErrorViewer/RequestResponsePanel (logs uses inline equivalents in columns.tsx). Each was imported only by its own test, so the tests go too. Also drops the now-stale agent_card_grid vi.mock in agents.test.tsx and the WebRTCTester eslint-suppressions entry. |
||
|
Sameer Kankute
|
3b40ac987f
|
Litellm oss 090626 (#30021)
* fix(mcp): report scoped server name during initialize (#29865) * fix mcp scoped server name * Update litellm/proxy/_experimental/mcp_server/mcp_context.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(mcp): cover scoped server name in the SSE initialize handler --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): show all session logs in the drawer, not just the first 50 (#29795) * fix(ui): show newest session logs first * test(ui): keep session log pagination coverage * fix(ui): show all session logs in the drawer, not just the first page The session detail drawer fetched session logs via sessionSpendLogsCall without page/page_size, so it only ever received the backend default of one page (50 rows). Sessions with more than 50 calls had the rest unreachable in the UI (#29153). sessionSpendLogsCall now takes page/page_size, and the drawer fetches the first page, reads total_pages, then fetches the remaining pages and accumulates them before the existing client-side sort. This keeps the single continuous list (and the selected-log lookup and keyboard navigation, which all assume the full session) correct. Fetching is bounded by a page cap, and the sidebar shows a "showing most recent N" note if a session exceeds it. The rows are lightweight metadata (the endpoint excludes messages/response), so the full set is small; request/response bodies are still loaded per log on demand. * fix(ui): default session drawer to most recent log, newest first Open a session with its most recent log selected, and order the sidebar newest-first to match the all-sessions logs overview. MCP calls stay grouped last. The latest log by time is computed explicitly, since the MCP grouping means it is not always the first row. * Apply fetching pages in batches suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): derive session total from accumulated rows when backend omits it Compute the session total after all pages are fetched, falling back to the accumulated row count rather than the first page's. Guards the truncation note against a backend response that omits total but spans multiple pages. --------- Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): handle Mistral multipart passthrough (#29927) * fix(proxy): handle Mistral multipart passthrough * chore: satisfy passthrough ci formatting * test(proxy): cover Mistral passthrough in CI shard * fix(vertex_ai): use REP host for context caching on eu/us multi-region endpoints (#29573) Context caching built the cachedContents URL as https://{location}-aiplatform.googleapis.com, which is an invalid host for the eu/us multi-region endpoints and returns 404. The inference path already resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com) via get_vertex_base_url(); reuse that helper in _get_token_and_url_context_caching so caching uses the same host as inference. Adds tests covering the eu/us multi-region cachedContents URLs (v1 and v1beta1). Fixes #29571 * Support per-model encrypted content affinity config (#29760) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: propagate upstream status code in proxy API exception handler (#29402) * fix: propagate upstream status code in proxy API exception handler When Google GenAI / Vertex returns a 404 for deprecated or missing models via streamGenerateContent, the exception was falling through to a generic handler that defaulted to 500. Now provider exceptions carrying a valid HTTP status_code correctly propagate it through to the ProxyException. * fix: apply black formatting to common_request_processing.py * fix: tighten status code range to 400-599 and deduplicate ProxyException raise * fix(tests): use valid vertex_location in context caching tests Replace "test_location" (contains underscore) with "us-central1" so tests pass the regex validation added in get_vertex_base_url(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sdk): add xAI OAuth provider (#29866) * Add xAI OAuth provider * Update oauth.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix xAI OAuth CI failures * Add xAI OAuth coverage tests * Move xAI OAuth coverage tests to core utils * Address xAI OAuth review comments * Prevent xAI OAuth api_base token exfiltration * Treat blank xAI OAuth api keys as absent * Wrap invalid xAI OAuth JSON responses * Use xAI OAuth behind explicit flag --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy) #27734 allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update (#27751) * fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update Fixes #27734 Sending null for budget_duration, team_member_budget, team_member_budget_duration, team_member_rpm_limit, or team_member_tpm_limit via /key/update or /team/update returned 200 OK but silently ignored the null value. The fields remained unchanged in the database. Root causes: - /key/update: prepare_key_update_data() popped budget_duration from the update dict but never re-added it (or budget_reset_at) when the value was None. - /team/update: _set_budget_reset_at() only acted when budget_duration was non-None, leaving a stale budget_reset_at in the DB. - /team/update: team_member_* null values bypassed the budget table update entirely because should_create_budget() requires at least one non-None field. * test(proxy): cover no-budget-row path in clear_team_member_budget_fields * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes (#30028) * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes When output_parse_pii=true on the Anthropic native path (anthropic/claude-*), response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with the original values before reaching the caller. Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta / text_delta events, and apply _unmask_pii_text before re-encoding. Wire it into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist. * fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks don't bypass it and silently swallow a JSONDecodeError. Add ensure_ascii=False to json.dumps so non-ASCII replacement values like accented names are preserved as UTF-8 on the wire rather than being \uXXXX-escaped. Add regression tests for both cases. * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) (#29925) * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) Bedrock Mantle serves the Responses API on two upstream paths: - gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses - every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses; any model declared mode=responses (price-map entry or user model_info) -> /v1/responses; everything else returns None and keeps the existing chat-completions emulation. Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests. * feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path Addresses the Greptile review point that frontier detection should be a price-map field rather than a hardcoded name match. The gate now routes a model to /openai/v1/responses when its price-map entry declares use_openai_responses_path, so a frontier model whose name does not follow the openai.gpt- convention can be onboarded by JSON alone. The name-convention check is kept as a fallback that needs no price-map entry, which preserves zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4 get the flag in both price maps. Adds tests for the data-driven flag path and for the flag presence on the gpt-5.x entries; both branches are mutation-tested. * test(model_prices): allow use_openai_responses_path in price-map schema The model_prices_and_context_window.json schema validator (test_aaamodel_prices_and_context_window_json_is_valid) enforces additionalProperties: false, so the new use_openai_responses_path flag on the gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean, alongside the other supports_* / capability flags. * Add Tensormesh serverless models to the model cost map (#30037) * Add Tensormesh serverless models to the model cost map * Flag reasoning support on the Tensormesh models that expose thinking mode * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update (#30001) * fix(proxy): reconcile stale key spend counter after budget reset * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update * fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass * revert: undo unrelated formatting changes in enterprise directory * test(proxy): add unit test for key spend update invalidating counter * test(proxy): fix mocked update_data and hash token expectations in unit test * fix(proxy): use Responses-API transformer in pass-through cost tracking (#29728) The `elif is_responses:` branch of `openai_passthrough_handler` was calling the chat-completions `transform_response` on a Responses API payload. The chat-completions transformer expects `choices: [...]` in the raw response; the Responses API uses `output: [...]` and `usage.input_tokens` / `usage.output_tokens` (not `prompt_tokens` / `completion_tokens`). The result was a KeyError 'choices' deep inside `convert_to_model_response_object`, swallowed by the surrounding `except Exception` in the handler, and the SpendLogs row was written by the fallback path with zeroed-out tokens, spend, and model. This bug silently undercounts cost for every successful pass-through call to either OpenAI's `/v1/responses` or Azure's `/openai/v1/responses` (deployments configured for the Responses API). Reproduced 2026-06-04 against a real Azure OpenAI Responses API deployment proxied through LiteLLM v1.88.0. Fix: use the dedicated `OpenAIResponsesAPIConfig.transform_response_api_response` for the Responses branch. This transformer already exists in LiteLLM (`litellm/llms/openai/responses/transformation.py`) and knows the Responses-API on-the-wire shape. `litellm.completion_cost` already handles `ResponsesAPIResponse` natively with `call_type="responses"`, so no downstream changes are needed. Tests: test_responses_api_uses_responses_transformer_not_chat_completions NEW. Real regression test — exercises the openai_passthrough_handler with a real-shaped Responses payload (no `choices`, has `output` and Responses-API `usage` keys) and NO mocked `get_provider_config`. Pre-fix: raises KeyError 'choices' inside the chat-completions transformer (the bug). Post-fix: returns a ResponsesAPIResponse, completion_cost is called with call_type="responses" and a ResponsesAPIResponse instance (asserted). Verified to fail on un-fixed handler + pass on fixed handler before commit. test_responses_api_cost_tracking UPDATED. Old test mocked `get_provider_config` (no longer called in the responses branch post-fix). Now mocks the Responses transformer directly (`OpenAIResponsesAPIConfig.transform_response_api_response`) to test the downstream cost-calc contract. Out of scope for this PR (separate followup): - Recognizing *.cognitiveservices.azure.com (the newer Azure OpenAI hostname) in the is_openai_*_route checks. Separate PR. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(skills): execute DB skills by matching the litellm_skill_ tool name prefix (#30116) Skill IDs are generated as litellm_skill_<uuid> and the model-facing tool name is the sanitized skill ID, but the post-call execution gates in SkillsInjectionHook only ran tools whose name starts with "skill_", so DB skills were silently returned to the client as raw tool calls. Fixes #28122. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): synthesize content_block_start when Responses stream omits output_item.added (#30115) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit |
||
|
Mateo Wang
|
13924fa1d6
|
feat: standardize rate limit errors with category, rate_limit_type, model, and llm_provider fields (#27687)
* feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError
LiteLLM previously surfaced rate-limit conditions through several unrelated
error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException).
This commit adds the data model needed to consolidate them under a single
class:
* RateLimitErrorCategory enum exposing four categorical values
(vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit,
litellm_batch_rate_limit) so callers can switch on the rate-limit source.
* New optional fields on RateLimitError:
- category (defaults to vendor_rate_limit, preserving today's behavior for
every existing call site in exception_mapping_utils);
- headers (preserves retry-after / rate_limit_type / reset_at across the
proxy boundary instead of dropping them on the floor);
- detail (mirrors FastAPI HTTPException.detail so the same instance can be
serialized through both paths).
litellm.RateLimitErrorCategory is re-exported at the package root to match
the existing exception-export pattern.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException
Adds a single proxy-side error class that subclasses BOTH
litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative
multiple inheritance.
Why both bases:
* Subclassing RateLimitError lets user code catch every rate-limit source
with one 'except RateLimitError' and switch on the new .category field.
* Subclassing HTTPException keeps every existing FastAPI plumbing path (the
isinstance(e, HTTPException) branches in proxy_server.py route handlers,
FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException))
working without modification, and preserves retry-after / rate_limit_type /
reset_at headers on the wire.
The class declaration order is (HTTPException, RateLimitError) so the MRO
puts HTTPException's no-super-call __init__ ahead of openai's cooperative
__init__ chain — preventing openai.APIError.super().__init__(message) from
landing in HTTPException.__init__(status_code=message).
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters
Replaces three bare HTTPException(status_code=429, ...) call sites with
ProxyRateLimitError, which is both a RateLimitError (catchable by category)
and an HTTPException (preserves existing FastAPI serialization). Drops the
now-unused HTTPException import in the iteration / per-session limiters.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters
Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
parallel-request limiters (key/team/user/model/customer rate limits) with
ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type
annotation accordingly.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters
Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
dynamic rate limiters (project-level TPM/RPM allocation, model-saturation
checks, priority-based limits, fail-closed guards) with ProxyRateLimitError.
The v3 limiter still imports HTTPException for an unrelated bare 'except
HTTPException:' branch.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter
Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error
with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT
so users can distinguish batch-level throttling (which counts requests/tokens
across an uploaded batch input file before submission) from the generic
key/team/user RPM/TPM limiter.
The HTTPException import is retained because the same module raises
HTTPException for unrelated 403/IO error paths.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): pin down unified rate-limit error contract
Adds a dedicated test module covering the new RateLimitErrorCategory enum,
RateLimitError.category default + override behavior, ProxyRateLimitError's
dual nature (RateLimitError + HTTPException), and a parametrized regression
guard that asserts every proxy hook module imports the unified class.
The regression guard catches the failure mode the refactor is designed to
prevent: someone re-introducing a bare HTTPException(status_code=429, ...)
in one of the hook modules instead of going through ProxyRateLimitError.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(logging): expose rate-limit category via StandardLoggingPayload
Adds an optional 'error_rate_limit_category' field to
StandardLoggingPayloadErrorInformation, populated from the unified
RateLimitError.category attribute (introduced in the previous commits on
this branch).
Why: the .category attribute is reachable off the raw exception today via
getattr(e, 'category', None), but the structured contract that downstream
custom callbacks / loggers / spend log writers consume is the
StandardLoggingPayload. Without this field, a user building custom
rate-limit metrics on top of callback data has to special-case the raw
exception object — which defeats the purpose of the StandardLoggingPayload
abstraction.
The field is None for non-rate-limit exceptions (so consumers can read it
unconditionally without isinstance checks) and is one of the
RateLimitErrorCategory string values otherwise.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): assert StandardLoggingPayload carries the category
Five tests covering: vendor default, explicit litellm_rate_limit and
litellm_batch_rate_limit values, None for non-rate-limit exceptions, and
None when no exception is provided. Pins down the contract that custom
callbacks can read 'error_information.error_rate_limit_category' off the
StandardLoggingPayload to drive custom rate-limit metrics without ever
reaching for the raw exception.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(types): silence mypy [misc] on intentional dual-base attr overlap
mypy emits two [misc] errors on the ProxyRateLimitError class line because
its two bases declare overlapping attributes with related-but-not-identical
annotations:
* status_code: int on starlette HTTPException vs. Literal[429] on openai's
RateLimitError (every openai status-error subclass narrows it the same
way and silences pyright with the same convention).
* headers: Mapping[str, str] | None on HTTPException vs. our Optional[
Dict[str, str]] (the proxy hooks always carry a stringified dict).
Both narrowings are intentional and enforced at construction time. Add a
type: ignore[misc] with an inline explanation rather than relax the
annotations on the parent or change the wire-format guarantees.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): add direct hook-invocation tests to lift patch coverage
Adds six end-to-end tests that drive each refactored hook past its
limit and assert the unified ProxyRateLimitError is raised with the
correct category and dual-base shape. Complements the
import-shape-only parametrized guard above by actually executing the
new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage
sees them as hit.
Hooks covered (one test each):
* parallel_request_limiter v1 — direct call to raise_rate_limit_error()
* parallel_request_limiter v3 — direct call to _handle_rate_limit_error
with a fabricated OVER_LIMIT response
* max_iterations_limiter — full async_pre_call_hook with mocked agent
registry, second call exceeds budget=1
* max_budget_limiter — async_pre_call_hook with mocked get_current_spend
* dynamic_rate_limiter v1 — async_pre_call_hook with mocked
check_available_usage forcing available_tpm == 0
* batch_rate_limiter — direct _raise_rate_limit_error call, asserts
category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the
generic LITELLM_RATE_LIMIT)
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: guard rate_limit_category extraction with isinstance check
* test(rate-limit): cover remaining hook raise sites for codecov
Adds five more direct hook-invocation tests so every PR-touched line
in the proxy hooks is exercised by tests in tests/test_litellm/, which
codecov measures:
* parallel_request_limiter v1 — check_key_in_limits inline raise
(the second raise site, separate from the raise_rate_limit_error
helper covered earlier)
* dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already
covered)
* dynamic_rate_limiter v3 — parametrized over all three raise sites:
model_saturation_check, priority_model, and the fail-closed
fallback for an unrecognized descriptor_key
* max_budget_per_session_limiter — full async_pre_call_hook with a
mocked agent registry and over-budget cached spend
All 42 tests in test_rate_limit_error_unification.py now pass and
together exercise every changed import + raise line across the eight
refactored proxy hooks.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: use computed error_message in ProxyRateLimitError detail
* fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn
The v1 ' raise_rate_limit_error' helper built an unused 'error_message'
variable and then assembled the actual ' detail' via an f-string that
interpolated 'additional_details' verbatim — producing
'Max parallel request limit reached None' when invoked without
arguments (flagged by code review).
Fix the helper to:
- use the constructed 'error_message' as the detail
- annotate the helper as NoReturn since it always raises
- drop the redundant 'raise'/'return' at the two call sites
Add two regression tests covering both the with- and without-
additional_details paths.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail
The v1 parallel_request_limiter's raise_rate_limit_error helper has a
long-standing bug: it computes a None-guarded 'error_message' string but
then ignores it and emits an f-string that interpolates the raw
'additional_details' arg. Callers that pass no argument get
'Max parallel request limit reached None' as the user-facing detail.
This commit:
* wires error_message into the detail kwarg so the None-guard actually
applies and operators see a clean message;
* changes the return-type annotation from ProxyRateLimitError to NoReturn
(the function always raises) so type-checkers know callers after this
invocation are unreachable.
Greptile P1 + P2 review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(types): demote TypedDict floating string to a # comment
A string literal placed after a field declaration in a TypedDict body is
not a per-field docstring — it's an orphaned string expression Python
discards. Tools like mypy / pyright that inspect TypedDict fields won't
surface that text either.
Move the documentation for error_rate_limit_category to a real comment
so the intent is visible to readers and type-checker tooling without
the misleading docstring framing.
Greptile P2 review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* security(exceptions): do not auto-copy vendor response headers to e.headers
A vendor 429 response can set arbitrary headers (Set-Cookie, CORS
overrides, …). Previously, when RateLimitError was constructed with only
a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to
a copy of response.headers. If a downstream proxy serializer ever
forwarded e.headers to the client, a malicious upstream could inject
browser-interpreted headers for the proxy origin.
Drop the fallback. Only headers passed explicitly via the headers= kwarg
make it onto self.headers (proxy hooks pass retry-after etc. — they
control what's surfaced). Vendor response headers stay reachable on
e.response.headers for callers that explicitly want them.
Today's proxy_server.py route handlers don't actually forward e.headers
on the wire (they construct ProxyException without passing headers), so
no current behavior changes — this is a defensive narrowing so the
fallback can never be turned into a vector when someone wires
e.headers through later.
Veria-AI security review feedback on PR #27687.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): regression guards for review-pass fixes
Pins down the three review-pass fixes:
* test_parallel_request_limiter_v1_helper_no_additional_details — calls
raise_rate_limit_error() with no args and asserts the detail does NOT
contain the literal string 'None'. Pre-fix, callers got 'Max parallel
request limit reached None'.
* test_rate_limit_error_does_not_auto_copy_response_headers — passes a
vendor httpx.Response with a Set-Cookie header to RateLimitError
WITHOUT an explicit headers= kwarg, asserts self.headers stays None
(no leak), then re-checks that an explicit headers= kwarg DOES
populate self.headers. Vendor headers remain reachable on
e.response.headers for callers that explicitly want them.
* The existing v1-helper test now also asserts the additional_details
string makes it through to the detail.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations)
trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by
the dimension that was exceeded, not just by who rate-limited (vendor vs.
litellm). Adds:
- RateLimitType str-enum exposed at `litellm.RateLimitType` with values
requests / tokens / concurrent_requests / budget / max_iterations.
- `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError;
None default so existing callers (vendor-429 path in exception_mapping_utils)
remain a no-op.
- StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom
callbacks can split rate-limit failures by cause without parsing free-text
error messages. Mirror to error_rate_limit_category extraction in
get_error_information(); single isinstance(RateLimitError) check covers both.
- map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels
("requests", "tokens", "max_parallel_requests") onto the public enum so
the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive
None on unknown values rather than silently picking a wrong dimension.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): wire rate_limit_type onto every limiter raise site
Each refactored proxy hook now populates rate_limit_type with the dimension
that actually tripped the limit, so downstream consumers (custom callbacks,
prometheus exporters via the StandardLoggingPayload) can split key/team/user
rate-limit failures by cause:
- parallel_request_limiter (v1): detect dimension from current vs. limit in
the post-cache branch (concurrent_requests > tokens > requests, matches the
boolean condition order). Base case (current is None, one limit set to 0)
picks the most-specific zero. raise_rate_limit_error() helper accepts an
explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches
every existing internal call site, including the global-limit branch).
- parallel_request_limiter (v3): forward status["rate_limit_type"] through
map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS
for the public field while the raw v3 jargon stays on the HTTP header for
wire-format backward compat.
- dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass
data["model"] through so callbacks see the model that hit the limit
(addresses the secondary "provider missing" complaint in the original
Slack thread, partially — the model is what dashboards typically split on).
- dynamic_rate_limiter (v3): forward status["rate_limit_type"] via
map_v3_rate_limit_type() at every raise site (model_saturation_check,
priority_model, fail-closed unknown-descriptor guard). Also pass model.
- batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map
directly without going through the helper's None branch.
- max_budget_limiter, max_budget_per_session_limiter: BUDGET.
- max_iterations_limiter: MAX_ITERATIONS.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation
27 new tests across five new test classes:
- TestRateLimitType: enum exposed at litellm.RateLimitType, all five values
defined, RateLimitError default is None (vendor 429 path makes no claim
about which dimension), accepts both string and enum forms with
str-coercion guarantee for downstream JSON serializers.
- TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts
string or enum, doesn't break existing callers that pass nothing.
- TestMapV3RateLimitType: pins each v3-internal → public-enum mapping
(tokens, requests, max_parallel_requests → concurrent_requests, unknown
→ None) so a future v3 refactor can't silently swap dimensions.
- TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type
field reaches the structured payload for both ProxyRateLimitError and
plain RateLimitError, is None when unspecified, and is None for
non-rate-limit exceptions (symmetric with error_rate_limit_category).
- TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the
v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error
(both "tokens" and "max_parallel_requests" paths), and the batch
limiter (both tokens and requests paths) — coverage tools see the new
rate_limit_type= kwargs as exercised, not just the import shape.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(rate-limit): cover _coerce_message branches and v1 dimension detection
Drives the patch coverage on the new orthogonal RateLimitType wiring up
to (or close to) 100% on the touched files.
ProxyRateLimitError._coerce_message — was 22% covered, now 100%:
* nested {error: {message}} dict
* nested {message: {message}} dict (alt key)
* dict without 'error'/'message' keys → JSON dump fallback
* non-JSON-serializable dict value → str() fallback
* non-string non-mapping detail (int) → str() coercion
v1 parallel_request_limiter dimension detection — was 0% covered, now
exercised across 6 parametrized cases:
* check_key_in_limits else-branch: current at concurrent / TPM / RPM cap
→ asserts rate_limit_type is concurrent_requests / tokens / requests.
* check_key_in_limits base case (current is None): max_parallel_requests
/ tpm_limit / rpm_limit set to 0 → asserts the most-specific zero
attribution wins per the helper's order.
LIT-2968
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver
Introduces a small helper layer used by every proxy-side rate-limit
hook so that the 429 they raise carries a populated llm_provider /
model — instead of an empty exception.llm_provider that downstream
loggers (Prometheus failure metric, observability callbacks) read as
'no provider attribution'.
ProxyHTTPRateLimitError inherits from both fastapi.HTTPException
(so the proxy server still renders it as a 429) and
litellm.exceptions.RateLimitError (so isinstance checks and
PrometheusLogger._get_exception_class_name pick up llm_provider).
We deliberately don't call RateLimitError.__init__ — it constructs
an httpx.Response we don't need and would just add failure surface;
attribute parity is what downstream consumers care about.
resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider
defensively. Internal limiter hooks fire from async_pre_call_hook —
well before get_llm_provider runs anywhere else in the request
lifecycle — so we have to call it ourselves at raise time. If the
model is missing or unparseable (alias, router-only model) we fall
back to llm_provider='litellm_proxy' rather than letting a second
exception leak out and break the request path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on parallel-request 429s
Both v1 and v3 parallel-request limiters fired bare HTTPException(429)
from inside async_pre_call_hook. The downstream Prometheus failure
metric reads exception.llm_provider via _get_exception_class_name —
the empty value showed up as exception_class='HTTPException' and
left model_id='None' on the time series.
Threads requested_model through every raise site in:
* parallel_request_limiter.py:
- check_key_in_limits (the per-key/per-model/per-user/per-team/
per-customer over-limit path)
- raise_rate_limit_error (zero-limit + global_max_parallel_requests
paths) — now takes an optional requested_model kwarg
* parallel_request_limiter_v3.py:
- _handle_rate_limit_error (the OVER_LIMIT translator), called
from both the should_rate_limit pre-check and the TPM
reservation path
Resolved via resolve_llm_provider_for_rate_limit so unknown / missing
models silently fall back to llm_provider='litellm_proxy' instead of
breaking the request path with a second exception.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s
Same plumbing change as the parallel limiters, applied to both
dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3:
* v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve
data['model'] -> (model, llm_provider) once and pass it into both
raises.
* v3: All three raise sites in _check_rate_limits — the
model_saturation_check enforced raise, the priority_model
enforced raise, and the fail-closed unknown-descriptor branch —
now attribute the 429 to the actual provider.
Falls back to llm_provider='litellm_proxy' when the model can't be
resolved.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s
batch_rate_limiter._raise_rate_limit_error now takes a
requested_model kwarg threaded from data['model'] in
_check_and_increment_batch_counters. The batch-creation 429 is what
gets raised when the input file's tokens/requests count would push
the per-key TPM/RPM window over its limit.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(proxy/hooks): populate llm_provider on budget/iterations 429s
Final batch of internal raise sites — the user/session-budget and
max-iterations hooks. Same pattern: resolve data['model'] once at
raise time, attach to ProxyHTTPRateLimitError so Prometheus and
observability callbacks can attribute the 429.
Hooks updated:
* max_budget_limiter (per-user max_budget exceeded)
* max_iterations_limiter (per-session agent iteration cap)
* max_budget_per_session_limiter (per-session dollar cap)
All three fall back to llm_provider='litellm_proxy' when data['model']
is missing or unparseable. Drops the now-unused HTTPException import
from each module.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(proxy/hooks): pin provider field on internal rate-limit 429s
Regression coverage for the 'provider field missing' bug across every
proxy-side rate-limit hook + the helper layer:
* ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError,
dict-detail stringification, None-provider normalization).
* resolve_llm_provider_for_rate_limit happy paths
(gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback
branches (None, '', unknown name) plus a 'get_llm_provider raises'
case that asserts we swallow the secondary exception.
* For each limiter (parallel v1/v3, dynamic v1/v3, batch,
max_budget, max_iterations, max_budget_per_session): assert the
raised exception is a RateLimitError carrying the resolved
model + llm_provider, and a sibling test that asserts the
fallback path returns 'litellm_proxy' without leaking a second
exception.
* Two PrometheusLogger._get_exception_class_name pins so the
Prometheus failure metric label flips from 'HTTPException' to
'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on
fallback) — that's what dashboards consume.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* perf(proxy/hooks): defer provider resolution to over-limit branches
* fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail
* Consolidate rate_limiter_utils imports in dynamic_rate_limiter
* fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError
ProxyHTTPRateLimitError inherits from RateLimitError but did not call
RateLimitError.__init__, so num_retries/max_retries were never set.
When Starlette's HTTPException lacks __str__, MRO falls through to
RateLimitError.__str__, which unconditionally reads these attributes
and raises AttributeError during logging/traceback formatting.
Initialize them to None defensively.
* fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError
HTTPException declares 'status_code: int' while openai.RateLimitError
(via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy
flags the multi-base override as [misc] in CI lint. The runtime semantics
are fine (we set self.status_code in __init__), so silence the
class-level annotation conflict with a targeted ignore.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix: annotate batch limiter _raise_rate_limit_error as NoReturn
* feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to #27687) (#27706)
* feat(prometheus): add rate_limit_category and rate_limit_type labels
Adds two new labels to litellm_proxy_failed_requests_metric so dashboards
can split 429s by rate-limit source (vendor vs. litellm-internal) and by
the dimension that was exceeded (requests/tokens/concurrent_requests/
budget/max_iterations) without parsing free-text error messages.
Closes the Prometheus side of LIT-2718. The unified RateLimitError.category
and .rate_limit_type fields landed in PR #27687 but were only surfaced on
StandardLoggingPayload (custom-callback channel); this exposes them on
the metric label set as well.
Both labels are populated only when the underlying exception is a
litellm.RateLimitError; non-rate-limit failures keep them empty.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* feat(prometheus): populate rate-limit labels + preserve exception_class back-compat
Two coupled changes in the Prometheus integration:
1. async_post_call_failure_hook now extracts the new RateLimitError
.category / .rate_limit_type fields (added in PR #27687) via a
_extract_rate_limit_labels helper and forwards them through
UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric.
Empty for non-rate-limit failures.
2. _get_exception_class_name special-cases ProxyRateLimitError and
keeps emitting 'HTTPException' for the exception_class label.
Without this shim, ProxyRateLimitError (which multi-inherits from
HTTPException + RateLimitError) would silently flip the label
from 'HTTPException' (the historical value for proxy-side 429s)
to 'ProxyRateLimitError', breaking existing dashboards / alerts
that key off exception_class='HTTPException'. Distinguishing
vendor vs. litellm 429s is now the job of the new
rate_limit_category label.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test(prometheus): cover rate-limit labels and exception_class back-compat
Adds 19 tests across:
- enum / label-list registration
- _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError,
non-rate-limit and None inputs (incl. parametrized over every
RateLimitErrorCategory x RateLimitType combo)
- _get_exception_class_name back-compat: ProxyRateLimitError keeps the
legacy 'HTTPException' string while vendor RateLimitError keeps the
historical 'Provider.ClassName' format
- end-to-end through async_post_call_failure_hook with both
ProxyRateLimitError and vendor RateLimitError, asserting both new
labels populate and exception_class stays back-compat
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import
Address greptile feedback:
- async_post_call_failure_hook docstring: drop the stale labelnames listing
and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric
as the source of truth so the doc cannot drift from the actual labelset.
- _get_exception_class_name: guard the lazy ProxyRateLimitError import with
ImportError so router-side fallback callsites don't blow up in non-proxy
installs that don't have fastapi (a transitive dep of
proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when
fastapi is available.
Also fix the existing enterprise callback test that asserted the old
labelset on litellm_proxy_failed_requests_metric — it now expects the new
rate_limit_category / rate_limit_type labels populated for vendor 429s.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bugbot): simplify rate-limit label coercion + guard None detail
- prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already
normalizes category/rate_limit_type to plain str, so the getattr(.value)
+ isinstance dance was dead code. Reduce to str(value) if not None.
- proxy_rate_limit_error.py _coerce_message: short-circuit None to ''
instead of falling through to str(None) = 'None', which produced the
literal message 'litellm.RateLimitError: None'.
* fix(rate-limit): surface unified category/type fields on BudgetExceededError
The most common budget cap (virtual-key max_budget enforcement in
auth_checks.py) raises litellm.BudgetExceededError, a bare Exception
subclass that bypassed the unified rate-limit error class introduced
by PR #27687. Custom callbacks reading
StandardLoggingPayload.error_information saw category=None and
rate_limit_type=None for these 429s, missing the most common budget
case (team / org / end-user budgets all hit the same code path).
Surface the fields off BudgetExceededError as plain attributes:
- category = RateLimitErrorCategory.LITELLM_RATE_LIMIT
- rate_limit_type = RateLimitType.BUDGET
- llm_provider = "" (or caller-supplied)
Switch get_error_information and _extract_rate_limit_labels from
isinstance(RateLimitError) gating to duck-typed attribute reads,
guarded by membership in the rate-limit enums so unrelated third-party
exceptions exposing a .category attribute can't leak garbage values
into the payload.
This is strictly additive: BudgetExceededError keeps its bare-Exception
base class, so `except BudgetExceededError:` handlers keep firing and
`except RateLimitError:` does not start catching budget errors.
* fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider
Two follow-ups uncovered during the second QA pass on PR #27687:
1. Guard third-party `.category` / `.rate_limit_type` attribute leakage.
The duck-typed read in `get_error_information` and
`_extract_rate_limit_labels` would forward any string attribute named
`category` / `rate_limit_type` on an unrelated third-party exception
into the StandardLoggingPayload and Prometheus labels — silently
mislabeling custom-callback payloads and blowing out Prometheus label
cardinality. Add `validate_rate_limit_category` /
`validate_rate_limit_type` helpers that gate on the documented enum
value sets; non-matching values are dropped to None.
2. Enrich BudgetExceededError.llm_provider from request_data.
Budget checks live in tenant-scoped helpers (key / team / org / tag /
end-user / project) that don't see the request model, so the
BudgetExceededError they raise carried llm_provider="" — leaving
custom-metrics consumers without provider attribution for the most
common 429 case. Resolve it once at the central
UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook
fires, so the StandardLoggingPayload the callback sees has the same
provider attribution as RPM/TPM 429s.
Regression tests pin both: 4 leakage tests + 4 enrichment tests. The
leakage tests would fail under the pre-validation version of either read
site; the enrichment tests would fail if the handler skipped the
resolver call.
* fix(rate-limit): resolve router model_name aliases to real provider (#27914)
* fix(rate-limit): resolve router model_name aliases to real provider
For nearly every real LiteLLM proxy deployment the request model is a
router model_name alias (e.g. 'tpm-locked' -> litellm_params.model:
openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about
router aliases — it raises 'LLMProviderNotProvidedError'. The resolver
then fell through to the defensive 'litellm_proxy' fallback, so the
'llm_provider' field this PR adds was effectively always
'litellm_proxy' in the field, defeating its purpose for the most common
proxy configuration.
Add a router-alias fallback step: when 'get_llm_provider' raises, scan
the active 'llm_router.model_list' for a deployment whose 'model_name'
matches the request model and resolve from its 'litellm_params.model'
instead. If multiple deployments share the same alias (load-balancing
case) the first one wins — every deployment under one alias should
agree on provider in any sensible config, and 'first' is deterministic
so the Prometheus label stays stable.
Defensive throughout: an uninitialized router, a malformed deployment,
a 'litellm_params.model' that itself fails 'get_llm_provider' — every
branch falls through to the existing 'litellm_proxy' fallback rather
than letting a secondary exception escape and mask the rate-limit
error we're trying to surface.
Tests:
- test_router_alias_resolves_to_underlying_provider: alias
'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai',
model='gpt-4o-mini'.
- test_router_alias_with_multiple_deployments_uses_first.
- test_router_alias_unknown_falls_back.
- test_router_alias_with_malformed_deployment_falls_back.
- Existing fallback test updated to also stub
'litellm.proxy.proxy_server.llm_router' so it exercises the
full 'no resolution anywhere' path.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(rate-limit): harden router alias resolver + test isolation
- Wrap _resolve_provider_from_router_alias loop in top-level try/except so
a non-iterable model_list / unexpected deployment shape can't escape and
mask the 429 with a 500.
- Type-check litellm_params before .get() to handle non-dict truthy values.
- Patch llm_router=None in the parametrized fallback test so a router left
by another test in the session can't redirect the unknown-model path.
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* fix(bugbot): preserve "BudgetExceededError" Prometheus label
Adding llm_provider to BudgetExceededError (so callbacks get provider
attribution from StandardLoggingPayload) made the provider-prefix step in
_get_exception_class_name silently flip the label from "BudgetExceededError"
to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the
historical value.
Short-circuit BudgetExceededError in _get_exception_class_name the same way
ProxyRateLimitError already is. Provider/category attribution still lands on
the new rate_limit_category / rate_limit_type labels.
* test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks
The v3 rate limiter only emits 'requests', 'tokens', or
'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to
return None, leaving the expected RateLimitType.REQUESTS untested.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels
- dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit
above the TPM/RPM if/elif so the lookup runs once per request, matching
the pattern in dynamic_rate_limiter_v3.py.
- prometheus.py: gate the new rate_limit_category / rate_limit_type
labels on litellm_proxy_failed_requests_metric behind
litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the
existing prometheus_emit_stream_label opt-in. Preserves the metric's
pre-unification label set so existing dashboards / recording rules
keep matching after upgrade; operators can enable the new labels once
downstream consumers include them.
- Tests updated: default-off back-compat case, opt-in path enables the
flag before asserting label presence.
* fix: stabilize prometheus label sets and drop redundant model normalization
- Cache PrometheusLogger.get_labels_for_metric per metric_name so that
the label set used to construct counters at __init__ time stays in
sync with the label set used at increment time, even if module-level
toggles like prometheus_emit_rate_limit_labels or
prometheus_emit_stream_label are flipped at runtime. Without this,
toggling these flags after the logger was created would cause
ValueError from prometheus_client because the runtime labels would
not match the counter's declared labelnames.
- Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__
where model is already normalized one step earlier.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* perf(dynamic_rate_limiter): only resolve provider when rate limit hit
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test(prometheus): clear cached metric labels after toggling rate-limit flag
The PrometheusLogger caches each metric's label set at construction
time so that labels used at counter.labels(...) time stay consistent
with the labels the metric was registered with. The enterprise
async_post_call_failure_hook test toggles
litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture
has already built the logger, so without invalidating the cache the
rate_limit_category / rate_limit_type labels never reach the mocked
counter and the assert_called_once_with check fails.
Co-authored-by: Yassin Kortam <yassin@berri.ai>
* test: fix CI failures from prom label cache + flaky time-window assertion
PrometheusLogger.get_labels_for_metric now caches the per-metric label
set at first read so the labels passed to counter.labels(...) stay in
lock step with the labels the counter was registered with. This broke
two existing test patterns:
- test_prometheus_labels.py: tests bind the real method onto a
MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels
whose .get(...) returns a truthy Mock — treated as a populated cache
and returned as the label set, producing empty filtered labels and
KeyError on labels["requested_model"] / ["route"]. Seed real {}
containers for _cached_metric_labels and label_filters before binding.
- test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels:
the fixture builds the logger before the test monkeypatches
litellm.custom_prometheus_metadata_labels, so the cached label set
never picks up the new metadata labels. Clear the cache after the
monkeypatch (same pattern already used for the rate-limit toggle in
test_async_post_call_failure_hook).
UI: view_logs/index.test.tsx "Last Minute" window assertion is off by
one at the minute boundary. start_date is floored to the minute, so the
dropped sub-minute fraction can push the truncated-seconds diff up to
(minMinutes+1)*60 exactly when the click lands near a minute rollover.
Switch the upper bound to toBeLessThanOrEqual.
* feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans
PR #28909 introduced the typed v2 OTel engine that builds spans from
StandardLoggingPayload, with SpanError carrying error_type + message and
the genai mapper stamping error.type onto every failed LLM-call span.
This PR's earlier commits added error_rate_limit_category and
error_rate_limit_type to the same StandardLoggingPayload.error_information
the v2 engine reads — but neither field reached a span attribute, so v2
OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm,
RPM vs TPM vs concurrent vs budget vs max_iterations) even after the
custom-callback and prometheus surfaces gained that decomposition.
Three coupled changes:
1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY /
LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace
(no GenAI semconv equivalent exists for who-rate-limited /
which-dimension).
2. payloads.py: extend SpanError with rate_limit_category +
rate_limit_type, populated by _parse_error() from the same
error_information.error_rate_limit_* fields the custom-callback
channel and prometheus rate_limit_category / rate_limit_type labels
read. Single source of truth across all three observability surfaces.
3. mappers/genai.py: stamp the two attributes on the LLM-call span when
present. drop_none guarantees they stay absent (not 'None') for
non-rate-limit failures so trace consumers can read them
unconditionally.
Three regression tests in test_otel_v2_emitter.py pin: a vendor /
litellm-internal RateLimitError lands category=litellm_rate_limit +
rate_limit_type=requests on the span; a BudgetExceededError lands
rate_limit_type=budget; a non-rate-limit failure (BadRequestError)
keeps the rate_limit_* attributes absent. Mutation-tested against
reverting either the SpanError extension or the _parse_error read site
— both new tests fail under either mutation.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
* test: align prometheus user-budget + logs quick-select tests with merged code
The merge into this branch left two test patterns out of step with the code
they exercise.
test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in
flipped litellm.prometheus_user_budget_label_include_email_alias after the
fixture had already built the PrometheusLogger. get_labels_for_metric now
snapshots each metric's label set at construction time, so the runtime flip
no longer reached the cached labels. Enable the flag before constructing the
logger, matching how the proxy applies config at startup.
view_logs/index.test.tsx referenced uiSpendLogsCall and moment without
importing them, and the merged index.tsx now fetches through
useLogFilterLogic (the hook the file stubs out) rather than calling
uiSpendLogsCall directly. Add the imports and restore the real hook for the
Quick Select window assertions so the call is actually observed.
* refactor(otel/v2): drop rate-limit decomposition from the LLM-call span
Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are
rejected at the gate before any upstream call, so async_post_call_failure_hook
tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the
v2 OTel logger never opens an LLM-call span for them; the
litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes
were dead for exactly the cases they were meant to surface. The only failure
that does open an LLM-call span carrying a RateLimitError is a vendor 429, where
rate_limit_type is always None and the category just restates
error.type=RateLimitError.
The decomposition still reaches downstream consumers through
StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus
rate_limit_category / rate_limit_type labels, both unchanged.
Removes the SpanError fields, the _parse_error reads, the genai mapper
attributes, the semconv keys, and the three span tests that asserted a scenario
that never reaches the mapper in production.
* fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests
* refactor(prometheus): drop transitive fastapi import from _get_exception_class_name
Read the legacy exception_class label from a prometheus_exception_class_name
marker on ProxyRateLimitError instead of importing the proxy module, keeping
the integrations layer free of a transitive fastapi dependency.
* chore(ui): sync schema.d.ts with unified rate-limit error spec
The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429
response description, so the generated dashboard types were out of sync.
Regenerated via npm run gen:api (Check UI API Types Sync).
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Co-authored-by: Yassin Kortam <yassin@berri.ai>
|
||
|
ryan-crabbe-berri
|
7edf3a9cb5
|
style(ui): run prettier --write across the dashboard (#29622)
Formatting-only pass; no logic changes. Brings the UI into compliance with .prettierrc so the new format-check CI job passes |
||
|
ryan-crabbe-berri
|
609e1e9763
|
fix(ui): render caller-supplied filter options in caller order (LIT-3151) (#29462)
FilterComponent iterated a hardcoded orderedFilters whitelist instead of the options prop, so any consumer whose filter names were not on that list rendered nothing. The Tool Policies page passes "Input Policy", "Output Policy", "Team Name" and "Key Name", none of which were whitelisted, so its Filters panel opened to an empty area. Drop the whitelist and render the options the caller passes, in the order they pass them, so each page owns its own filter set and ordering. The Logs page array is reordered to match its prior on-screen order; VirtualKeys and TeamVirtualKeys already matched the old whitelist order and are unaffected. |
||
|
ryan-crabbe-berri
|
2eeca2d096
|
fix(ui): restore log filter loading indicator (#28282)
When a new filter is applied to spend logs, React Query's keepPreviousData left stale rows on screen for 10–15s with no indication that a fetch was in progress. The previous custom isFilteringResults flag was removed in the #25847 toolbar refactor and only partially restored on the Fetch button. Use React Query's isPlaceholderData to discriminate a real filter change (queryKey changed, data not yet arrived) from a same-key live-tail refetch, and feed it into the existing isLoading prop on the toolbar pagination text and the table body. Live-tail polls still keep previous rows without flicker. Co-authored-by: Ryan <ryan@Ryans-MBP.localdomain> |
||
|
ryan-crabbe-berri
|
727a471ae9
|
[Refactor] UI - Spend Logs: consolidate filter state and extract components (#25847)
* [Refactor] UI - Spend Logs: consolidate filter state, extract components, remove dead code
- Lift filter state into index.tsx and pass to hook (removes selectedX vars + sync useEffect)
- Move main useQuery into useLogFilterLogic hook (removes isMainQueryEnabled toggle)
- Delete dead RequestViewer component (300 lines, replaced by LogDetailsDrawer)
- Extract LogsTableToolbar component (search, date range, pagination, live tail)
- Extract filter options config to filter_options.ts
- Remove dead code: handleRefresh, handleSelectLog, handleCloseDrawer, formatTimeUnit,
showFilters/showColumnDropdown state, dropdownRef/filtersRef
* Fix PR feedback: use antd Switch instead of Tremor in new file, fix typo
* Collapse dual-path filtering into single React Query
All 10 filter keys now go through the useQuery — the imperative
performSearch / debouncedSearch / backendFilteredLogs path is deleted.
Filter values are debounced via useDebouncedValue(300ms) before hitting
the query key so text inputs don't fire per-keystroke.
Removed: performSearch, debouncedSearch, backendFilteredLogs,
lastSearchTimestamp, hasBackendFilters, clientDerivedFilteredLogs,
the sort/page/time refetch useEffect, and the filteredLogs chooser memo.
* Clean up remaining smells: remove isFetchingDeferred, internalize selectedTimeInterval, fix circular import
- Remove useDeferredValue/isButtonLoading — pass logsQuery.isFetching directly
- Move selectedTimeInterval into LogsTableToolbar as internal state
- Move PaginatedResponse type from index.tsx to log_filter_logic.tsx
* Fix quick-select dropdown overlapping sidebar
* Fix stale quick-select label after Reset Filters
Move selectedTimeInterval back to parent so handleFilterReset can
reset it to the 24-hour default. The toolbar receives it as a prop.
* refactor useLogFilterLogic tests for controlled-hook + backend-query shape
The hook no longer owns filter state or does client-side filtering — it
receives filters/setFilters as props and drives filteredLogs from a
useQuery over uiSpendLogsCall. Reshape the tests around that contract:
introduce a controlled harness that owns filter state, collapse the 10
per-filter assertions into a single it.each over filterKey → API param,
and drop the client-side passthrough tests (the .min test file and the
"return all logs when no filters" / "empty when logs null" cases) that
no longer correspond to any hook behavior.
* cover new useLogFilterLogic invariants: activeTab gate, filterByCurrentUser fallback, debounce negative, partial merge
Follow-up to the test refactor. Adds coverage for invariants the
refactored hook contract introduced but that the first pass didn't
assert:
- query enablement: expand the single accessToken-null case into an
it.each over all four credential props (accessToken, token, userRole,
userID), plus a separate test for activeTab !== "request logs"
- filterByCurrentUser: when true with a blank User ID filter, the
outbound request carries user_id = userID
- debounce: also assert the negative case — no call in the first 100ms
after a filter change (first waiting out the initial mount fire)
- handleFilterChange: partial updates merge without clobbering other
filter keys (protects the spread + default-fill semantics)
- handleFilterReset: calls setCurrentPage(1) alongside restoring
filters
* fix typo dropping the live-tail banner border
Tailwind silently ignores unknown classes, so border-greem-200 was
leaving the auto-refresh banner with only its bg-green-50 fill and no
outline.
* memoize columns and derived table data in SpendLogsTable
The table's columns array, four-pass data pipeline, and sort-change
handler were all being rebuilt on every parent render. That made every
filter click re-instance all 23 TanStack-Table columns, re-run
filter/reduce/map over all rows, and recreate per-row click closures —
all before the intentional 300ms debounce timer even got a chance to
fire.
Local measurement (40 rows, dev mode):
filter click → query fires: 1957ms → 1217ms (−38%)
Wrap createColumns in useMemo keyed on sortBy/sortOrder, hoist
onSortChange into a useCallback, and move the searchedLogs /
sessionComposition / sessionRepresentativeMap / filteredData derivations
into a single useMemo keyed on filteredLogs.data + searchTerm.
These were pre-existing issues on main — not regressions from the
hook refactor — but the refactor made them user-visible because the
new query debounce put render cost on the critical path.
* apply dropdown filters instantly, debounce only text inputs
Dropdown selects now bypass the 300ms debounce so a click updates the
table immediately. Text inputs (Key Hash, Error Message, Request ID,
User ID) still debounce. handleFilterReset also clears the pending
debounced value so a half-typed text filter can't re-fire after reset.
* fix(ui/spend-logs): restore lost loading/debounce behavior + cover dropped tests
Regressions from the spend-logs-view refactor:
- debounce the 'Public model / search tool' text filter (was firing a
backend query per keystroke) via TEXT_FILTER_KEYS
- restore Fetch-button smoothing through table repaint using
useDeferredValue on the rendered data (explicit staleness)
- show AntDLoadingSpinner during the auth-resolve phase instead of a
blank screen on first load
- only live-tail-poll while the tab is visible
(refetchIntervalInBackground: false)
- extract getLiveTailRefetchInterval helper for the poll decision
Tests:
- LogDetailContent: retries display (>0 / 0 / absent), overhead-absent
- log_filter_logic: regression guard that the public-model filter
debounces; getLiveTailRefetchInterval unit tests
- logs_utils: getTimeRangeDisplay quick-select window labels
* test(ui/spend-logs): cover the cold-load auth-not-ready spinner guard
Asserts SpendLogsTable shows a loading spinner (not a blank screen)
while credentials are unresolved, and renders the table once present.
|
||
|
Yuneng Jiang
|
b379f4f98b
|
[Feature] UI - Logs: Add 'Last Minute' to time-range quick select
Adds a Last Minute option as the first entry in the logs page time-range quick-select dropdown. Tests verify the UI passes a ~1-minute window (start_date / end_date) to uiSpendLogsCall when selected. |
||
|
Sameer Kankute
|
45c22081ee
|
Fix ui unit test | ||
|
Sameer Kankute
|
4b03cb68a2
|
feat(proxy): move search tool access to object permissions
Store search tool allowlists only on object permissions, wire auth/management/UI flows to object_permission.search_tools, and remove legacy team-metadata search credential code and tests. Made-with: Cursor |
||
|
Ryan Crabbe
|
be248627b9
|
Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_fix-logging-settings-admin-only | ||
|
Yuneng Jiang
|
5c0349a635
|
[Feature] UI - Spend Logs: sortable Model and TTFT columns
Extend the /spend/logs/ui sort_by whitelist to accept "model" and "ttft_ms", and wrap the Model and TTFT (s) column headers with the existing SortableHeader component so users can sort by either. TTFT has no stored column, so it is computed inline as (completionStartTime - startTime) milliseconds. Non-streaming rows (completionStartTime null or equal to endTime) yield NULL and the ORDER BY uses NULLS LAST so they always sort to the bottom regardless of direction, matching the existing "-" display in the UI. Adds parametrized backend tests for sort_by=model and a dedicated test covering streaming + non-streaming TTFT ordering in both directions. |
||
|
ishaan-berri
|
8a9faa81b2
|
feat(guardrails): LLM-as-a-Judge guardrail (#26360)
* feat(guardrails): add LLM_AS_A_JUDGE to SupportedGuardrailIntegrations * feat(types): add EvalVerdict, StandardLoggingEvalInformation; wire eval_information into SpendLogsMetadata * feat(guardrails): add self-contained llm_as_a_judge guardrail hook * fix(a2a): filter agent-only litellm_params from acompletion kwargs; pass agent_id into body * feat(ui): add LLMJudgeFields criteria builder component * feat(ui): wire LLM-as-a-Judge into add guardrail form * feat(ui): update EvalViewer — title 'LLM Judge Results', weighted score column, summary row * fix(ui): wire EvalViewer into LogDetailContent to show LLM judge results on logs page * fix(guardrails-ui): route llm_as_a_judge to criteria builder step; rename to LiteLLM LLM as a Judge; add litellm logo * fix(guardrail-viewer): stack lifecycle + eval details vertically to avoid badge overflow in narrow drawer * fix(guardrail-create): surface config validation errors on create instead of silently orphaning guardrail in DB * fix(guardrail-registry): hardcode llm_as_a_judge in initializer registry so it loads regardless of package install path * fix(llm-as-a-judge): fix P1 code quality issues - validate weights/on_failure, guard pre_call, handle multimodal, move imports to module level, fix spurious finally logging * fix(guardrail_endpoints): use correct PK field in rollback delete and log rollback failure * fix(llm_as_a_judge): support Pydantic object in _get_litellm_param fallback chain * fix(LLMJudgeFields): replace @tremor/react Button with antd Button * fix(llm_as_a_judge): remove dead registry dicts, fix KeyError in prompt builder, set correct status on judge failure * test(llm_as_a_judge): add unit tests for guardrail hook * fix(llm_as_a_judge): remove @log_guardrail_information decorator to fix duplicate guardrail_information entries The decorator and the manual finally block both called add_standard_logging_guardrail_information_to_request_data, producing two entries per request. The decorator also misclassified HTTPException(422) blocks as guardrail_failed_to_respond (it checks for 400). The finally block correctly tracks status throughout, so removing the decorator is sufficient. * fix(test_gcs_pub_sub): ignore metadata.eval_information in comparison * fix(test_spend_management): ignore metadata.eval_information in payload comparison * fix(types/guardrails): add input_type and messages to ApplyGuardrailRequest * fix(guardrail_endpoints): pass input_type and messages through apply_guardrail endpoint * fix(guardrail_endpoints): auto-detect post_call guardrails and use input_type=response * fix(a2a_endpoints): merge agent litellm_params guardrails into data before post_call hooks * fix(llm_as_a_judge): use float sum with tolerance for weight validation * fix(guardrail_registry): split long import line for black formatting * fix(llm_as_a_judge): guard guardrail_name Optional for mypy * fix(llm_as_a_judge): set guardrail_status=guardrail_intervened when score fails, regardless of on_failure mode * fix(a2a_endpoints): use try/finally so deferred spend log fires even when guardrail blocks with 422 * fix(litellm_logging): declare _defer_async_logging and _enqueue_deferred_logging on Logging class for mypy * fix(logging_worker): restore queue.join() in flush() to wait for in-flight callbacks |
||
|
Ryan Crabbe
|
2c3c8aa4ea
|
Move "Store Prompts in Spend Logs" toggle to Admin Settings
Previously, the "Store Prompts in Spend Logs" and "Maximum Spend Logs Retention Period" settings were surfaced via a gear-icon modal on the Logs page. The gear was visible to every authenticated user even though the backend endpoints (/config/update, /config/list) require PROXY_ADMIN — so non-admins could open the modal but the request would 403 on load and save, giving a confusing UX. Move the controls into a new "Logging Settings" tab under Admin Settings, which is already gated to admins at the sidebar. Remove the gear button and the onOpenSettings prop chain (ConfigInfoMessage → LogDetailContent → LogDetailsDrawer). ConfigInfoMessage now points users to "Admin Settings → Logging Settings" inline. |
||
|
Rick
|
c26e304abc
|
fix(ui): stale filters applied after sort/page/time change on Request Logs (#25789)
The useEffect that re-fetches logs on sort/page/time changes:
useEffect(() => {
if (hasBackendFilters && accessToken) {
performSearch(filters, currentPage);
}
}, [sortBy, sortOrder, currentPage, startTime, endTime, isCustomDate]);
intentionally omits `filters` and `hasBackendFilters` from its dep array
to avoid double-fetches when a filter is applied. The side-effect is a
stale-closure bug: the effect captures `filters` and `hasBackendFilters`
from the render where its deps last changed, not from the render where
the user selected, e.g., a Key Alias.
Reproduce: set Key Alias → results appear correctly → change page or
sort → the effect fires with the OLD `filters` snapshot (no key_alias)
→ API request is sent without the filter → table shows unfiltered data.
Fix: store the latest `filters` and `hasBackendFilters` in refs that are
kept in sync on every render. The sort/page/time effect reads from the
refs instead of the closure so it always uses the current filter state
without altering the dep array.
Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4 (1M context) <noreply@anthropic.com>
|
||
|
Rick
|
4b2fd870ca
|
fix(ui): Fetch button ignores active filters on Request Logs page (#25788)
When backend filters (e.g. Key Alias) are active on the Request Logs page, the manual Fetch button called logs.refetch() which re-runs the main TanStack Query. That query does not carry backend-only filter params such as key_alias, so the button had two problems: 1. It fired a redundant API request without the active filters. 2. It did not refresh the filtered result set — backendFilteredLogs stayed frozen at the last debounce-triggered fetch. Fix: expose refetchWithFilters() from useLogFilterLogic and route the Fetch button through it when hasBackendFilters is true. This cancels any in-flight debounce and calls performSearch with the current filter state, keeping all active filters intact. Co-authored-by: Bytechoreographer <Bytechoreographer@users.noreply.github.com> Co-authored-by: Claude Sonnet 4 (1M context) <noreply@anthropic.com> |
||
|
Ryan Crabbe
|
27484c4a41
|
fix: isolate logs team filter dropdown from root teams state bleed
The Logs view's Team ID filter dropdown was reading `allTeams` from the root `teams` state in page.tsx, which the Teams page search overwrites with its filtered subset. Applying a team search on the Teams page made filtered-out teams disappear from the Logs filter dropdown. Swap the Team ID filter to use the existing `TeamDropdown` component via a small `FilterTeamDropdown` wrapper that adapts it to the filter slot's `FilterOptionCustomComponentProps` contract. The dropdown now drives its own `useInfiniteTeams` query against `/v2/team/list` with server-side search and an isolated react-query cache, unreachable from root state. Rename the now-unused `hookAllTeams` destructure to `allTeams` so the `KeyInfoView` passthrough receives the hook's unpolluted fetch instead of the polluted prop, and drop the dead `allTeams` prop from `SpendLogsTable` and both of its call sites. |
||
|
Sameer Kankute
|
277be4c50e
|
Add input + output tokens for anthropic message type | ||
|
Ishaan Jaffer
|
e0a988e39a
|
feat(ui/log-details): pass rawInputTokens, cacheReadTokens, cacheCreationTokens to CostBreakdownViewer from SpendLogs | ||
|
Ishaan Jaffer
|
0148effd6e
|
feat(ui/cost-breakdown): show separate Input / Cache Read / Cache Write line items in cost breakdown drawer | ||
|
Ryan Crabbe
|
8a2186a654
|
[Fix] Reset page to 1 on filter reset and use pageSize param in fallbacks
Add setCurrentPage(1) to handleFilterReset in the hook so page resets regardless of which component consumes it. Replace hardcoded page_size: 50 in empty-state fallbacks with the pageSize parameter for consistency. |
||
|
Ryan Crabbe
|
fbddab6178
|
Handle search error state and cancel wasted API call on filter reset
- Set backendFilteredLogs to empty response on performSearch error so the UI shows "0 results" instead of appearing stuck in a loading state - Cancel debounced search on filter reset instead of firing a request whose result would be ignored (hasBackendFilters is false after reset) - Clear backendFilteredLogs immediately on filter change to prevent stale results from previous filter showing during debounce window - Normalize response.data with nullish coalescing to handle missing data - Use `not team_object.models` for None-safe emptiness check |
||
|
Ryan Crabbe
|
12a55a8a6f
|
Fix logs page showing unfiltered results when filter matches zero rows
The backendFilteredLogs state initialized to { data: [], ... } which was
indistinguishable from "API returned empty results". When backend filters
were active, the filteredLogs memo fell back to showing unfiltered `logs`
because it couldn't tell whether a search had completed or not.
Fix: use null as initial state so we can distinguish "not yet searched"
(null → show empty placeholder) from "search returned empty" ({ data: [] }
→ show empty results). This prevents the fallback to unfiltered logs that
caused the model filter to display mismatched data.
|
||
|
Ishaan Jaff
|
2ea9e207bd
|
Litellm ishaan march 20 (#24303)
* feat(redis): add circuit breaker to RedisCache to fast-fail when Redis is down (#24181) * feat(redis): add circuit breaker env var constants * feat(redis): add RedisCircuitBreaker and apply guard decorator to all async ops * fix(dual_cache): fall back to L1 instead of re-raising on Redis increment failures * test(caching): add circuit breaker unit tests * fix(redis): fast-fail concurrent HALF_OPEN probes — only one probe at a time * fix(dual_cache): return None fallback when in_memory_cache is absent and Redis fails * test(caching): add regression tests for HALF_OPEN concurrency and None fallback * Fix blocking sync next in __anext__ (#24177) * Fix blocking sync next * Update tests/test_litellm/litellm_core_utils/test_streaming_handler.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix PEP 479 regression in __anext__ sync iterator exhaustion asyncio.to_thread re-raises thread exceptions inside a coroutine, where PEP 479 converts StopIteration to RuntimeError before any except clause can catch it. Add _next_sync_or_exhausted() module-level helper that catches StopIteration in the thread and returns a sentinel instead, then raise StopAsyncIteration in the coroutine. Also rewrites the non-blocking test to use asyncio.gather() instead of asyncio.create_task() (which returned None on Python 3.9 / pytest-asyncio in CI), and adds an exhaustion regression test that drains the wrapper fully and asserts no RuntimeError leaks out. --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * feat: add git-subdir source type to claude-code/plugins API (#24223) Support a third plugin source type `git-subdir` alongside the existing `github` and `url` types, as documented in the official Claude Code plugin marketplaces spec. New format: {"source": "git-subdir", "url": "...", "path": "subdir/path"} - Validates url and path fields are present and non-empty - Rejects absolute paths, '..' segments, backslashes, and percent-encoded traversal sequences (including double-encoded variants via regex check) - Extracts path validation into _validate_git_subdir_path() helper - Updates Pydantic field description to document all three source types - Adds isValidUrl() check for url/git-subdir source types in the UI form - Adds "Git Subdir" option to the UI form with a required Path field - Adds unit tests covering success, update, missing/empty fields, path traversal variants, and unknown source type Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * [FEAT] add extract_header and extract_footer to Mistral OCR supported params (#24213) * docs: add git-subdir source type to claude-code plugin marketplace docs (#24289) * fix(ui): swap J/K keyboard navigation in log details drawer (#24279) (#24286) J should navigate down (next) and K should navigate up (previous), matching vim/standard conventions. * fix: use async_set_cache in user_api_key_auth hot path (#24302) * fix: use async_set_cache in auth hot path to avoid blocking event loop * test: assert no blocking set_cache call in _user_api_key_auth_builder * test: broaden blocking call check to all sync DualCache methods * test: fix regression test to actually catch blocking cache calls * fix: ruff lint unused variable + UI build MessageManager error - litellm/caching/redis_cache.py: remove unused variable 'e' in circuit breaker exception handler (F841) - add_plugin_form.tsx: use MessageManager.error() instead of undefined message.error() for git URL validation Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * docs: add REDIS_CIRCUIT_BREAKER env vars to config_settings reference Add REDIS_CIRCUIT_BREAKER_FAILURE_THRESHOLD and REDIS_CIRCUIT_BREAKER_RECOVERY_TIMEOUT to the environment variables reference table so test_env_keys.py passes. Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Vincenzo Barrea <manamana88@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Robert Kirscht <rkirscht242@gmail.com> Co-authored-by: Imgyu Kim <kimimgo@gmail.com> Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> |
||
|
yuneng-jiang
|
3ea69c9539 | Merge remote-tracking branch 'origin' into litellm_yj_march_19_2026 | ||
|
yuneng-jiang
|
9b519c4754
|
Merge pull request #24192 from BerriAI/litellm_migrate_antd_message_to_context_api
[Fix] UI: AntD Messages Not Rendering |
||
|
yuneng-jiang
|
34d954f8cb |
[Fix] UI: Migrate AntD message API to use context-based MessageManager
AntD v5 static message API doesn't render without an App wrapper or useMessage() context holder. Mirrors the existing notification pattern by adding message.useMessage() to AntdGlobalProvider and routing all calls through a new MessageManager module. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
0b07f628ff |
[Test] UI: Add vitest coverage for 10 previously untested components
Add unit tests for: - SimpleToolCallBlock, SimpleMessageBlock, CollapsibleMessage, HistoryTree (log details drawer) - OnboardingForm (onboarding flow) - TeamsHeaderTabs, TeamsTable (teams page) - transform_key_info, filter_helpers (key/team helpers) - queryKeysFactory (query key generation utility) 47 new tests covering conditional rendering, user interactions, data transformation, and error handling. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
2c0c08722a
|
Merge pull request #24035 from BerriAI/litellm_fix_guardrail_mode_type_crash
[Fix] UI - Logs: Guardrail Mode Type Crash on Non-String Values |
||
|
yuneng-jiang
|
7714d1be0b |
address greptile review feedback (greploop iteration 3)
Add typeof string guards to all array element returns in resolveMode to prevent non-string values from sneaking through via any-widening. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
yuneng-jiang
|
6902355f5b |
address greptile review feedback (greploop iteration 2)
Replace unsafe `as string[]` cast in modeMatches with runtime type check via `.some()`. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
yuneng-jiang
|
1c8b5f77c9 |
address greptile review feedback (greploop iteration 1)
Add modeMatches() helper so array guardrail_mode values (e.g. ["pre_call", "post_call"]) place the entry in all matching timeline buckets, not just the first. Updated test to verify both buckets. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
yuneng-jiang
|
b13a7c6790 |
Fix guardrail_mode.replace crash when backend returns non-string value
The backend type for guardrail_mode is Optional[Union[str, List[str], Dict]] but the UI typed it as just string, causing a crash when .replace() was called on null/object/array values. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
yuneng-jiang
|
a771fe55e4 |
[Fix] Update log filter test to match empty-result behavior
The test expected fallback to all logs when backend filters return empty, but the source was intentionally changed to show empty results instead of stale data. Updated test to match. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
yuneng-jiang
|
57bba3b863 |
[Fix] UI - Logs: Fix empty filter results showing stale data
Remove `.length > 0` check so that when a backend filter returns an empty result set the table correctly shows no data instead of falling back to the previous unfiltered logs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
yuneng-jiang
|
82fc819abf | Merge remote-tracking branch 'origin' into litellm_internal_dev_03_14_2026 | ||
|
yuneng-jiang
|
1b0c4bdbb7 |
Add unit tests for 5 previously untested UI components
Tests for HelpLink, ScoreChart, AgentCard, ToolPoliciesView, and CostBreakdownViewer (33 tests total). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Sameer Kankute
|
7e662afe01 |
feat(azure): Azure Model Router cost breakdown in UI + additional_costs from hidden_params
- Backend: Use request model from hidden_params for Azure Model Router additional_costs when response has actual model - Backend: Add additional_costs to total cost calculation - UI: Show all non-null/non-zero additional_costs in CostBreakdownViewer - UI: Render cost breakdown when only additional_costs exist - Tests: Backend test for hidden_params flow; frontend tests for additional_costs Made-with: Cursor |
||
|
yuneng-jiang
|
bead0b7d90 |
[Test] UI - Logs: Add unit tests for 5 untested view_logs components
Add vitest tests for TypeBadges, ErrorViewer, ConfigInfoMessage, TimeCell, and TruncatedValue covering rendering, user interactions, and edge cases. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ishaan Jaff
|
dd183a7fcb
|
[Feat] UI - Allow sorting MCPs by created_at, Display name date (#22825)
* Add column sorting to MCP servers table - Added sorting state management to DataTable component - Enabled getSortedRowModel for tanstack/react-table - Made column headers clickable with sort indicators (↑↓⇅) - Added enableSorting: true to sortable columns in mcp_server_columns - Columns now support ascending/descending sort by clicking headers - Updated package-lock.json and tsconfig.json from build process Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> * Make table sorting opt-in to avoid conflicts with existing consumers Address Greptile feedback (score 2/5): - Added enableSorting prop to DataTable (defaults to false) - Only enable sorting features when explicitly requested - Pass enableSorting=true from MCP servers component - This prevents unintended sorting on other DataTable consumers: * view_logs (has server-side sorting) * pass_through_settings * UsagePage - Sorting UI (indicators, click handlers) only shown when enabled Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com> |
||
|
Ishaan Jaff
|
b7f43d411a
|
feat(ui): add time to first token (TTFT) to logs (#22819)
* feat(ui): add TTFT (s) column to request logs table * feat(ui): add Time to First Token metric to log detail drawer * docs: add TTFT screenshot |