The upsert update branches for model_cost_map_reload_config were
overwriting param_value with only the force_reload flag, dropping
interval_hours. This caused scheduled reloads to self-destruct
after their first execution.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
_route_streaming_logging_to_handler is called via asyncio.create_task()
after streaming chunks are already delivered to the client. Any
unhandled exception in this logging task propagates as an unhandled
asyncio task exception, polluting error logs.
Wraps the method body in try/except to log errors without propagating.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The OpenAI SDK raw_response.parse() can return a plain str instead
of a Pydantic model when Azure returns a non-JSON content type (e.g.,
HTML error page, proxy error). Calling .model_dump() on the str then
raises AttributeError.
Adds isinstance(response, str) checks before all 4 model_dump() call
sites in the Azure chat completion and embedding paths.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Parse VIDEO modality in promptTokensDetails → prompt_tokens_details.video_tokens
- Parse VIDEO modality in candidatesTokensDetails → completion_tokens_details.video_tokens
- Parse VIDEO modality in cacheTokensDetails and subtract from prompt video tokens
- Add video_tokens field to PromptTokensDetailsWrapper and CompletionTokensDetailsWrapper
- Fix implicit caching text-token fallback to not fire when cacheTokensDetails is present
- Add 4 unit tests covering: prompt video tokens, response video tokens,
auto-calculated text fallback with video, and explicit video cache subtraction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers:
- Region + modelId correctly extracted for ap-northeast-1, us-east-1, us-west-2
- No region in path leaves modelId and optional_params unchanged
- Cross-region inference prefixes (us., eu., ap.) are not treated as region segments
- Explicitly set aws_region_name is not overridden by region in model path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a user passes model="bedrock/ap-northeast-1/moonshotai.kimi-k2.5", get_llm_provider
strips the "bedrock/" prefix and passes "ap-northeast-1/moonshotai.kimi-k2.5" to the
converse handler. Two bugs occurred:
1. modelId was encoded as "ap-northeast-1%2Fmoonshotai.kimi-k2.5" (region included),
which AWS rejects as "not a valid model identifier"
2. The region ap-northeast-1 was never extracted, so the request went to the wrong
default region instead
Fix: after stripping routing prefixes in converse_handler.py completion(), check if the
remaining path starts with a known AWS region and strip it from modelId, injecting it
into optional_params so _get_aws_region_name picks it up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract the repeated usage-combining block from both
_completion_streaming_iterator and _acompletion_streaming_iterator into a
shared static helper method _combine_fallback_usage. This brings both
functions under the PLR0915 50-statement limit, removing the noqa
suppressions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add `# noqa: PLR0915` suppression to match the async twin
`_acompletion_streaming_iterator` which already carries the same
suppression. The function's complexity is inherent (nested class,
generator with fallback logic, cleanup code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The observatory test workflow failed because the "Verify tunnel
connectivity" step used a single curl with no retries. Cloudflare quick
tunnels need time for DNS propagation, and the first lookup can return
NXDOMAIN (curl exit 6). Replace with a retry loop (10 attempts, 5s
apart) matching the pattern already used in the health check step.
Also add `# noqa: PLR0915` to `_completion_streaming_iterator` in
router.py, matching the suppression already on its async twin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
actions/checkout treats short commit hashes as branch names, causing
fetch failures. The checkout only needs the config file from the
repo, so use the default branch instead of a specific ref.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The secrets context is not available in step-level if: conditions,
causing the workflow file to fail validation. Move the conditional
check into the shell script instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass AZURE_API_KEY, AZURE_API_BASE, OBSERVATORY_URL,
OBSERVATORY_API_KEY, and REQUEST_ID through step-level env
blocks so they are never interpolated directly into shell scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Validate inputs.tag matches vX.Y.Z format to prevent script
injection via workflow_dispatch
- Pass tag via env var instead of direct interpolation in shell
- Add cleanup step to kill cloudflared and remove docker container
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add timeout-minutes: 30 to prevent runaway jobs
- Build /run-test payload with jq --arg to safely escape
TUNNEL_URL and LITELLM_MASTER_KEY values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fail early if request_id is missing or null from the /run-test
response instead of polling /run-status/null for 15 minutes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoids shell quoting issues with single quotes in JSON and
multi-line output truncation when using GITHUB_OUTPUT.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add permissions block (contents: read) per GitHub security scan
- Poll /run-status/{request_id} instead of global /queue-status
to avoid race conditions with concurrent test runs
- Add result verification step that fails the workflow if tests
did not pass or the run errored
- Fix auth header to use X-LiteLLM-Observatory-API-Key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New reusable workflow that spins up a LiteLLM container from the
release image, exposes it via cloudflared tunnel, and triggers
test runs on the Railway-hosted observatory
- Integrates into ghcr_deploy.yml for RC and stable releases
- Can also be triggered manually via workflow_dispatch
- Add placeholder litellm_config.yaml for observatory test models
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The transformation.py file was using FEATHERLESS_API_KEY (missing _AI_)
while the rest of the codebase (get_llm_provider_logic.py, utils.py)
correctly uses FEATHERLESS_AI_API_KEY. This caused 401 auth errors when
the user set FEATHERLESS_AI_API_KEY as documented.
Now checks FEATHERLESS_AI_API_KEY first (canonical name) with fallback
to FEATHERLESS_API_KEY (legacy compatibility). Same fix applied to
FEATHERLESS_AI_API_BASE.
Refs: #22490
Add detailed UI walkthrough for Project Management feature including:
- Beta notice with link to API documentation
- Overview of projects and organizational hierarchy
- Prerequisites and setup instructions
- Separate section for enabling projects in UI settings
- Step-by-step guide for creating and managing projects
- Use cases for key organization within teams
- Next steps and related documentation links
- Proper sidebar navigation integration
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
A new QueryClient() was instantiated inside 5 component render functions
and at module-level in 3 more pages, with no shared QueryClientProvider
in either layout. This caused isolated, ephemeral caches with no
cross-page sharing and cache destruction on every re-render.
Moves QueryClient to a single module-level constant in AntdGlobalProvider
(the existing root-level "use client" wrapper in app/layout.tsx) and
removes the per-page QueryClient instantiations and QueryClientProvider
wrappers from all 8 affected pages.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Enables guardrail processing for OCR requests and responses. Adds OCR handler under litellm/llms/mistral/ocr/guardrail_translation/ to process document URLs on input and extracted page markdown on output. Includes route-to-call-type mappings for /ocr and /v1/ocr endpoints. Adds 14 unit tests and 4 e2e tests verifying handler discovery, input/output processing, and integration with UnifiedLLMGuardrails.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Moves is_accepted=True from GET /onboarding/get_token to POST /onboarding/claim_token,
so the flag accurately reflects that a password has been set. Both endpoints now reject
already-used links, with get_token rejecting before any user data is returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(ui): add missing provider logos and map all backend providers to UI
- Downloaded 26 SVG logos from lobehub/lobe-icons for providers that were
missing visual branding (AI21, Baseten, Cloudflare, GitHub, Huggingface,
Hyperbolic, Lambda, LM Studio, Meta Llama, Moonshot, Nebius, Novita,
Nvidia NIM, Replicate, Recraft, Topaz, V0, Vercel, Watsonx/IBM,
Xinference, Friendli, Morph, Cometapi, Featherless, Langfuse, GitHub Copilot)
- Extended Providers enum from 47 to 107 entries to cover all backend
providers from provider_create_fields.json
- Extended provider_map to map all new enum keys to litellm_provider values
- Extended providerLogoMap to assign logos to all providers where available,
reusing parent logos for variants (e.g. Anthropic Text -> anthropic.svg)
- Fixed SVG currentColor issue: replaced fill='currentColor' with explicit
colors since CSS inheritance doesn't work in <img> elements
- Updated test reference from Providers.Watsonx to Providers.WATSONX
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs(agents): add UI dashboard dev notes to Cursor Cloud instructions
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* refactor(ui): remove non-LLM providers from Add Model dropdown
Remove Custom, Custom OpenAI, GitHub, Humanloop, Langfuse, Litellm Proxy,
and Milvus from the Providers enum, provider_map, and providerLogoMap.
These are not LLM API providers (they are internal tools, vector stores,
or observability platforms) and should not appear in the Add Model form.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
The `gemini/gemini-2.5-flash-image` entry had `litellm_provider` set to
`vertex_ai-language-models` instead of `gemini`. This causes a provider
mismatch in `_check_provider_match()` when the model is used via the
Gemini API provider (`custom_llm_provider="gemini"`), resulting in a
noisy error log on every request:
"This model isn't mapped yet. model=gemini/gemini-2.5-flash-image,
custom_llm_provider=gemini"
The `vertex_ai/gemini-2.5-flash-image` entry already exists with the
correct `vertex_ai-language-models` provider, and the sibling
`gemini/gemini-2.5-flash-image-preview` entry correctly uses `gemini`.
Some OpenAI-compatible providers return null for top_logprobs when
logprobs=true but top_logprobs is unset or 0. The OpenAI spec requires
top_logprobs to be Array<TopLogprob> (never null), so this triggers
Pydantic validation errors while parsing responses.
Add a Pydantic v2 field_validator on ChatCompletionTokenLogprob that
normalizes None -> [] before type validation. This preserves the typed
List[TopLogprob] contract for downstream consumers while remaining
narrowly scoped to null only (other invalid types are still rejected).
Fixes#21932
* fix: add sync streaming mid-stream fallback + fix 429 for all streaming paths
Some LiteLLM providers (Vertex AI, Bedrock, Predibase, Codestral) use a
deferred HTTP pattern where the streaming HTTP request is made lazily on
the first iteration, not during completion()/acompletion(). This means
errors surface during __next__/__anext__, outside the Router's
retry/fallback machinery.
Two gaps existed:
1. __anext__ had a blanket 4xx filter (PR #18698) that blocked 429 from
MidStreamFallbackError — fixed here by exempting 429.
2. __next__ had NO MidStreamFallbackError support at all, and the Router
had no sync streaming fallback wrapper — both added here.
Changes:
- streaming_handler.py: Extract shared _handle_stream_fallback_error()
used by both __next__ and __anext__. Maps exceptions, filters
non-retriable 4xx (excluding 429), wraps everything else in
MidStreamFallbackError.
- router.py: Add _completion_streaming_iterator() (sync mirror of
_acompletion_streaming_iterator). Modify _completion() to wrap
streaming responses. Add is_pre_first_chunk check to both async
and sync iterators to skip continuation prompt on pre-call errors.
Fixes#22296
Relates to #20870, #8648, #6532
* fix: no-op assertion in sync streaming fallback test
The assertion `... is None or True` always evaluated to True,
meaning it never actually verified anything. Replace with a
proper check that messages match the original (no continuation
prompt on pre-first-chunk errors).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
