mirror of
https://github.com/BerriAI/litellm.git
synced 2026-08-28 05:25:59 +00:00
4874 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Sameer Kankute
|
1ccc1e5b23
|
chore: litellm oss staging160626 (#30527)
* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit |
||
|
Yassin Kortam
|
cd26f7d77a
|
feat(proxy): add verification_uri_complete to CLI SSO device flow (#30571)
* feat(proxy): add verification_uri_complete to CLI SSO device flow
Add an opt-in verification_uri_complete to POST /sso/cli/start. The URL is
the existing /sso/key/generate?source=litellm-cli&key=<login_id> browser-start
URL with an added user_code query param. The code is carried through the OAuth
flow via the same state channel that already carries login_id, and the post-SSO
verify page pre-fills the user_code input (HTML-escaped) so same-host clients
confirm rather than transcribe.
The manual flow is unchanged and remains the default: when no user_code is
present the verify page renders the empty input byte-for-byte as before, and
submission still hashes and compare_digest-checks both the user_code and the
browser_complete_token. Pre-filling is a UX shortcut, not an auth bypass.
Resolves LIT-3693
* fix(proxy): validate CLI SSO user_code and clarify pre-filled verify page
Address Greptile review on the verification_uri_complete flow. Guard the
user_code query param with the canonical server-issued format
([A-HJ-NP-Z2-9]{4}-[A-HJ-NP-Z2-9]{4}) before it is threaded into the OAuth
state, so an actor who knows a login_id cannot bloat the size-limited state
with an arbitrary value; a non-conforming code falls back to the manual flow.
Make the verify-page instruction conditional so the pre-filled page reads
"Confirm the verification code below" instead of pointing at a terminal that,
in the daemon use case, does not exist.
* fix(proxy): modern union syntax for new CLI SSO params and regen dashboard types
Use str | None instead of Optional[str] on the CLI SSO signatures touched by
this PR so the ruff strict-rule budget (UP045) stays under its ceiling, and
regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts so the dashboard API
types pick up the new optional user_code query param on /sso/key/generate.
* fix(proxy): gate CLI SSO verification_uri_complete behind operator opt-in (default off)
Gate verification_uri_complete behind a new general_settings flag
allow_cli_sso_verification_uri_complete, default false. When off, /sso/cli/start
does not return verification_uri_complete and /sso/key/generate ignores the
user_code query param, so the default deployment keeps the existing manual flow.
Same-host clients, where the device that starts the flow and the browser run on
the same machine, opt in explicitly. The submitted code is still hashed and
compare_digest-checked and browser_complete_token is still required. Documents
the flag on ConfigGeneralSettings and regenerates the dashboard API types.
|
||
|
Sameer Kankute
|
816fca939f
|
chore(oss): litellm oss staging 150626 (#30463)
* fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <kursad.lacin@brado.net> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): apply black formatting to modelscope chat transformation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove unused AllMessageValues import Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert: restore base_model_iterator.py to original PR state Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @override to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <github@jaytau.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: ztko <96878659+koztkozt@users.noreply.github.com> Co-authored-by: Nahrin <nahrin@nahrinoda.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Humphrey <a739376838@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: Dushyant Acharya <dushyantacharya873@gmail.com> Co-authored-by: Yuriy <yuriy.shuyskiy@gmail.com> Co-authored-by: Recep S <22618852+us@users.noreply.github.com> Co-authored-by: Moshe Malawach <moshe.malawach@protonmail.com> Co-authored-by: Moshe Malawach <moshemalawach@users.noreply.github.com> Co-authored-by: Rongkun Yan <2493404415@qq.com> Co-authored-by: Varshith <kvarshithgowda@gmail.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
5c41aabab0
|
feat(ui): cut the teams page over to the /ui/teams path route (#30343)
* feat(ui): cut the teams page over to the /ui/teams path route OldTeams renders from app/(dashboard)/teams/page.tsx via useAuthorized. The component already fetched its own paginated, filtered team list through v2TeamListCall; setTeams was only a round-trip back into the shell's lifted state, so it becomes internal useState. The organizations prop was redundant with the useOrganizations hook the component already calls, and searchParams was never read, so both are dropped. The shell keeps its own teams state and fetch for the still-coupled api-keys and models arms. Tests no longer inject teams through a prop; they mock teamListCall to drive what the component renders. * test(ui): drop unnecessary as-any casts on teamListCall mocks teamListCall returns Promise<any>, so mockResolvedValue already accepts the payload untyped. The casts pushed the repo-wide no-explicit-any lint budget over its ceiling in CI. * test(ui): drop redundant as-any casts from team mock fixtures The mocked teamListCall resolves an any-typed payload, so the inner team fixtures no longer need casts to carry a null organization_id, a keys_count field, or partial key objects. |
||
|
ryan-crabbe-berri
|
e6423d3b74
|
fix(ui): stop Virtual Keys page from infinite render loop (#30397)
The keys-filter effect called setFilteredKeys with a fresh array on every [keys, filters] change, and the caller passed keys?.keys || [] which mints a new array identity each render when the list is empty or loading. That unstable reference re-fired the effect every render, looping until React's max update depth. The effect now bails out when the filtered result is unchanged (matching the sibling teams/orgs effects), and the caller memoizes the array it passes in. |
||
|
ryan-crabbe-berri
|
685ec00afd
|
feat(ui): cut the organizations page over to the /ui/organizations path route (#30336)
* feat(ui): cut the organizations page over to the /ui/organizations path route
OrganizationsTable now owns its data through React Query instead of lifted
shell state: useOrganizations (extended with optional org_id/org_alias
filters) replaces the organizations/setOrganizations prop pair, and a new
useUserModels hook replaces the userModels prop. Create and delete
invalidate the organizations list queries rather than refetching into a
parent setter. The dead currentOrg and guardrailsList props are removed.
The shell keeps its own fetchOrganizations call because the teams and
api-keys arms still read the lifted organizations state; the userModels
state had no remaining readers and is deleted.
* fix(ui): seed organization detail initialData from any cached list, not just the unfiltered one
useOrganization's initialData only read organizationKeys.list({}), so on a
session that only ever fetched a filtered organization list the detail view
fell back to a loading state and a redundant info call. Scan every cached
list variant via the lists() prefix instead, with regression tests covering
the filtered-cache hit and the no-cache fallthrough.
|
||
|
yuneng-jiang
|
2fad75ffda
|
build(ui): pin esbuild to 0.28.1 via overrides (#30390)
esbuild reaches the dashboard tree transitively through vite (the vitest stack), and vite 7.x still requests ^0.27.0, so it never advances past 0.27.x on its own. An overrides entry forces 0.28.1, which the osv-scan job reports as the fixed version for the dev dependency. Keeping it in package.json (not just the lockfile) means a lockfile regeneration cannot silently roll it back. |
||
|
ryan-crabbe-berri
|
1fab5c4ca6
|
chore(ui): remove dead UI components unreferenced by any page (#30340)
These components are not reachable from any page: WebRTCTester, the agents agent_card/agent_card_grid pair (the agents route renders a Table directly), and view_logs ErrorViewer/RequestResponsePanel (logs uses inline equivalents in columns.tsx). Each was imported only by its own test, so the tests go too. Also drops the now-stale agent_card_grid vi.mock in agents.test.tsx and the WebRTCTester eslint-suppressions entry. |
||
|
Yassin Kortam
|
ec9353cb69
|
feat(guardrails): add Cisco AI Defense integration (#28249) (#30338) | ||
|
ryan-crabbe-berri
|
0dc203bd65
|
feat(ui): cut the users page over to the /ui/users path route (#30334)
ViewUserDashboard renders from app/(dashboard)/users/page.tsx via useAuthorized and useTeams instead of the legacy ?page=users switch arm. The keys and setKeys props were declared but never destructured by the component, so they are removed from the interface rather than wired into the wrapper. |
||
|
ryan-crabbe-berri
|
8b9a90e0fc
|
feat(ui): migrate agents and router-settings to path routes (#30323)
* feat(ui): cut agents and router-settings over to path routes Both pages depended on a slice of the legacy shell's lifted state, now replaced with React Query hooks in their route wrappers: agents pulls teams from useTeams, and router-settings feeds the Fallbacks model dropdown from useAllProxyModels. The shell's modelData copy only populated after visiting the Models page in the same session, so the dropdown was empty on a fresh load of router-settings; the hook fixes that as a side effect of the cutover. * refactor(ui): delete the dead modelData prop chain AddFallbacks fetches its own model list when its modal opens and never reads the models prop, so the whole shell modelData -> GeneralSettings -> Fallbacks -> AddFallbacks chain fed a prop nobody consumed; RouterSettings declared it without using it at all. Remove the chain and the router-settings wrapper's useAllProxyModels adaptation that was feeding it. Also corrects this PR's earlier claim: the Fallbacks dropdown was never broken by the empty shell state, because the component self-fetches. |
||
|
ryan-crabbe-berri
|
e5a3083c2e
|
refactor(ui): remove unreachable /chat page (#30178)
The /ui/chat route is not linked from anywhere: no sidebar entry, no redirect, and no backend reference. It is only reachable by typing the URL by hand. Delete the route (src/app/chat) and its components (src/components/chat), which nothing else imports, and drop the deleted files' entries from the eslint suppressions baseline. |
||
|
yuneng-jiang
|
d96ab467f1
|
chore(deps): bump vitest, brace-expansion, pypdf and tornado (#30220)
* chore(deps): bump aiohttp to 3.14.1 and vitest to 3.2.6
Lockfile-only bump for aiohttp (3.13.5 -> 3.14.1, within the existing
pyproject constraint) and dashboard devDependency bumps for vitest,
@vitest/coverage-v8, @vitest/ui (3.2.4 -> 3.2.6) plus transitive
brace-expansion (5.0.5 -> 5.0.6). Clears the currently published
advisories flagged by osv.dev against uv.lock and the dashboard
lockfile. Verified: 154 custom_httpx unit tests and all 3943 dashboard
vitest tests pass; live proxy completion and streaming calls succeed on
the bumped venv
* chore(deps): raise aiohttp floor to 3.14.0
The lockfile bump alone only protects environments built from uv.lock.
Raising the pyproject floor extends the same minimum to package
consumers installing litellm from PyPI, and prevents a future lockfile
regeneration from resolving below 3.14.0
* Revert "chore(deps): raise aiohttp floor to 3.14.0"
This reverts commit
|
||
|
ryan-crabbe-berri
|
d258e022d1
|
feat(ui): cut admin-panel, logging-and-alerts, model-hub-table, and usage over to path routes (#30268)
admin-panel pulls proxySettings from the shared useProxySettings query hook (dropping the last reader of the legacy page's copy), the model hub wrapper keeps the admin-vs-public branch as an early return, and the usage wrapper feeds NewUsagePage from the useTeams and useOrganizations query hooks instead of the lifted switch state. new_usage maps to the /usage segment while the old ?page=usage report keeps its legacy arm, asserted in the unit test so the two cannot be confused. |
||
|
ryan-crabbe-berri
|
76b4c4b111
|
fix(ui): gate dashboard layout on ui config load so deep links work under SERVER_ROOT_PATH (#30312)
* fix(ui): gate dashboard layout on ui config load so deep links work under SERVER_ROOT_PATH * test(ui): create ui config deferred per test so the pending state stays repeatable |
||
|
ryan-crabbe-berri
|
40301820e7
|
feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes (#30267)
* feat(ui): cut caching, cost-tracking, transform-request, ui-theme, and logs over to path routes Completes the simple-leaf portion of the page-by-page App Router migration. All five legacy switch arms passed only identity props (accessToken/userRole/userID, plus token/premiumUser for caching and logs), all of which useAuthorized() provides, so each route wrapper is a thin useAuthorized() + render. MIGRATED_PAGES routes the sidebar and redirects the legacy ?page= URLs; the e2e fixture picks all five up in the migration smoke and sidebar specs automatically. * refactor(ui): colocate caching, cost-tracking, transform-request, and ui-theme components Each had the legacy switch as its only importer. caching takes its whole closure (cache_dashboard, cache_health, cache_settings, response_time_indicator); CostTrackingSettings moves as the cost-tracking components folder; the transform-request and ui-theme single-file panels move under their routes. view_logs stays at src/components: six other pages (guardrails monitor, tool policies, pass-through, MCP toolsets, usage) import it. Suppressions re-keyed. * chore: retrigger ci e2e_ui_testing failed on three specs unrelated to this PR's pages (team-info tabs, MCP create form) and local_testing_part1 on test_batch_completions; all pass on the pre-merge commit and none touch files in this diff. |
||
|
ryan-crabbe-berri
|
2893f9b67b
|
feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes (#30263)
* feat(ui): cut policies, guardrails, prompts, tool-policies, and skills over to path routes Continues the page-by-page App Router migration. All five legacy switch arms passed only accessToken/userRole, so each route wrapper is a thin useAuthorized() + render. skills keeps a claude-code-plugins alias in MIGRATED_PAGES because the old switch matched both page ids, mirroring the api_ref/api-reference precedent. * refactor(ui): colocate the prompts panel under its route The new route wrapper was its only importer, so the 32-file folder moves wholesale into (dashboard)/prompts/components; tree-escaping relative imports (networking, molecules, common_components) become @/components aliases and the suppressions baseline is re-keyed. policies, guardrails, claude_code_plugins, and ToolPoliciesView stay at src/components: each has consumers on other pages (playground selectors, AI Hub, public model hub), so their shared/page splits go in the colocation follow-up. * fix(ui): move the PromptsPanel file along with its folder @/components/prompts resolved to the prompts.tsx FILE next to the prompts/ folder, not the folder itself; the colocation moved only the folder, so the wrapper's ./components import and the panel's ./prompts/* imports both broke and next build failed. Move the panel in as components/index.tsx and fix its now-escaping relative imports. Caught by next build; tsc --noEmit missed it because incremental mode reused a stale tsbuildinfo. * test(ui): lock skills alias resolution in legacyKeyForPathname Both skills and claude-code-plugins map to the skills segment, and sidebar highlighting depends on first-match-wins returning the sidebar key; assert it so a future reorder of MIGRATED_PAGES cannot silently break highlighting. Mirrors the api_ref/api-reference assertion. Flagged by Greptile. |
||
|
Sameer Kankute
|
079c136742
|
chore(oss): litellm oss staging 120626 (#30292)
* feat(bedrock): add bedrock mantle gemma 4 models (#30264) * feat(bedrock): add bedrock mantle gemma 4 models * test(bedrock): harden mantle local cost fixture * feat(responses): enable the responses API for the Tensormesh provider (#30209) * feat(responses): enable the responses API for the Tensormesh provider * Update litellm/llms/openai_like/providers.json Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(langfuse_otel): mark LLM spans as generations (#30250) * fix(bedrock): stop stream_chunk_size leaking into invoke request bodies (#30240) stream_chunk_size is a LiteLLM-internal knob for re-chunking the HTTP response stream. The invoke transformations splat optional_params into the provider request body without dropping it, and Bedrock rejects unknown fields, so any bedrock/invoke request that sets the parameter fails with ValidationException: stream_chunk_size: Extra inputs are not permitted. Drop it in the invoke dispatcher (covers cohere, titan, mistral, meta, ai21) and in the Claude messages-format request builder (the route used for bedrock/invoke Anthropic models) * fix(bedrock): stop buffering streamed tool-call argument deltas (#30231) * fix(bedrock): stop buffering streamed tool-call argument deltas Two issues made Bedrock tool-use streaming arrive as a single end-of-stream burst through LiteLLM while plain text streamed fine. First, the anthropic-beta allowlist mapped fine-grained-tool-streaming-2025-05-14 to null for bedrock and bedrock_converse, so the header was silently stripped. Without that beta, Anthropic models on Bedrock buffer tool input server-side and emit all toolUse.input deltas at once (verified against converse-stream and invoke-with-response-stream directly). Bedrock accepts the beta via additionalModelRequestFields.anthropic_beta, so it is now forwarded. Second, the streaming reads re-chunked the AWS event stream with iter_bytes(chunk_size=1024). httpx's ByteChunker only releases full 1024-byte blocks, so the small early events (messageStart, contentBlockStart, first deltas) sat in the buffer until enough bytes accumulated, pushing time-to-first-byte from ~1.4s to ~8.5s on buffered tool-use streams. The default is now no re-chunking; an explicit stream_chunk_size is still honored. * test(bedrock): cover explicit stream_chunk_size on sync invoke path * test(bedrock): cover stream_chunk_size plumbing through converse completion * test(bedrock): cover stream_chunk_size default in legacy BedrockLLM streaming * test(bedrock): merge converse handler tests into existing mapped test file pytest imports test modules by basename in non-package test dirs, so the new tests/test_litellm/llms/bedrock/chat/test_converse_handler.py collided with the pre-existing tests/test_litellm/llms/chat/test_converse_handler.py and broke collection in CI. Move the new tests into the existing file * feat(otel): emit v2 cost breakdown + stamp tracer scope version (#30156) Read the StandardLoggingPayload cost_breakdown into a typed LLMCost on LLMCallSpanData and emit each component under litellm.cost.* (absent components omitted, so spans stay sparse). Stamp litellm.__version__ as the instrumentation scope version so every v2 span carries a deterministic scope.version. Tests under tests/test_litellm/integrations/otel/. * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) (#30223) * fix(proxy): cancel in-flight upstream LLM request on client disconnect (opt-in) On the non-streaming path, base_process_llm_request awaited the LLM call with no disconnect monitoring; when the HTTP client went away the upstream request kept running until completion or request_timeout (6000s default), holding a backend slot (e.g. a vLLM GPU slot) for output nobody would read Add an opt-in general_settings.cancel_on_disconnect flag, default off, so the default code path is unchanged. When enabled, a receive-based watcher task observes http.disconnect and cancels the asyncio.gather driving the upstream call. The resulting CancelledError is converted to HTTPException 499 only when the disconnect event is set, so server-initiated cancellations still propagate as-is. The 499 then flows through _handle_llm_api_exception like any other failure, meaning post_call_failure_hook still releases max_parallel_requests slots and fires spend and alerting callbacks; it is logged at info level instead of a full traceback Also removes the dead check_request_disconnection helper in proxy_server.py (zero call sites) along with its behavior-pin tests Builds on the receive-based design from #25776 Addresses #13774. Re-fixes #22805 (regressed after the #14295 revert) Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(proxy): scope 499 quiet logging to disconnects and harden watcher Address the two P2 findings from the Greptile review on #30223. The info-level logging in _log_llm_api_exception now applies only to the disconnect-specific HTTPException (status 499 plus the shared _CLIENT_DISCONNECT_DETAIL message), so any other 499 raised by hooks or guardrails keeps its full traceback. The disconnect watcher now catches exceptions from request.receive() (e.g. a transport reset) and logs a warning instead of dying silently, making the degradation to no-op visible; a test pins that the LLM call is not cancelled in that case --------- Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> * fix(bedrock): grant aws-external-anthropic:* in OIDC session policy for claude_platform (#30200) (#30205) The inline STS session policy passed to assume_role_with_web_identity acts as an IAM PERMISSION CEILING — effective permissions are the intersection of the role's identity policies and this policy. Any action not listed is silently denied even when the IAM role grants it. #27678 added the bedrock/claude_platform/<model> route but its service-side action namespace is aws-external-anthropic:*, not bedrock:*. Without a matching statement here, every claude_platform request via OIDC (GCP federation, EKS Pod Identity webhook, etc.) 403s with 'no session policy allows the aws-external-anthropic:CreateInference action' — even with a fully permissive identity policy. Add a second ClaudePlatformLiteLLM statement covering CreateInference, CreateBatchInference, CancelBatchInference, DeleteBatchInference, CountTokens, Get*, List*. Keep aws:SecureTransport=true parity with the bedrock statement. Static creds + IRSA flow through different code paths and are not affected. Fixes #30200 * fix(proxy): set Retry-After header on RouterRateLimitError 429 responses (#30098) * Set Retry-After header on RouterRateLimitError responses When all deployments for a model are in cooldown, the proxy returns a 429 whose cooldown timing is only available by parsing the error message string. RouterRateLimitError already carries cooldown_time, so expose it as a standard retry-after header in _handle_llm_api_exception. The value is rounded up so clients never retry before the cooldown window ends. Fixes #27823. * Set Retry-After after response-headers hook so cooldown wins The cooldown-derived retry-after was assigned before the post_call_response_headers_hook merge, so a callback returning a retry-after key (including a stale or empty value) silently clobbered it. Move the RouterRateLimitError block after the callback merge so the cooldown value is authoritative for this error type. * fix(router): route aspeech through async_function_with_fallbacks (#30104) * fix(router): route aspeech through async_function_with_fallbacks Router.aspeech selected a deployment and awaited litellm.aspeech directly, so TTS requests got no retry on failure and no failover to backup deployments; the except block only fired an exception alert and re-raised. Every other router endpoint (acompletion, aembedding, atranscription, arerank) already delegates to async_function_with_fallbacks Mirror the atranscription pattern: move deployment selection and the litellm.aspeech call into a private _aspeech method, then have the public aspeech set kwargs["original_function"] = self._aspeech and await self.async_function_with_fallbacks(**kwargs). _aspeech also picks up the shared _get_async_openai_model_client helper and the same total/success/fail call accounting the sibling endpoints use Fixes #27778. * fix(router): apply deployment kwargs and rpm semaphore in _aspeech Bring _aspeech fully in line with _atranscription: call _update_kwargs_with_deployment so deployment metadata, model_info, timeout, and default litellm params flow into the request, and wrap the litellm.aspeech call with the max_parallel_requests semaphore plus async_routing_strategy_pre_call_checks so TTS respects rpm limits the same way the other router endpoints do Also add a unit test that exercises _aspeech directly and asserts the deployment metadata reaches the underlying call * fix(slack_alerting): stop false-positive hanging request alerts for requests below the alerting threshold (#30106) * fix(slack_alerting): skip hanging request alerts below the threshold The hanging request check alerted on any cached request whose completion status was not yet recorded, with no minimum age check. Since the background loop runs every alerting_threshold / 2 seconds, any request that happened to be in flight at a check fired a "hanging - Ns+ request time" alert even if it was only seconds old, producing a steady stream of false positives. Add a created_at timestamp to HangingRequestData, stamped when the request enters the hanging request cache, and skip requests younger than alerting_threshold without evicting them, so a later check can still alert if they never complete. Extend the cache TTL from threshold + 60s to 1.5x threshold + 60s; with the age check, entries only become alertable after threshold seconds, and the check period is threshold / 2, so the old TTL could evict a genuinely hanging request before any check saw it cross the threshold. Fixes #27855. * fix(slack_alerting): alert once per hanging request The min-age gate stops false positives for young in-flight requests, but a genuinely hanging request still re-alerted on every checker tick within the cache TTL. With the wider TTL (1.5x threshold + 60s) that is 1-2 extra Slack notifications per stuck request at the default 600s threshold. Flag a HangingRequestData entry as alerted once its alert fires and skip flagged entries on later ticks, so each hang produces exactly one alert. The cache reference is mutated in place, so the TTL is untouched and still handles cleanup. Adds a regression test asserting one alert across multiple ticks. Fixes #27855. * fix(health): treat all-proxy-models keys as unrestricted in /health (#30087) * fix(health): treat all-proxy-models keys as unrestricted in /health A key granted all model permissions stores the literal "all-proxy-models" marker in its models list. The /health access filter compared that marker against real model_names, so the model list filtered down to nothing and the WebUI health check returned healthy_count=0, unhealthy_count=0 with HTTP 503. Skip the filter (both the live path and the background-cache model_id scoping) when the marker is present, matching how auth_checks treats SpecialModelNames.all_proxy_models. Fixes #29744. * fix(health): resolve all-team-models sentinel to the team allowlist Same failure shape as the all-proxy-models case: a key carrying the literal "all-team-models" entry matches no real model_name, so the /health access filter would zero out the model list. Resolve the sentinel to the key's team models when team_id is set, matching get_key_models in model_checks.py. Without a team_id the sentinel stays unresolved and matches nothing, denying rather than widening access, mirroring _resolve_key_models_for_auth_check. * feat(proxy): auto-enable drop_params for Claude Code requests (#30218) * feat(proxy): auto-enable drop_params for Claude Code requests Claude Code identifies itself with a claude-cli/<version> user agent and sends Anthropic-specific params (top_k, thinking, etc.) on every request. When the proxy routes those requests to a non-Anthropic provider, the unsupported params fail the call unless drop_params is configured. Detect the Claude Code user agent in add_litellm_data_to_request and default drop_params to true for those requests, without overriding an explicit drop_params value sent by the caller. * feat(proxy): respect operator litellm_settings drop_params over Claude Code default An explicit drop_params in the operator's litellm_settings (true or false) now suppresses the Claude Code user agent default, so an operator who deliberately configured drop_params: false keeps strict param validation for Claude Code clients too. The auto-default only fills the gap when neither the request body nor the config sets a value. * fix(snowflake): migrate to native endpoints with auto-routing for Claude models (#29964) * fix(snowflake): migrate to native Cortex REST API endpoints Replaces the legacy /api/v2/cortex/inference:complete endpoint with the native OpenAI-compatible /api/v2/cortex/v1/chat/completions endpoint, fixing error 390142 (Incoming request does not contain a valid payload) when using model: snowflake/<model> in LiteLLM proxy. Changes: - litellm/llms/snowflake/chat/transformation.py: route to native /cortex/v1/chat/completions, remove Snowflake-specific tool_spec payload transformation, remove content_list response handling, add stream to supported params - litellm/llms/snowflake/anthropic/transformation.py (new): SnowflakeCortexAnthropicConfig routes Claude models to /cortex/v1/messages with anthropic-version header and Anthropic->OpenAI response transform - tests: 29 unit tests covering URL routing, auth headers, payload format, and response parsing * fix(snowflake): map max_tokens to max_completion_tokens for native endpoint * fix: handle multi-turn tool conversations and OpenAI→Anthropic tool format conversion - _extract_system_and_messages now preserves tool_calls from assistant messages and converts them to Anthropic tool_use content blocks - tool role messages are converted to user role with tool_result content blocks (as required by Anthropic Messages API) - Added _transform_tools_to_anthropic() to convert OpenAI tool format (type/function/parameters) to Anthropic format (name/input_schema) - Added comprehensive tests for multi-turn tool conversations Addresses review feedback on PR #29964 * test: add coverage for malformed JSON and non-string tool arguments * fix(tests): update chat transformation tests for native OpenAI-compatible endpoint * style: apply black formatting * fix: resolve mypy type errors in anthropic transformation * fix: correct mypy type: ignore error codes (attr-defined) * fix: use max_tokens instead of max_completion_tokens for Snowflake endpoint compatibility * refactor: merge Anthropic config into unified SnowflakeConfig with auto-routing - Remove separate SnowflakeCortexAnthropicConfig and anthropic/ directory - SnowflakeConfig now auto-routes based on model name: - Claude models → /messages endpoint (Anthropic format) - All others → /chat/completions endpoint (OpenAI format) - No new provider needed (stays as SNOWFLAKE = 'snowflake') - Tool message transformation for Claude: tool_calls → tool_use blocks, tool role → user with tool_result - OpenAI → Anthropic tool format conversion (parameters → input_schema) - Addresses Greptile feedback about unwired SnowflakeCortexAnthropicConfig * fix: use max_completion_tokens for /chat/completions (Snowflake deprecated max_tokens on this endpoint) * fix(tests): update assertions for Claude auto-routing to /messages endpoint * fix(snowflake): add tool_choice conversion and preserve max_completion_tokens in Anthropic path * fix(snowflake): use ChatCompletionMessageToolCall objects and strip model prefix on OpenAI path * fix(snowflake): collect multiple system messages to prevent guardrail override * chore: remove committed .pyc files and add __pycache__ to .gitignore * fix: remove unused Union import * fix: restore original .gitignore (accidentally replaced in earlier commit) * feat(snowflake): add streaming response handler for both Anthropic and OpenAI SSE formats * fix: remove unused AsyncIterator and Iterator imports * fix: add missing total_tokens to ChatCompletionUsageBlock * fix(snowflake): coalesce consecutive tool results into single user message for Anthropic * fix(snowflake): handle message_start event for streaming input_tokens tracking * fix: evict last deleted model in multi-instance deployments (#28608) * fix: evict last deleted model in multi-instance deployments _delete_deployment had an early return when db_models was empty, preventing eviction of the last deleted model during reconciliation. - Remove len(db_models)==0 early return from _delete_deployment - Return None (not []) from _get_models_from_db on DB failure so callers can distinguish a transient failure from a genuinely empty DB - Guard _update_llm_router against None to skip updates on DB failure Fixes #28443 * test: remove dead MagicMock assignment in type_mismatch test * fix: update test to pass [] not None to _update_llm_router test_ProxyConfig__update_llm_router_bad_proxy_logging_raises was passing None as new_models to get through to the proxy_logging_obj check, but the None guard we added now returns early before reaching that path. Pass [] instead so the test exercises the intended AttributeError case. Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * chore: regenerate API types to sync schema.d.ts with proxy OpenAPI spec Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> * fix: invalidate Redis spend counter on /key/reset_spend (#29694) * fix: set Redis spend counter to reset_to value on /key/reset_spend Previously, the Redis spend counter was always set to 0.0 after a reset, even when reset_to was a non-zero value (partial reset). This caused the budget to be under-enforced for up to 60 seconds until the counter expired and fell through to the DB. Now the counter is set to the actual reset_to value, so partial resets are reflected correctly and budget enforcement is consistent. * test: update reset_key_spend test to match direct cache set The implementation now sets spend_counter_cache directly instead of calling _invalidate_spend_counter. Update the test to verify the in_memory_cache.set_cache call with the correct key, value, and ttl. --------- Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> * fix: add scaleway models pricing (#27659) * fix: Add embeddings support for Scaleway provider * fix: resolve merge conflicts * fix(main): clarify backend route handling for Swagger static assets (#30196) * fix(main): clarify backend route handling for Swagger static assets * fix(allowlist): add BACKEND_MOUNT_PATHS for Swagger static assets * fix(voyage): route multimodal embeddings to correct endpoint (#30193) * fix(voyage): route multimodal embeddings to correct endpoint * test(voyage): cover multimodal embedding edge cases * test(voyage): cover api key fallback * fix(voyage): raise early on missing api key and malformed image url * test(voyage): cover utils routing and helper * fix(voyage): route supported openai params for multimodal models * style: apply black formatting * fix(ui): infer Azure API version from API base (#30204) * fix(ui): infer Azure API version from API base * fix(ui): address Azure API version feedback * Update litellm/llms/snowflake/chat/transformation.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * feat(datadog): add team-scoped Datadog callback support (#29947) Enable teams to configure their own Datadog credentials via POST /team/{team_id}/callback, following the same pattern as Langfuse. * Merge pull request #29528 from aanchal22/litellm_byok-alias-merge fix(proxy): atomic merge for team model aliases and team.models on BYOK create * feat: add EmpirioLabs as an OpenAI-compatible provider (#30278) Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> * fix: resolve failing tests and lint in snowflake/team endpoints - Black-format snowflake/chat/transformation.py to fix lint failure - Update Anthropic config test to expect default max_tokens of 4096 (matches implementation) - Add AsyncMock + execute_raw mock to team_model_add cache-refresh pin test - Add model_dump mock and patch cache/logging in test_uses_atomic_array_append_with_dedup Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(test): update test_db_error_new_model_check for new _delete_deployment logic _delete_deployment no longer short-circuits on empty db_models — it now treats [] as a valid empty-DB state and proceeds to check config models. Mock get_config to return the two router deployments so they appear in combined_id_list and are protected, which matches the real-world scenario where a DB error occurs but the models are config-backed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list (#30295) * feat(proxy): register cancel_on_disconnect in ConfigGeneralSettings and config list Follow-up to #30223 per maintainer review: documents the flag in ConfigGeneralSettings with a short description and adds it to allowed_args in get_config_list so the UI and /config/list expose it. A test pins that /config/list returns the field with type Boolean, which requires both registrations to be present * chore(ui): regenerate schema.d.ts for cancel_on_disconnect --------- Co-authored-by: kursad <kursad.lacin@brado.net> * fix(datadog): never fall back to env DD_API_KEY for caller-supplied destinations Team/key-scoped Datadog loggers could be pointed at an arbitrary dd_agent_host or dd_site while omitting dd_api_key, causing the proxy's global DD_API_KEY to be sent as the DD-API-KEY header to that destination. Gate the env-var fallback behind an allow_env_credentials flag, set to False when the destination is caller-supplied, mirroring the existing langfuse/langsmith pattern. --------- Signed-off-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: daitran-tensormesh <dai@tensormesh.ai> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> Co-authored-by: Muspi Merol <me@promplate.dev> Co-authored-by: fangkang <fangkangm@gmail.com> Co-authored-by: Chris Hoogeboom <chris.hoogeboom@gmail.com> Co-authored-by: kursadlacin <kursadlacin@gmail.com> Co-authored-by: kursad <kursad.lacin@brado.net> Co-authored-by: CreateRandom <18438707+CreateRandom@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Co-authored-by: sfc-gh-nashukla <navnit.shukla@snowflake.com> Co-authored-by: Rudra Dudhat <contact.rdudhat@gmail.com> Co-authored-by: Michael <52305679+michaelxer@users.noreply.github.com> Co-authored-by: michaelxer <michaelxer@users.noreply.github.com> Co-authored-by: Quentin Champenois <26109239+Quentinchampenois@users.noreply.github.com> Co-authored-by: mauriceberentsen <mauriceberentsen@live.nl> Co-authored-by: lost9999 <56498264+lost9999@users.noreply.github.com> Co-authored-by: GaetanVDB07 <86427581+GaetanVDB07@users.noreply.github.com> Co-authored-by: Aanchal Khandelwal <aan2210khandelwal@gmail.com> Co-authored-by: Adam Dalloul <adam_dalloul@icloud.com> Co-authored-by: Adam Dalloul <adam.d.developer@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Sameer Kankute
|
7d1f68e72a
|
fix(proxy): populate access_via_team_ids on /v1/model/info (#30274)
* fix(proxy): populate access_via_team_ids on /v1/model/info Team metadata enrichment previously only ran on /v2/model/info with include_team_models=true, leaving /v1/model/info without access_via_team_ids for project model-picker flows. Co-authored-by: Cursor <cursoragent@cursor.com> * docs(dashboard): sync OpenAPI schema for /v1/model/info query params Add include_team_models and teamId to the generated schema for /model/info and /v1/model/info after the proxy endpoint gained team-access filtering. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): always return direct_access on /v1/model/info Set direct_access to true or false on every enriched model so clients can filter without treating a missing field as ambiguous. Co-authored-by: Cursor <cursoragent@cursor.com> * perf(proxy): fail fast when teamId is set without a connected DB on /v1/model/info Raise the db_not_connected error before building, enriching, and translating the model list instead of after, so a teamId query against a proxy with no database no longer wastes the full enrichment pipeline. * fix(proxy): fail fast when include_team_models is set without a database include_team_models=True relies on _populate_team_access_on_models to set direct_access/access_via_team_ids, which only runs when a database is connected. Without one, _filter_models_to_user_accessible discarded every model and the endpoint returned an empty list with HTTP 200. Mirror the teamId guard so the request fails fast with a clear db_not_connected error before any model-list work. * fix(proxy): populate direct_access on single-model /model/info lookup The /v1/model/info list path populates model_info.direct_access (and access_via_team_ids) when a database is connected, but the litellm_model_id single-model lookup returned early without it. This made the two endpoints disagree, breaking the parity assertion in test_get_specific_model. Run the same population on the single-model path so both responses match. * fix(proxy): apply no-DB fast-fail before litellm_model_id branch The teamId/include_team_models no-DB guard sat after the litellm_model_id early return, so ?litellm_model_id=X&teamId=Y with no DB returned 200 with unpopulated access fields instead of the 500 raised on every other path. Move the guard ahead of the branch so the fast-fail is uniform. * fix(proxy): apply teamId/include_team_models filters on single-model lookup The litellm_model_id early-return branch in model_info_v1 populated the team access fields but returned before the teamId and include_team_models filters ran, so a single-model lookup surfaced the deployment regardless of team access when the DB was connected. Run both filters on the single-model list before returning so the documented query params behave the same with and without litellm_model_id. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
e8e5b47fc2
|
feat(passthrough): add configurable pass-through request timeouts (#30266)
* feat(passthrough): add configurable pass-through request timeouts Allow operators to set general_settings.pass_through_request_timeout and per-endpoint timeout values, and apply them to native HTTP passthrough routes and SDK passthrough paths such as Bedrock /converse. * fix(passthrough): address CI lint and regenerate dashboard API types * refactor(passthrough): extract timeout utils to proxy-free module, fix router_timeout drop - Move resolve_llm_passthrough_timeout + resolve_pass_through_request_timeout to litellm/passthrough/timeout_utils.py (no fastapi/proxy imports at module scope) - router.py and passthrough/main.py now import from timeout_utils directly, avoiding the fastapi transitive import in pure SDK contexts - pass_through_endpoints.py re-imports from timeout_utils for backward compat - resolve_llm_passthrough_timeout now accepts router_timeout so Router(timeout=X) is respected for passthrough calls instead of being silently dropped - Use _get_httpx_client (cached) instead of bare HTTPHandler(...) in sync path to avoid creating an unclosed client per call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(router): use _explicit_timeout for passthrough to not shadow general_settings self.timeout defaults to litellm.request_timeout (6000s) when the user doesn't pass timeout= to Router(). Using it as router_timeout caused general_settings.pass_through_request_timeout to be silently ignored. Only pass router_timeout when the user explicitly set Router(timeout=X). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(timeout_utils): avoid fastapi transitive import by using sys.modules resolve_pass_through_request_timeout previously did a lazy `from litellm.proxy.proxy_server import general_settings` which loads the proxy module (and transitively fastapi) even in pure SDK contexts. Replace with a sys.modules lookup: if the proxy module is already loaded (i.e. we're inside the proxy), read general_settings from it; otherwise skip and fall back to the 600s default. No import is triggered. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(lint): remove unused imports from pass_through_endpoints.py DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS and resolve_llm_passthrough_timeout are not used in this file; only resolve_pass_through_request_timeout is. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pass_through_endpoints): re-export DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS Tests import this constant directly from pass_through_endpoints.py; re-add it to the import from timeout_utils for backward compatibility. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(pass_through_endpoints): re-export resolve_llm_passthrough_timeout for backward compat Tests import both DEFAULT_PASS_THROUGH_REQUEST_TIMEOUT_SECONDS and resolve_llm_passthrough_timeout from pass_through_endpoints.py; use noqa comments to suppress the unused-import lint warning on re-exports. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Sameer Kankute
|
7519e37d26
|
feat(proxy): enforce key/team guardrails on bedrock passthrough routes (#30194)
* feat(proxy): enforce key/team guardrails on bedrock passthrough routes
/bedrock/... passthrough routes silently skipped all guardrail hooks because
CallTypes.allm_passthrough_route had no entry in guardrail_translation_mappings.
Add a dispatcher (LlmPassthroughRouteHandler) registered for that call type that
routes to BedrockPassthroughGuardrailHandler for Bedrock Converse endpoints; wire
post_call_success_hook into both the JSON and AWS event-stream response paths in
common_request_processing, including full de-anonymization for streaming responses
with proportional text distribution across original event-stream delta frames.
* refactor(proxy): address greptile feedback on bedrock passthrough guardrails
Move botocore event-stream logic from proxy/ to BedrockPassthroughGuardrailHandler.de_anonymize_converse_stream; _handle_event_stream_allm_passthrough_route becomes a thin provider dispatcher. Pass custom_headers through _handle_non_streaming_allm_passthrough_route so early-return guardrail responses include x-litellm-call-id and related headers.
* style: run black on handler and common_request_processing
* refactor(proxy): dedupe non-streaming passthrough guardrail handling
Replace the inline JSON/eventstream block in the streaming-request branch
with a call to _handle_non_streaming_allm_passthrough_route so both paths
share one implementation and cannot diverge.
* fix(bedrock): preserve trailing bytes when re-encoding converse stream
The event-stream re-encoder only emitted parsed frames, so any trailing
bytes left after the parse loop (truncated/corrupt final frame, or fewer
than 16 bytes after the last complete frame) were silently dropped from
the de-anonymized output. Capture and re-append them so the transformer
never truncates the stream.
* fix(proxy): guard non-dict post-call hook return on bedrock passthrough JSON path
* fix(proxy): guard malformed JSON body on bedrock passthrough guardrail path
* fix(proxy): close guardrail bypass via tool result text and default-mode post-call guardrails on bedrock passthrough
Pre-call extraction only read top-level Converse text blocks, so blocked
content placed under toolResult.content[].text was forwarded to Bedrock
without the key/team guardrail seeing it. Extraction now walks nested tool
result text and write-back mutates the owning block in place.
Post-call buffering for passthrough used _has_post_call_guardrails, which
excludes event_hook=None guardrails. Those guardrails run at post_call, so
their output processing was skipped and the raw upstream body was returned.
Add a passthrough-specific predicate that counts them.
* refactor(proxy): route bedrock event-stream de-anonymization through llm passthrough dispatcher
Remove the hardcoded bedrock provider guard from common_request_processing
by delegating event-stream de-anonymization to LlmPassthroughRouteHandler,
which resolves the provider from the existing handler registry. Keeps
proxy/ provider-agnostic and reuses the same dispatch path as the input
and output guardrail handlers.
Also log instead of silently dropping the result when post_call_success_hook
returns a non-dict on the JSON and event-stream passthrough paths.
* fix(proxy): close guardrail bypass on bedrock invoke passthrough routes
Pre-call extraction and post-call output processing only handled Converse
shapes, so /bedrock/model/{modelId}/invoke and invoke-with-response-stream
returned unguarded. An authenticated caller could move blocked content into
an InvokeModel payload and skip the key/team guardrail entirely.
Non-Converse Bedrock routes now fall back to the generic passthrough handler,
which scans the full request and response payloads so blocking guardrails
still run, matching how other passthrough providers are guarded.
* fix(proxy): keep non-bedrock passthrough streams streaming under post-call guardrails
* fix(bedrock): scan non-text converse blocks for passthrough guardrails
Key/team guardrails on bedrock converse passthrough only saw top-level
text blocks, so a caller could hide prompt content in toolUse.input or
toolResult.content[].json and have it forwarded to Bedrock without the
configured guardrail inspecting it, bypassing blocking guardrails by
default. Walk those arbitrary-JSON subtrees and write masked values back
in place. Extend the non-streaming converse response path to the
equivalent model-output fields (toolUse.input, reasoningContent text and
citationsContent text) while leaving structural values such as reasoning
signatures and citation sources untouched.
* fix(bedrock): make passthrough guardrail string collection iterative and type-safe
Rewrite _collect_strings with an explicit stack so it no longer recurses,
satisfying the recursive-function CI guard, and widen the holder container
type so mypy accepts indexing JSON nodes by str or int keys.
* fix(proxy): scope passthrough post-call guardrail buffering to the request
Buffering the Bedrock event stream into a single non-streaming response was
gated on whether any post_call guardrail existed globally, so every
converse-stream request lost streaming once any post_call guardrail was
registered, even for keys that did not reference it. Mirror the gate used by
post_call_success_hook (should_run_guardrail against the request's merged
guardrails) so only requests whose key/team actually trigger a post_call
guardrail are buffered.
* fix(bedrock): guardrail non-text converse stream deltas on passthrough
de_anonymize_event_stream only routed delta.text through the post-call guardrail, so model output streamed in reasoningContent.text, toolUse.input or citationsContent.content[].text was forwarded raw and skipped masking/blocking. Collect every user-visible text field per contentBlockDelta, concatenate per logical stream so split mask tokens still reassemble, run them through the hook, then redistribute the guardrailed text back into the matching delta fields. This brings streaming coverage in line with the non-streaming Converse output handler.
* refactor(proxy): keep bedrock event-stream content-type detection in llms
Move the vnd.amazon.eventstream content-type check out of the proxy
passthrough path into BedrockPassthroughGuardrailHandler via the
LlmPassthroughRouteHandler dispatcher, so proxy code stays
provider-agnostic. Also patch the actually-called
_has_post_call_guardrails_for_passthrough in the malformed-body
regression test instead of the unused _has_post_call_guardrails.
* fix(proxy): forward upstream headers on bedrock guardrail passthrough responses
Mirror the non-guardrail passthrough path by merging the upstream
response headers (via get_response_headers) into the guardrailed
non-streaming and event-stream responses, so headers like
x-amzn-requestid survive when a post-call guardrail rewrites the body.
Drop the stray fastapi HTTPException import from the SDK-tree handler
test in favor of a local sentinel exception.
* fix(bedrock): scan tool definitions and additional request fields for passthrough guardrails
Converse passthrough guardrails only scanned system and message content, so
a key holder could route blocked or PII text through toolConfig tool names,
descriptions and input schemas or through additionalModelRequestFields, all
of which are still forwarded to Bedrock. Collect strings from those fields
too so key/team guardrails inspect and rewrite them, matching how the
chat-completions path forwards tool definitions to guardrails.
* fix(bedrock): log instead of silently dropping passthrough guardrail edge cases
* test(local): skip httpbin timeout probe when the service returns 5xx
local_testing_part1 was failing on test_post_delay_exceeds_per_request_timeout_raises
because httpbin.org/delay/10 intermittently answers 503 instead of delaying, so
HTTPHandler.post raised MaskedHTTPStatusError rather than the expected Timeout. The
test already means to skip when httpbin is unavailable, but its guard only probed
GET /get and ignored a server error on the delay endpoint. Treat a 5xx from httpbin as
'service unavailable' and skip, which is outside this repo's control, while still
asserting Timeout when httpbin genuinely delays.
* fix(proxy): set content-type on buffered bedrock passthrough event-stream responses
* fix(bedrock): skip passthrough output write-back when guardrail returns no texts
* fix(proxy): apply response-headers hook on guardrailed bedrock passthrough responses
* refactor(bedrock): import event-stream crc32 from binascii not botocore internals
* fix(proxy): scope bedrock passthrough stream buffering to de-anonymizable endpoints
Only buffer a passthrough event stream into a non-streaming response when the
resolved provider and endpoint actually have an event-stream guardrail handler
that can rewrite frames (Bedrock converse-stream). Other Bedrock event-stream
endpoints such as invoke-with-response-stream keep streaming, since the Converse
handler leaves their frames untouched and buffering would silently break the
streaming contract for no content change.
* test(ui-e2e): re-issue deep-link navigation when auth bootstrap drops the page param
navigateToPage deep-links to /ui?page=<page> then proceeds once the network
settles, but a fresh load can race the auth bootstrap: the app momentarily
treats the session as anonymous, bounces through /ui/login, and returns to the
default Virtual Keys page with the ?page= query param dropped. The helper never
checked where it actually landed, so any single bounce left callers asserting
against the wrong page and timing out (mcpServers, modelHub, addModel).
Confirm the requested page is what rendered and re-issue the navigation when it
was clobbered; auth is warm by the second load so the param sticks. Migrated
path routes are left alone since they intentionally leave the legacy root.
---------
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
|
||
|
Sameer Kankute
|
cfcdf8714a
|
feat: litellm oss 110626 (#30202)
* Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) (#29775) * Add gpt-realtime-whisper Realtime transcription support (OpenAI + Azure) Adds first-class support for the gpt-realtime-whisper streaming speech-to-text model, which uses the Realtime transcription session API rather than the file-based /audio/transcriptions path. Model registration: registers gpt-realtime-whisper and azure/gpt-realtime-whisper with audio-duration pricing (input_cost_per_second = 0.017/60, matching the published $0.017/minute input audio rate). REST endpoint: implements POST /v1/realtime/transcription_sessions (plus /realtime and /openai/v1 aliases) to mint an ephemeral transcription session for the WebRTC flow. Adds request/response types, OpenAI and Azure URL builders, a shared base handler (refactored from the client_secrets handler), the acreate_realtime_transcription_session SDK function, and route registration. The proxy encrypts the ephemeral key returned under client_secret.value and records the session type in the token so the follow-up /realtime/calls replays type=transcription rather than type=realtime. WebSocket: forwards intent=transcription through to the Azure handler (OpenAI already received it) with URL-encoding, so gpt-realtime-whisper opens a transcription session. Transcription-only sessions no longer trigger an erroneous response.create. Cost tracking: transcription sessions emit no response.done events; their usage arrives on conversation.item.input_audio_transcription.completed as {type: duration, seconds}. That usage is captured out-of-band (usage only, no transcript duplication) and billed by input_cost_per_second, with a token-billed fallback for token-priced transcription models. Adds tests for pricing math, URL builders, request/response types, the proxy route and SDK function, WebSocket intent forwarding, transcription-session streaming behavior, and the /realtime/calls session-type replay. * Address PR review: URL-encode all Azure WS query params; forward query_params through provider_config branch * Address PR review: session_type validation, model auth fix, cost perf, billing fallback, detail/docs cleanup * Improve test coverage: detection from backend, error paths, unknown usage type, resolved_model None * Backport realtime transcription websocket fixes * Enforce authorized realtime transcription model * Enforce realtime transcription model access * Enforce realtime resolved model scopes * Enforce WebRTC transcription model scope * Lazy evaluate debug log in pass-through endpoint (#30177) * Pass through debug lazy logging * fix(proxy): convert remaining eager pass-through debug logs to lazy formatting * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint (#30157) * fix(parallel_ai): migrate search integration from v1beta to v1 endpoint The Parallel Search API moved from /v1beta/search (processor: base/pro, parallel-beta header) to /v1/search (mode: turbo/basic/advanced, no beta header). Request fields moved too: max_results, source_policy, and excerpt settings are now nested under advanced_settings, and source_policy uses include_domains/exclude_domains. The v1 response returns publish_date per result, which now maps to SearchResult.date instead of being hardcoded to None. The legacy processor param is mapped to the equivalent mode so existing callers keep working. * fix(parallel_ai): default mode to basic and simplify param handling The v1 API defaults to advanced mode when mode is omitted, while v1beta defaulted to the base processor. Without an explicit default, callers who pass no mode would be silently upgraded to a tier costing 2.25x more while litellm's cost map reports the basic-tier price. Sending mode=basic preserves the v1beta default and keeps cost tracking accurate. Also replaces the handled_params set with pop-as-consumed param handling so mapped params no longer need to be tracked in two places, and extends the tests to pin the default mode, processor=base mapping, mode-over-processor precedence, and top-level v1 param passthrough. * fix(parallel_ai): avoid double /v1 when api_base is already versioned A PARALLEL_AI_API_BASE like https://api.parallel.ai/v1 previously produced .../v1/v1/search. Strip a trailing /v1 before appending the search path and cover the api_base variants with a parametrized test. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * feat(focus): add Mavvrik destination for FOCUS export (#29935) * fix: preserve responses streaming flag (#30189) * fix: preserve responses streaming flag * test: cover async responses streaming flag * fix(spend/daily-activity): stable offset pagination via id tiebreaker (#30164) (#30167) date alone is not a unique sort key for LiteLLM_DailyUserSpend or LiteLLM_DailyTeamSpend (many rows per date: api_key x model x model_group x provider x endpoint). Offset pagination over a non-unique sort landed on arbitrary boundaries, so a client paging through all results and summing per-page metrics (the Usage dashboard) got non-deterministic totals - sometimes inflated, sometimes deflated, different at different page_size values. Adding the row's UUID id (present on both tables) as a secondary sort gives every page a stable cursor. order=[{date desc}, {id asc}]. Fixes #30164 * fix(oci): inject a default maxTokens so omitted max_tokens doesn't truncate responses (#30018) * fix(oci): inject default maxTokens so omitted max_tokens doesn't truncate OCI GenAI applies a tiny server-side maxTokens default (~20 tokens) when the request omits it, so any call that doesn't send max_tokens comes back cut off mid-string with finishReason "length". MLflow judges never send max_tokens, so their JSON responses arrived as unterminated strings and json.loads failed in MLflow's gateway adapter. When no maxTokens/maxCompletionTokens target is set, inject DEFAULT_OCI_CHAT_MAX_TOKENS (env-overridable, defaults 4096), mirroring the Anthropic config's default-max-tokens behaviour. An explicit max_tokens still wins, and reasoning models still route to maxCompletionTokens. Used a fixed default rather than the catalog max_output_tokens because the catalog value is unreliable for some models (grok-4 reports max_output_tokens equal to its context window, not a real output cap, which would risk 400s). Adds TestOCIDefaultMaxTokens covering Cohere and generic injection, the explicit-override case, and the reasoning maxCompletionTokens branch. * test(oci): e2e regression that omitted max_tokens isn't truncated Real-proxy integration test asserting a chat completion that omits max_tokens completes with finish_reason "stop" instead of being cut off at OCI's ~20-token server default. Fails before the maxTokens-default injection (finish_reason "length", ~19 tokens), passes after. * test(oci): update cohere default-params test for injected maxTokens test_cohere_default_parameters asserted no maxTokens was injected, encoding the old behaviour where OCI's ~20-token server default truncated responses. Now that transform_request injects DEFAULT_OCI_CHAT_MAX_TOKENS, assert maxTokens equals that default while the other params (topK/topP/frequencyPenalty) stay pass-through with no hardcoded default. * fix(oci): make DEFAULT_OCI_CHAT_MAX_TOKENS a plain constant Drop the os.getenv override. The env knob was not requested and introducing a new env var forced a cross-repo dependency on litellm-docs (test_env_keys.py validates every referenced env var against the docs table there). A plain 4096 constant keeps the PR self-contained; callers who want a different limit pass max_tokens explicitly per request. * fix(oci): route all OpenAI commercial models to maxCompletionTokens OCI serves OpenAI models (gpt-4.1, gpt-5.1 through 5.5, o-series) that the litellm catalog doesn't track, so the supports_reasoning lookup returned False for them and the provider sent maxTokens, which the reasoning families reject with HTTP 400. With the injected default maxTokens this broke every request to those models, not just ones with an explicit max_tokens. Route the whole openai.* vendor prefix to maxCompletionTokens since OpenAI accepts max_completion_tokens on every chat model; the openai.gpt-oss-* open weights are served by OCI's own stack and keep maxTokens. Verified live against gpt-5.2, gpt-5, gpt-4o, gpt-4.1, gpt-oss-120b, llama-3.3, command-a and grok-3-mini * test(oci): hoist transformation imports and drop unused ones Makes the generic-chat test file ruff-clean: the per-test local imports of OCIChatConfig/OCIVendors shadowed the module-level import (F811) and left it unused (F401), and json plus three OCI type imports were never referenced * fix(oci): translate response_format json_schema to OCI's accepted shape (#29691) * fix(oci): translate response_format json_schema to OCI's accepted shape OCI GenAI rejected every json_schema response_format with HTTP 400 "Please pass in correct format of request", which broke structured-output callers such as MLflow LLM judges (they always send a json_schema). The provider forwarded OpenAI's raw json_schema body unchanged. For GENERIC models OCI's ResponseJsonSchema accepts only name/description/schema/isStrict, so OpenAI's `strict` key (and any other extra) 400s the request; the key must be renamed to isStrict and the body whitelisted. For Cohere models there is no JSON_SCHEMA type at all; the schema has to ride on JSON_OBJECT as {"type": "JSON_OBJECT", "schema": ...}. Cohere type values must also be the canonical uppercase TEXT/JSON_OBJECT. _normalize_response_format now branches by vendor and emits the exact shape each one accepts (verified live against OCI GenAI for Cohere, Meta, Gemini and Grok). Drops the unused, incorrect Cohere response-format pydantic models. Two existing tests asserted the broken behavior (lowercase type, raw jsonSchema on Cohere); they are rewritten to assert the corrected shape, and generic/Cohere json_schema regression tests are added. * fix(oci): raise early on json_schema response_format with no body A GENERIC model request with {"type": "json_schema"} and no json_schema object fell through to the JSON_OBJECT branch and emitted a bodyless {"type": "JSON_SCHEMA"}, which OCI rejects with an opaque HTTP 400. Raise a descriptive 400 at translation time instead. Cohere is unaffected since it always maps to JSON_OBJECT. * test(oci): gateway integration test for response_format json_schema Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): accept default n=1 on Cohere instead of hard-failing (#29705) * fix(oci): accept default n=1 on Cohere instead of hard-failing Cohere on OCI has no numGenerations field, so n was mapped to False and map_openai_params raised "param `n` is not supported on OCI" whenever a client sent n. But n=1 (and None) is the OpenAI default single-generation request, which every OCI model produces anyway, so standard clients that always send n=1 (such as the MLflow gateway) were rejected with a 500. Drop n=1/None silently for Cohere; only n>1 is genuinely unsupported and still raises (or drops under drop_params). Generic models are unaffected and keep numGenerations, including n>1. * docs(oci): explain why n is not advertised for Cohere despite tolerating n=1 * test(oci): gateway integration test for Cohere default n=1 Added to tests/integration/ (the real-network integration suite) reusing the existing OCI proxy harness, not tests/llm_translation/ which is mock-only. --------- Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(oci): drop max_retries instead of hard-failing on OCI (#29727) max_retries is a litellm-level control param (litellm applies retries itself), not a generation param OCI accepts. The provider mapped it to False and raised "param `max_retries` is not supported on OCI" whenever it was present. The litellm proxy injects max_retries on every request, so any OCI call through the proxy 500'd unless drop_params was set. Drop max_retries silently in map_openai_params. Adds a unit test (Cohere and generic) and a gateway integration test that a plain request succeeds through a proxy without drop_params. Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix(spend-logs): rehydrate metadata JSONB text on ui_view_spend_logs (#29682) Fixes #29674. `/spend/logs/ui` raw-SQL path returns the JSONB metadata column as a string — prisma's query_raw skips the ORM-layer hydration. The UI reads metadata.status / metadata.error_information as object fields, so provider-failure rows look like successes. Fix: json.loads the metadata field right after query_raw, fall back to {} on malformed JSON. 3 existing error-code/error-message tests called json.loads on response.data[0]["metadata"] — they were leaning on the bug. Updated to read the dict directly. Plus 2 new regression tests (failure metadata roundtrip + invalid-json fallback). Reverting the fix makes both new tests fail with AssertionError: metadata should be dict, got <class 'str'>. * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) (#30020) * fix(proxy): release max_parallel_requests slot when a stream is cancelled mid-flight (#27955) * fix: refund max_parallel_requests on disconnect from outer streaming generators The cancellation refund previously lived in async_post_call_streaming_iterator_hook, but that hook is nested inside the outer streaming generators and a nested async generator only receives GeneratorExit on garbage collection (non-deterministic). With only the v3 limiter enabled, /chat/completions also bypasses the hook entirely (needs_iterator_wrap() is false). Move the release into async_data_generator and async_streaming_data_generator, the generators Starlette closes on client disconnect, so the refund fires deterministically on every streaming route. Warn when no event loop is running, and document the window TTL refresh on the decrement * fix(mcp): propagate model into model_call_details for passthrough tool calls (#30122) * fix(mcp): propagate model into model_call_details for passthrough tool calls The @client decorator on call_mcp_tool creates the logging object via function_setup without a model kwarg, so model_call_details["model"] starts as None. execute_mcp_tool only set logging_obj.model as an instance attribute, which the spend-log writer never reads (it reads kwargs["model"] from model_call_details). MCP passthrough tools/call rows therefore persisted with model="" while list_tools rows showed "MCP: list_tools", degrading the Logs UI display and bucketing all MCP tool spend under an empty model in DailyUserSpend. Propagate the model into model_call_details alongside the existing attribute assignment so the StandardLoggingPayload and SpendLogs writer pick it up. Covers the /mcp passthrough, REST /mcp-rest/tools/call, and orchestrated paths (the latter already passed model into function_setup, so this is a no-op there). * test(mcp): trim regression test docstring * fix(mcp): surface upstream challenges for delegated OAuth (#30124) * fix(mcp): surface upstream challenges for delegated OAuth * docs(mcp): clarify delegated upstream auth comments * perf(benchmarks): add CPU timing metrics to streaming benchmark (#29980) * Add CPU timing metrics to streaming benchmark * Fix spacing around timing sample dataclass * fix(gemini): don't emit empty choices on metadata-only stream chunks (#29167) web_search + reasoning makes Gemini stream mid-chunks that carry only grounding/thought metadata — no content part, no finishReason. _process_candidates skips content-less candidates and the existing fallback only ran when finishReason was set, so choices stayed empty and the downstream streaming handler raised IndexError on choices[0]. Emit an empty-delta choice for content-less chunks regardless of finishReason. Fixes #28884 * fix(key): allow /key/update to clear budget_limits with [] or null (#30085) * Fix /key/update rejecting budget_limits clear requests with HTTP 400 Sending budget_limits: [] or null to /key/update returned HTTP 400, so once a key had budget windows the last one could never be removed. prepare_key_update_data only json.dumps'd budget_limits when the value was truthy, so [] and None passed through raw to the Prisma Json? column; jsonify_object only serializes dicts, and prisma-client-py has no DbNull sentinel for Json? writes, so Prisma rejected both shapes. Serialize the clear case explicitly as the JSON literal null, matching how memory_endpoints encodes metadata for the same column type. Truthy values keep the existing reset_at window initialization path. Fixes #30067. * Require admin access for budget_limits changes on /key/update Clearing budget_limits via [] or null is a budget mutation, but _validate_update_key_data only counted max_budget and spend as budget changes before deciding whether to skip _check_key_admin_access. A non-admin key owner or a team member with /key/update could therefore remove a key's per-window spend caps without admin authorization. Treat any explicit budget_limits value in the request (set, change, or clear) as a budget change so it gates through the same admin check as max_budget. model_fields_set is used because an explicit null is indistinguishable from an omitted field by value alone. * fix(proxy): persist guardrail info in spend logs for /v1/responses (#30092) Pre-call guardrail blocks on /v1/responses wrote guardrail_information as null in LiteLLM_SpendLogs because _handle_logging_proxy_only_error splits request_data by LoggedLiteLLMParams keys and litellm_metadata, where the Responses API stores request metadata including standard_logging_guardrail_information, was not among them. It fell into optional_params, so merge_litellm_metadata never saw it. Add litellm_metadata to LoggedLiteLLMParams so it routes into litellm_params the same way metadata does on the chat completions path Fixes #28971. * fix(proxy): handle non-standard SSE frames in Anthropic passthrough logging (#26000) Some third-party Anthropic-compatible providers emit non-standard SSE frames (OpenAI-style [DONE] sentinels, non-JSON keep-alive lines) in streaming responses. These caused json.JSONDecodeError in _build_complete_streaming_response, breaking the passthrough logging pipeline so the request was never logged or billed. Skip whole-line 'data: [DONE]' sentinels and catch JSONDecodeError per event. Matching the full line (not a substring) keeps a valid chunk whose text payload contains '[DONE]' from being dropped. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Sameer Kankute <sameer@berri.ai> * feat(newrelic): Add New Relic extension (#26989) * initial New Relic integration. * Minor fixes for basic observability. * Implemented basic support for the success path. Generates New Relic custom events needed by the AI Monitorin interface. * Supportability metric is sent on first request. * Emit supportability metric every hour instead of once a day. * Add the start/end times to the messages before sending them so that the start time and end time reflect the correct time and both are not set to 'now'. * Make use of `turn_off_message_logging` configuration that is available by default from CustomLogger. * Enabling New Relic agent to be wired when docker container starts if an environment variable is set. * If we cannot find trace information, send the AI events without the trace ID attached. * Use a fake trace_id if we cannot find one. * Implementing a configuration so that users can use litellm configuration to disable sending LLM messages to New Relic. There is a second method to do this via New Relic env var. * Mised file. * Cleaning up logic to turn off recording content via either the LiteLLM configuration or an env var. * Removing debugging. Fixed logic / comments around how often to send supportability metric. * Initial version of public doc for New Relic. * Use a proper name for the doc file. * Updating newrelic.md document. * Updating LiteLLM documentation for New Relic extension. * Moving New Relic imports into the methods to support unit tests. * Adding unit tests for the New Relic extension. * Updating linting and the unit tests that are not running in the CI environment. * Address reviewer feedback on New Relic integration. - Fix _record_error_metric to use app.record_custom_metric() instead of module-level newrelic.agent.record_custom_metric() so the call works outside of an active transaction context - Remove unreachable except ImportError block in _get_trace_context - Update stale "23 hours" comment to "27 hours" (matches 97200s threshold) - Remove commented-out debug code from _process_success - Fix docs typo: NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STOREDA -> NEW_RELIC_CUSTOM_INSIGHTS_EVENTS_MAX_SAMPLES_STORED - Update TestRecordErrorMetric to verify app.record_custom_metric call Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Reformating for the linter. * Addressing additional automated feedback. - Removed a legacy comment about the New Relic header - Reordered imports in one file - Switched another file to use the import at the top of the file instead of inline when used - Added unit tests for untested methods that were identified * Addressing new feedback. - Proper handling of time to floats. Created a util method and updated code to use it. - added the missing guard to ensure the app is enabled * Addressing feedback. - When an error occurs, still check if the periodic supportability metric should be emitted - Added a check to ensure the extension is ready in the error handler to match _process_success * Updating the NR event timestamps to more accurately reflect when the messages were generated. * Addressing feedback for potential better practice. * Addressing feedback on accessing default values. Added tests for most of these cases. * Adding a new catch exception block based on feedback. * Addressing feedback about a potential issue around a timestamp for the supportability metric. * Addressing minor feedback on length of generated, fallback traceId. * Addressing feedback. - A few more cases were found where the dictionary access might not return the correct value. - Handling cases where `traceparent` is not lower cased * Addressed feedback where the newrelic options might not apply correctly. * Addressing some feedback. * Addressing feedback. * Validating testing / formatting for our changes. * Updating linting, adding tests, defining data type for UI. * Configuration for the logging callback definition. * Adding a newrelic image for the UI to use. * Putting the New Relic callback in proper alphabetic order. * Copying the logo to a committed output directory so it shows up in a locally built container. * Adding missing definition of new env vars that were causing a build failure. * Addressing automated feedback from greptile. * Adding a few more unit tests to increase the code coverage just a bit more. * Additional unit tests to push coverage to almost 90%. * Adding a custom newrelic docker image build process. This removes the need to add the newrelic agent to the core litellm container or dependencies. * Clarifying message when the New Relic agent is not installed and someone is trying to use the newrelic extension. Either use the proper image when using docker, or install the agent manually when running from source. * Ensuring pip is available to install the New Relic agent. * Updating the definition and handling of traceId (no spanId). Clarifying behavior of env vars vs UI configuration for the newrelic extension. * Removing entries from the New Relic logger configuraiton UI as these values must be set as part of running the image. * Removing a stale doc file that has moved to the litellm-docs repo. Cleanup of Dockerfile to remove a LABEL that was incorrect. * Updating container image name to be the best guess for the new name. * Addressing feedback from greptile. - Added a comment around token_count=0 - Updated the boolean parser to allow a wider set of options which matches existing patterns in other parts of LiteLLM. * Removing option for a separate New Relic container image. The agreement is to handle this in the New Relic integration docs. * Updating error message when New Relic agent is not available. * Wiring in the test message from the LiteLLM callback UX. * Missed saving one of the file conflicts. * Fixed a lint error I introduced. Somehow, I dropped another string and now added it back. * Adding newrelic to the schema definition. * Added an admin check on the call before sending test message as mentioned by the AI code review. * Updating to use should_redact_message_logging(kwargs) as part of the logic to determine if message content should be sent to New Relic or not. This still uses the `record_content` property as well, but both have to be true in order for content to be included. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add Azure AI Foundry DeepSeek V3.1 and V4 Pro/Flash global pricing to cost map (#30134) Co-authored-by: Cursor <cursoragent@cursor.com> * fix(logging): translate Responses bridge result to ModelResponse for spend logs (#28985) PR #29394 fixed the AnthropicResponse.model_validate crash for the streaming anthropic_messages -> OpenAI Responses bridge by unwrapping terminal events and returning the inner ResponsesAPIResponse. The spend_logs row lands and usage/cost are correct, but the row's response field stores the Responses API shape (output[...].content[...].text). The proxy UI Logs tab reads response.choices[0].message via parseMessages in prettyMessagesUtils.ts with no fallback for the Responses shape, so the OutputCard renders "No response data available" for every cross-routed call. The same shape mismatch affects every downstream consumer of spend_logs that assumes the canonical chat-completion shape This change keeps the unwrap from #29394 but routes the resulting ResponsesAPIResponse (and the bare-response non-streaming path) through LiteLLMResponsesTransformationHandler.transform_response, which is the same conversion already used by the chat-completion Responses bridge. Spend_logs now stores a ModelResponse with choices[0].message.content, so the UI and other consumers see the assistant text. On a translation failure (eg. empty output on an incomplete response) the handler falls back to a minimal ModelResponse carrying model and usage so the row still lands rather than being dropped as a Non-Blocking error Also corrects a stale comment in the Responses adapter that implied the call type was reclassified to acompletion; the code preserves anthropic_messages and the success handler translates back to ModelResponse for the row Fixes #28595 * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions (#30024) * fix(anthropic-adapter): re-emit first delta on streaming content-block transitions The `/v1/messages` -> `/v1/chat/completions` streaming adapter (`AnthropicStreamWrapper`) silently dropped the first non-empty delta of every content block that started via a *transition* (e.g. text -> tool_use -> text, text -> thinking). When an upstream chunk both triggers a new content block (its type differs from the active block) and carries that block's first delta, the wrapper emitted `content_block_stop` -> `content_block_start` and then only re-queued the trigger chunk when it was an `input_json_delta` (bundled tool args). The synthesized `content_block_start` always carries an empty body, so the first `text_delta` / `thinking_delta` was lost — the client output started from the second token (e.g. "Hi, how can I help you?" rendered as ", how can I help you?", or text resuming after a tool call lost its first sentence). This is especially visible with Claude Code-style clients that consume Anthropic Messages streaming events strictly. Fix: re-queue the trigger chunk's translated delta whenever it carries non-empty content (text/thinking/signature/tool args), via a shared `_trigger_delta_has_content` helper used by both the sync and async paths. Empty trigger deltas are still suppressed so no spurious empty `content_block_delta` is introduced. Fixes #30014 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test(anthropic-adapter): cover all _trigger_delta_has_content branches Add a direct parametrized unit test for the re-emit predicate so every delta type (text/input_json/thinking/signature), the empty-payload guards, and the malformed/non-delta cases are exercised independently of upstream chunk translation. Raises patch coverage for the new helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> * feat: add opt-in healthy_only filter to GET /v1/models (#30130) * feat: add opt-in healthy_only filter to GET /v1/models Adds an opt-in `healthy_only=true` query parameter to GET /v1/models and GET /models that hides models whose backing deployments are all marked unhealthy by background health checks. - Add Router.async_get_fully_unhealthy_model_names(), mirroring the semantics of get_fully_blocked_model_names(): a model is hidden only when every backing deployment is unhealthy and the health state is not stale (fail open otherwise). - Reuses the existing DeploymentHealthCache populated by _run_background_health_check(), so no new health state is introduced. - No-op when allowed_fails_policy is set, mirroring _async_filter_health_check_unhealthy_deployments semantics. - team_public_model_name aliases are aggregated alongside model_name. - Hiding is presentation-only; default behavior is unchanged. Fixes #30128 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: address Greptile review notes - Note team-alias asymmetry vs get_fully_blocked_model_names - Debug-log when healthy_only is set but no health state is available Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> * Dedupe team soft budget alerts by team_id instead of token (#30097) _team_soft_budget_check sends type="soft_budget" alerts with event_group=TEAM, but SoftBudgetAlert.get_id always returned the request token. The alert cache key was therefore scoped per virtual key, so every active key in a team over its soft budget fired its own alert within budget_alert_ttl. Branch on event_group so team-level alerts dedupe by team_id, matching TeamBudgetAlert, while key and project level alerts keep per-token dedupe. Fixes #27398. * feat(bedrock guardrails): support contextual grounding qualifiers (request-side) (#30057) * test: add failing tests for Bedrock contextual grounding (request-side) Drive the request-side of Bedrock contextual grounding: callers tag message content blocks as grounding_source/query, the post_call hook assembles an ApplyGuardrail(OUTPUT) call carrying source + query + response(guard_content), and the bedrock converse transform must render the tags as prompt text instead of silently dropping them. Non-grounding payloads must stay byte-identical. * feat(bedrock guardrails): support contextual grounding qualifiers Bedrock contextual grounding scores a model response against a reference source and the user query, expressed via a per-content-block `qualifiers` array on ApplyGuardrail. The guardrail hook previously sent plain text only, so grounding could not be driven through it even though the response-side contextualGroundingPolicy parsing already existed. Callers now tag message content blocks `{"type":"grounding_source"}` / `{"type":"query"}` (mirroring the existing `guarded_text` marker). On the generate path the bedrock converse transform renders them as plain text; at post_call the hook harvests them from the request and assembles one ApplyGuardrail(OUTPUT) call carrying grounding_source + query + the response (as guard_content). Requests without these tags produce a byte-identical payload, so existing behaviour is unchanged. * Feat(guardrail): Adding support for custom Ovalix guardrail (#21887) * Feat(guardrail): Adding support for custom Ovalix guardrail * Internal CR comments fixes * greptileai comments fixes * fix conflict * fixes * fix sha256 * clarify Ovalix actor-id hash is for normalization, not PII protection * fix(github_copilot): normalize per-event item_id in /responses streaming (#30072) GitHub Copilot's native /v1/responses stream assigns a different item_id to every event of a single output item (output_item.added, the part.added / delta / done events, and output_item.done). Spec-strict clients like the Vercel AI SDK key streaming parts by item_id and abort with "reasoning part <id> not found" / "text part <id> not found" when a delta references an unregistered id. Override transform_streaming_response in GithubCopilotResponsesAPIConfig to anchor every event of an output item to the id from its output_item.added. Copilot accepts that id paired with the final encrypted_content on the next turn, so multi-turn replay is unaffected. Fixes #30071 * feat: add /model/block and /model/unblock endpoints (#30125) * feat: add /model/block and /model/unblock endpoints Add dedicated proxy-admin POST /model/block and /model/unblock endpoints over the existing blocked flag on LiteLLM_ProxyModelTable, mirroring the /key/block and /key/unblock pattern. Calling a model whose deployments are all blocked now returns a clear 403 "Model is blocked" instead of a generic no-deployment error, including direct-dispatch route types (e.g. eval) via a pre-route guard. Includes audit-log entries for block/unblock and unit tests. Closes #29742 Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * chore: regenerate dashboard API types for model block/unblock endpoints Regenerate ui/litellm-dashboard/src/lib/http/schema.d.ts from the proxy OpenAPI spec (npm run gen:api) so it includes the new endpoints. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: widen router block-helper param type and add direct unit tests Type the _are_all_deployments_blocked deployments parameter to match its callers (DeploymentTypedDict) so mypy passes, and add tests/test_litellm/test_router_block_helpers.py with direct unit tests for the three block helper methods so router_code_coverage recognizes them. Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * fix: restore type-ignore on messages arg after black reflow Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> * refactor: raise model-block 403 in proxy layer, not SDK Router Keep the SDK Router's documented behavior for blocked deployments (filtered -> "no healthy deployment") and move the 403 PermissionDeniedError into the proxy layer (route_llm_request), where model blocking is an admin concept. This avoids a backwards-incompatible 403 for SDK users who set blocked=True on their own deployments, per maintainer review. Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Sameer Kankute <sameer@berri.ai> * fix: add week unit support to get_next_standardized_reset_time (#30100) * fix: add week unit support to get_next_standardized_reset_time The function handled d/h/m/s/mo units but silently fell through to the default next-midnight branch for the w (week) unit. This was inconsistent: _extract_from_regex already accepted w in its character class, and duration_in_seconds already returned value * 604800 for it. Add the missing elif unit == 'w' branch that delegates to _handle_day_reset with value * 7, which reuses the existing Monday- alignment logic for 1w and the generic N-day-from-midnight path for larger multiples. Add test_week_based_resets covering 1w from a Wednesday (expects next Monday) and 2w from a Monday (expects 14 days forward at midnight). Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * test: exercise relative week semantics with non-Monday base dates + add docstring Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> --------- Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> * fix: black formatting and remove undocumented MAVVRIK_FOCUS_FREQUENCY env var * fix: black formatting with correct version and sync schema.d.ts for healthy_only param * fix: resolve mypy errors and add transcription_sessions to JSON schema endpoint enum * fix: restore MAVVRIK_FOCUS_FREQUENCY guard and exclude it from docs key scan * fix: address Greptile P2 comments - move constant, use UTC datetime, skip redundant team lookup * revert: restore original team lookup logic in can_key_call_resolved_model --------- Signed-off-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Signed-off-by: FugoP <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com> Co-authored-by: nina-hu <nina.huuu@gmail.com> Co-authored-by: Sahith Jagarlamudi <104647530+s-jag@users.noreply.github.com> Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Praveen Ghuge <95286176+pghuge-cloudwiz@users.noreply.github.com> Co-authored-by: alex107ivanov <30668368+alex107ivanov@users.noreply.github.com> Co-authored-by: hcl <chenglunhu@gmail.com> Co-authored-by: Fede Kamelhar <federico.kamelhar@oracle.com> Co-authored-by: Armaan Sandhu <74664101+Ar-maan05@users.noreply.github.com> Co-authored-by: Teo Xian Zhong Augustine <35527068+auggie246@users.noreply.github.com> Co-authored-by: King Star <mcxin.y@gmail.com> Co-authored-by: Saksham Maggo <122939011+SakshamMaggo@users.noreply.github.com> Co-authored-by: Filippo Menghi <113345637+Cyberfilo@users.noreply.github.com> Co-authored-by: Kelvin <leikaiwei@outlook.com> Co-authored-by: Josh Bonczkowski <josh.bonczkowski@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: M. Dennis Turp <mdturp@pm.me> Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Piotr Minkina <piotrminkina@users.noreply.github.com> Co-authored-by: Martín Alcalá Rubí <martin@tryolabs.com> Co-authored-by: T. Kobayashi <13004314+nix-tkobayashi@users.noreply.github.com> Co-authored-by: João Costa <13508071+jpv-costa@users.noreply.github.com> Co-authored-by: Shalom <shalom@ovalix.io> Co-authored-by: codgician <15964984+codgician@users.noreply.github.com> Co-authored-by: FugoP <kim@pomsora.com> Co-authored-by: AgentGymLeader <264910004+AgentGymLeader@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
ryan-crabbe-berri
|
2d576b5695
|
feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes (#30261)
* feat(ui): cut mcp-servers, search-tools, tag-management, vector-stores, and memory over to path routes Continues the page-by-page App Router migration. All five legacy switch arms passed only accessToken/userRole/userID, so each route wrapper is a thin useAuthorized() + render. MIGRATED_PAGES routes the sidebar and redirects the legacy ?page= URLs; the e2e fixture picks all five up in the migration smoke and sidebar specs automatically. * refactor(ui): colocate MemoryView under the memory route The legacy switch was its only importer, so the three files move wholesale into (dashboard)/memory/components. mcp_tools, SearchTools, tag_management, and vector_store_management stay at src/components for now: each has importers on other pages (teams, playground, guardrails), so their colocation needs a shared/page split as a follow-up. |
||
|
ryan-crabbe-berri
|
3ad385a8a4
|
feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes (#30236)
* feat(ui): cut budgets, workflows, and guardrails-monitor over to path routes Continues the page-by-page App Router migration (#30185, #30226). All three legacy switch arms passed only accessToken, so each route wrapper is a thin useAuthorized() + render. MIGRATED_PAGES routes the sidebar and redirects the legacy ?page= URLs; the e2e fixture picks all three up in the migration smoke and sidebar specs automatically. * refactor(ui): colocate budgets, workflows, and guardrails-monitor components budgets and workflow_runs were imported only by the legacy switch, so they move wholesale into their route folders; the budgetItem type hoists into the shared useBudgets hook, which owns the API response shape, so the hooks layer no longer imports from a page folder. GuardrailsMonitor keeps LogViewer, mockData, and MetricCard at the shared src/components home because ToolDetail and ToolPolicies import them; the rest moves. eslint suppressions are re-keyed accordingly. * fix(ui): restore MetricCard test-utils path and merge duplicate import MetricCard.test.tsx got the moved-tree depth rewrite before being moved back to src/components/GuardrailsMonitor, leaving a five-level path that escapes the project root; the suite failed at import. Also merge the two imports from useBudgets in budget_panel.tsx. Both flagged by Greptile. |
||
|
ryan-crabbe-berri
|
a2c916fb45
|
feat(ui): migrate projects and access-groups to path routes (#30226)
* feat(ui): cut projects and access-groups over to path routes Same recipe as playground (#30185): MIGRATED_PAGES entries route the sidebar and redirect the legacy ?page= URLs, the switch arms are deleted, and the e2e fixture grows two entries. Both components were already zero-prop and self-fetching via React Query hooks, so the route wrappers are trivial. * refactor(ui): move Projects and AccessGroups components into their route folders Both folders were imported only by the legacy switch, so they colocate wholesale under (dashboard)/{projects,access-groups}/components. Their React Query hooks stay in the shared (dashboard)/hooks layer. eslint suppressions are re-keyed to the new paths. * test(ui): enable enable_projects_ui in e2e global setup The projects migration smoke clicks the Projects sidebar link, which only renders when the enterprise-gated enable_projects_ui setting is on; the seeded e2e database starts with it off, so the locator timed out in both e2e_ui_testing jobs. CI already launches the proxy with LITELLM_LICENSE for premium UI coverage, so flip the setting in globalSetup via the same /update/ui_settings call the admin UI toggle makes, failing loudly if the PATCH is rejected. * test(ui): use Playwright request context instead of raw fetch in global setup The frontend lint bans raw fetch() outside src/lib/http/; the e2e convention for proxy API calls is Playwright's APIRequestContext, as in routerSettings.spec.ts. |
||
|
ryan-crabbe-berri
|
530c0b2326
|
feat(ui): migrate playground to path routing and colocate its files (#30185)
* feat(ui): cut playground over to the /ui/playground path route
Follows the api-reference recipe: the sidebar and deep links route
llm-playground to the path route, ?page=llm-playground redirects, and
the legacy switch arm is deleted. The route's page.tsx was already the
real implementation, so no view extraction was needed.
* refactor(ui): move playground-owned files into its route folder
Per the (dashboard) README convention, page-owned code lives in the
page's folder: chat_ui/compareUI/complianceUI components, the chat
hooks, and the playground-only llm_calls helpers move under
(dashboard)/playground/. Modules with non-playground consumers (chat
message primitives; fetch_models, chat_completion, responses_api) stay
at their lowest common ancestor in src/components/{chat_ui,llm_calls}
because legacy pages still import them. eslint-suppressions entries are
re-keyed to the new paths so the grandfathered baseline still applies.
* test(ui): teach sidebar e2e spec about migrated path routes
The sidebar spec asserted ?page=<key> for every item, which the
playground cutover correctly broke: the sidebar now links to
/ui/playground and the legacy URL redirects there. Drive the expected
URL from the migration fixture (now a page-id -> segment map) so
future cutovers only add a fixture entry. Also wrap one import line
in AgentBuilderView.tsx that the move left unformatted; the changed-
files prettier check flagged it.
|
||
|
Yassin Kortam
|
a992ed18df
|
feat(spend_logs): opt-in native Postgres partitioning for SpendLogs retention (#29466)
High-volume deployments see LiteLLM_SpendLogs grow unbounded because retention via DELETE leaves dead tuples that autovacuum cannot reclaim fast enough. With a range-partitioned table, retention drops whole partitions instead: an instant metadata operation that returns disk to the OS immediately. The feature is gated behind general_settings.use_spend_logs_partitioning (default false). With the flag off, the cleanup job never queries the catalog and behaves exactly as today. With it on, the job verifies the table is partitioned, pre-creates upcoming partitions, and drops expired ones; expired rows the drops cannot reach (DEFAULT partition, partitions spanning the cutoff) are still deleted row-wise so retention is never bypassed. If the table is not partitioned it falls back to batched DELETE only. Converting an existing table is a manual, documented operation in db_scripts/partition_spend_logs.sql; db_scripts/unpartition_spend_logs.sql rolls it back. Both scripts rename the old table's indexes aside before recreating them, since a table rename keeps the schema-unique index names and would otherwise silently skip the CREATE INDEX IF NOT EXISTS block. Granularity and pre-create lookahead are tunable via SPEND_LOG_PARTITION_INTERVAL (day/week/month, invalid values fall back to day) and SPEND_LOG_PARTITION_PRECREATE_AHEAD. |
||
|
Mateo Wang
|
49ca04d8c3
|
feat(bedrock): aws_bedrock_project_id for bedrock-mantle project / workspace association (#30163)
* feat(bedrock): support aws_bedrock_project_id for bedrock-mantle project association Adds a litellm_params field to associate bedrock-mantle requests with an Amazon Bedrock project, sent as the OpenAI-Project header on the OpenAI-compatible chat and responses paths and as the anthropic-workspace header on the Anthropic messages paths. This lets a single model entry opt into a project-scoped data retention mode (e.g. provider_data_share for Claude Fable 5) while the account-wide setting stays on default. The param is carried via litellm_params only and is explicitly excluded from optional_params so it can never leak into a request body. Fixes #30070 * chore(ui): regenerate schema.d.ts for aws_bedrock_project_id Generated with npm run gen:api after adding the field to LiteLLM_Params * fix(proxy): ban client-supplied aws_bedrock_project_id in request bodies The deployment pins aws_bedrock_project_id so the project's data retention policy applies to its requests. Without this guard an authenticated caller could supply the field in the request body and, since client kwargs win the router merge, run requests under any project reachable with the deployment's shared AWS credentials. Adds the field to _BANNED_REQUEST_BODY_PARAMS so it is rejected at the auth boundary by default while remaining available through the existing admin opt-ins (allow_client_side_credentials proxy-wide or configurable_clientside_auth_params per deployment). |
||
|
ryan-crabbe-berri
|
4def6916da
|
refactor(ui): consolidate dashboard to one shell in the (dashboard) layout (#30166)
* refactor(ui): consolidate dashboard to one shell in the (dashboard) layout Moves the legacy ?page= switch page into the (dashboard) route group and hoists Navbar, sidebar, ThemeProvider, and DebugWarningBanner into the shared layout with real props, deleting the degraded duplicate shell that wrapped migrated routes. The active page key now derives from the URL at render time, so navigating between legacy and migrated pages no longer remounts the shell. useProxySettings becomes a React Query hook taking accessToken, shared by the navbar, the AdminPanel arm, and migrated pages; this replaces the lifted proxySettings state and the Navbar setProxySettings prop drilling. The invitation onboarding flow (?invitation_id=) keeps rendering without chrome via a layout escape hatch. Dead dark mode state and the no-op antd ConfigProvider are removed. * fix(ui): include accessToken in useProxySettings query key The queryFn closes over accessToken, so the key must include it for the cache to be honest about its inputs. Settings are instance-global today, which made the omission harmless, but a token change while mounted would have served the cached entry without refetching. * test(ui): point CreateKeyPage test at the moved page The page moved into the (dashboard) route group and no longer renders the navbar (the layout owns chrome now), so the valid-token test asserts the default page content (UserDashboard stub) instead. |
||
|
ryan-crabbe-berri
|
496f5b9859
|
fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui (#30169)
* fix(ui): serve migrated-page links unprefixed on the dev server migratedHref and legacyPageHref always prepended /ui, which is where the proxy mounts the static export but not where next dev serves the app (basePath is empty; the app lives at the root on localhost:3000). Every sidebar link to a migrated page and every ?page= bookmark redirect therefore 404'd in dev, and would do so for each page cut over in the App Router migration. uiBase now returns the bare root under NODE_ENV=development. The check is inlined at build time, so production output is unchanged for both the default /ui mount and server_root_path deployments. * test(ui): pin NODE_ENV in production-mode migratedPages tests The production-mode describes relied on vitest defaulting NODE_ENV to test; a developer with NODE_ENV=development exported in their shell would see them fail. Stub it explicitly so the suite is deterministic regardless of ambient environment. |
||
|
Yassin Kortam
|
dff25fef44
|
feat(proxy): add option to disable server-side prepared statements for DB lookups (#29984) | ||
|
ryan-crabbe-berri
|
a75ed0079c
|
chore(ui): make knip recognize .mjs scripts and openapi-typescript (#30052)
The knip entry/project globs only matched scripts/**/*.ts, so the two .mjs scripts went unanalyzed and produced "no matches" config hints. openapi-typescript was also reported as unused because gen-api-types.mjs invokes its binary through a dynamic execFileSync path that knip cannot trace statically; ignoreDependencies records that it is genuinely used. |
||
|
Sameer Kankute
|
3b40ac987f
|
Litellm oss 090626 (#30021)
* fix(mcp): report scoped server name during initialize (#29865) * fix mcp scoped server name * Update litellm/proxy/_experimental/mcp_server/mcp_context.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * test(mcp): cover scoped server name in the SSE initialize handler --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): show all session logs in the drawer, not just the first 50 (#29795) * fix(ui): show newest session logs first * test(ui): keep session log pagination coverage * fix(ui): show all session logs in the drawer, not just the first page The session detail drawer fetched session logs via sessionSpendLogsCall without page/page_size, so it only ever received the backend default of one page (50 rows). Sessions with more than 50 calls had the rest unreachable in the UI (#29153). sessionSpendLogsCall now takes page/page_size, and the drawer fetches the first page, reads total_pages, then fetches the remaining pages and accumulates them before the existing client-side sort. This keeps the single continuous list (and the selected-log lookup and keyboard navigation, which all assume the full session) correct. Fetching is bounded by a page cap, and the sidebar shows a "showing most recent N" note if a session exceeds it. The rows are lightweight metadata (the endpoint excludes messages/response), so the full set is small; request/response bodies are still loaded per log on demand. * fix(ui): default session drawer to most recent log, newest first Open a session with its most recent log selected, and order the sidebar newest-first to match the all-sessions logs overview. MCP calls stay grouped last. The latest log by time is computed explicitly, since the MCP grouping means it is not always the first row. * Apply fetching pages in batches suggestion from @greptile-apps[bot] Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(ui): derive session total from accumulated rows when backend omits it Compute the session total after all pages are fetched, falling back to the accumulated row count rather than the first page's. Guards the truncation note against a backend response that omits total but spans multiple pages. --------- Co-authored-by: Yufeng He <40085740+he-yufeng@users.noreply.github.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy): handle Mistral multipart passthrough (#29927) * fix(proxy): handle Mistral multipart passthrough * chore: satisfy passthrough ci formatting * test(proxy): cover Mistral passthrough in CI shard * fix(vertex_ai): use REP host for context caching on eu/us multi-region endpoints (#29573) Context caching built the cachedContents URL as https://{location}-aiplatform.googleapis.com, which is an invalid host for the eu/us multi-region endpoints and returns 404. The inference path already resolves these to the REP host (https://aiplatform.{geo}.rep.googleapis.com) via get_vertex_base_url(); reuse that helper in _get_token_and_url_context_caching so caching uses the same host as inference. Adds tests covering the eu/us multi-region cachedContents URLs (v1 and v1beta1). Fixes #29571 * Support per-model encrypted content affinity config (#29760) Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix: propagate upstream status code in proxy API exception handler (#29402) * fix: propagate upstream status code in proxy API exception handler When Google GenAI / Vertex returns a 404 for deprecated or missing models via streamGenerateContent, the exception was falling through to a generic handler that defaulted to 500. Now provider exceptions carrying a valid HTTP status_code correctly propagate it through to the ProxyException. * fix: apply black formatting to common_request_processing.py * fix: tighten status code range to 400-599 and deduplicate ProxyException raise * fix(tests): use valid vertex_location in context caching tests Replace "test_location" (contains underscore) with "us-central1" so tests pass the regex validation added in get_vertex_base_url(). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sdk): add xAI OAuth provider (#29866) * Add xAI OAuth provider * Update oauth.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * Fix xAI OAuth CI failures * Add xAI OAuth coverage tests * Move xAI OAuth coverage tests to core utils * Address xAI OAuth review comments * Prevent xAI OAuth api_base token exfiltration * Treat blank xAI OAuth api keys as absent * Wrap invalid xAI OAuth JSON responses * Use xAI OAuth behind explicit flag --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix(proxy) #27734 allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update (#27751) * fix(proxy): allow clearing budget_duration and team_member fields by sending null on /key/update and /team/update Fixes #27734 Sending null for budget_duration, team_member_budget, team_member_budget_duration, team_member_rpm_limit, or team_member_tpm_limit via /key/update or /team/update returned 200 OK but silently ignored the null value. The fields remained unchanged in the database. Root causes: - /key/update: prepare_key_update_data() popped budget_duration from the update dict but never re-added it (or budget_reset_at) when the value was None. - /team/update: _set_budget_reset_at() only acted when budget_duration was non-None, leaving a stale budget_reset_at in the DB. - /team/update: team_member_* null values bypassed the budget table update entirely because should_create_budget() requires at least one non-None field. * test(proxy): cover no-budget-row path in clear_team_member_budget_fields * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes (#30028) * fix(presidio): unmask PII tokens in Anthropic native SSE streaming bytes When output_parse_pii=true on the Anthropic native path (anthropic/claude-*), response chunks arrive as raw bytes in SSE format. _stream_pii_unmasking was yielding those bytes unchanged, so <PERSON_1> tokens were never replaced with the original values before reaching the caller. Add _unmask_sse_bytes_chunk to parse each data: line, find content_block_delta / text_delta events, and apply _unmask_pii_text before re-encoding. Wire it into _stream_pii_unmasking so bytes chunks are unmasked when pii_tokens exist. * fix(presidio): handle CRLF line endings and non-ASCII PII in SSE unmask Strip trailing \r before the [DONE] guard so CRLF-terminated SSE chunks don't bypass it and silently swallow a JSONDecodeError. Add ensure_ascii=False to json.dumps so non-ASCII replacement values like accented names are preserved as UTF-8 on the wire rather than being \uXXXX-escaped. Add regression tests for both cases. * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) (#29925) * feat(bedrock_mantle): path-aware Responses routing (/v1/responses vs /openai/v1/responses) Bedrock Mantle serves the Responses API on two upstream paths: - gpt frontier models (gpt-5.5 / gpt-5.4) on /openai/v1/responses - every other Responses-capable model (e.g. gpt-oss) on the standard /v1/responses BedrockMantleResponsesAPIConfig gains a `use_openai_path` flag; the provider gate in utils.py picks the path per model: openai.gpt-* (non gpt-oss) -> /openai/v1/responses; any model declared mode=responses (price-map entry or user model_info) -> /v1/responses; everything else returns None and keeps the existing chat-completions emulation. Adds gpt-5.5 / gpt-5.4 price-map entries, registry wiring, and the routing-matrix tests. * feat(bedrock_mantle): data-driven frontier routing via use_openai_responses_path Addresses the Greptile review point that frontier detection should be a price-map field rather than a hardcoded name match. The gate now routes a model to /openai/v1/responses when its price-map entry declares use_openai_responses_path, so a frontier model whose name does not follow the openai.gpt- convention can be onboarded by JSON alone. The name-convention check is kept as a fallback that needs no price-map entry, which preserves zero-change routing for a future gpt-6 before its entry loads. gpt-5.5 / gpt-5.4 get the flag in both price maps. Adds tests for the data-driven flag path and for the flag presence on the gpt-5.x entries; both branches are mutation-tested. * test(model_prices): allow use_openai_responses_path in price-map schema The model_prices_and_context_window.json schema validator (test_aaamodel_prices_and_context_window_json_is_valid) enforces additionalProperties: false, so the new use_openai_responses_path flag on the gpt-5.5 / gpt-5.4 entries failed validation. Add it to the schema as a boolean, alongside the other supports_* / capability flags. * Add Tensormesh serverless models to the model cost map (#30037) * Add Tensormesh serverless models to the model cost map * Flag reasoning support on the Tensormesh models that expose thinking mode * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update (#30001) * fix(proxy): reconcile stale key spend counter after budget reset * fix(proxy): invalidate stale key spend counter after budget reset or manual spend update * fix(proxy): remove read-time stale counter reconciliation to prevent budget bypass * revert: undo unrelated formatting changes in enterprise directory * test(proxy): add unit test for key spend update invalidating counter * test(proxy): fix mocked update_data and hash token expectations in unit test * fix(proxy): use Responses-API transformer in pass-through cost tracking (#29728) The `elif is_responses:` branch of `openai_passthrough_handler` was calling the chat-completions `transform_response` on a Responses API payload. The chat-completions transformer expects `choices: [...]` in the raw response; the Responses API uses `output: [...]` and `usage.input_tokens` / `usage.output_tokens` (not `prompt_tokens` / `completion_tokens`). The result was a KeyError 'choices' deep inside `convert_to_model_response_object`, swallowed by the surrounding `except Exception` in the handler, and the SpendLogs row was written by the fallback path with zeroed-out tokens, spend, and model. This bug silently undercounts cost for every successful pass-through call to either OpenAI's `/v1/responses` or Azure's `/openai/v1/responses` (deployments configured for the Responses API). Reproduced 2026-06-04 against a real Azure OpenAI Responses API deployment proxied through LiteLLM v1.88.0. Fix: use the dedicated `OpenAIResponsesAPIConfig.transform_response_api_response` for the Responses branch. This transformer already exists in LiteLLM (`litellm/llms/openai/responses/transformation.py`) and knows the Responses-API on-the-wire shape. `litellm.completion_cost` already handles `ResponsesAPIResponse` natively with `call_type="responses"`, so no downstream changes are needed. Tests: test_responses_api_uses_responses_transformer_not_chat_completions NEW. Real regression test — exercises the openai_passthrough_handler with a real-shaped Responses payload (no `choices`, has `output` and Responses-API `usage` keys) and NO mocked `get_provider_config`. Pre-fix: raises KeyError 'choices' inside the chat-completions transformer (the bug). Post-fix: returns a ResponsesAPIResponse, completion_cost is called with call_type="responses" and a ResponsesAPIResponse instance (asserted). Verified to fail on un-fixed handler + pass on fixed handler before commit. test_responses_api_cost_tracking UPDATED. Old test mocked `get_provider_config` (no longer called in the responses branch post-fix). Now mocks the Responses transformer directly (`OpenAIResponsesAPIConfig.transform_response_api_response`) to test the downstream cost-calc contract. Out of scope for this PR (separate followup): - Recognizing *.cognitiveservices.azure.com (the newer Azure OpenAI hostname) in the is_openai_*_route checks. Separate PR. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(skills): execute DB skills by matching the litellm_skill_ tool name prefix (#30116) Skill IDs are generated as litellm_skill_<uuid> and the model-facing tool name is the sanitized skill ID, but the post-call execution gates in SkillsInjectionHook only ran tools whose name starts with "skill_", so DB skills were silently returned to the client as raw tool calls. Fixes #28122. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(anthropic): synthesize content_block_start when Responses stream omits output_item.added (#30115) * fix(team): reserve team budget raises for proxy admins on /team/update (#30030) The caller's PERSONAL max_budget was the wrong yardstick for /team/update: a team's spend ceiling has nothing to do with the admin's own key budget. That comparison was an unintended side effect of reusing _check_user_team_limits() (which exists for the /team/new path) and broke the UI, which re-sends the unchanged budget on every save. New behavior on /team/update for standalone teams: - A team admin (already authorized via _verify_team_access) may freely KEEP or LOWER the team budget, and change models/tpm/rpm, without being gated by their personal limits. - GROWING a team's spend ceiling is a budget-authority action reserved for proxy admins -> 403 for team admins. "Growing" covers both raising max_budget above the team's current finite value and removing the cap entirely (max_budget=null, detected via model_fields_set so an explicit null is distinguished from an omitted field). For a team that currently has no cap, setting a finite value is a restriction and is allowed. - Org-scoped teams remain governed by _check_org_team_limits() (capped by the org budget). Also reverts the #29525 existing_team_max_budget workaround in _check_user_team_limits() back to the create-only form; /team/new still enforces the creator's personal caps. docs(access_control): resolve the contradiction in the team-admin section — team admins can keep/lower the budget and manage rate limits/models, but cannot raise the team budget (proxy-admin only). tests: unit + behavior coverage for raise-blocked, cap-removal-blocked (team admin), raise/removal allowed (proxy admin), uncapped-team restriction allowed, keep/lower/resend allowed, and unchanged create-path guards. Co-authored-by: Cursor <cursoragent@cursor.com> * test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974) * test(ui): add a data-driven App Router migration E2E smoke Add a growing Playwright smoke for migrated pages: for each segment it deep-links to the path route, asserts the URL and that the dashboard shell rendered, then clicks off to a legacy page and asserts navigation still works. Driven by e2e_tests/fixtures/migratedPages.ts, so adding a page is one line. Runs in two situations against the same proxy: the default mount (npm run e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root). globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage state is valid under a prefix. Seeded with api-reference; append the rest as their migrations merge. * test(ui): support headed slow-motion + watch pauses in the migration smoke Honor SLOWMO in the server-root-path config (the default config already did), and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state. Both are no-ops by default, so CI behavior is unchanged. * test(ui): make the migration smoke a sidebar-click user journey Rework the smoke from deep-linking to a real navigation journey: start at the landing page, click the migrated page in the sidebar (expanding submenus for nested items), assert the path route rendered, reload it (the check a wrong server_root_path breaks), bounce to a legacy page and back, and — once two pages are migrated — navigate directly between two migrated pages. Verifies via URL + shell render, driven by the same fixture list. * test(ui): address review on the migration smoke Escape ROOT and segment before interpolating them into RegExp URL matchers so a future segment containing regex metacharacters can't silently widen the match. Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead of silently re-running the default mount and passing without exercising the prefix. * test(ui): drop unused watch helper and fix stale smoke README * test(ui): run the migration smoke under a server root path in CI * test(ui): harden + instrument the server-root-path proxy reboot in CI * test(ui): run the server-root-path migration smoke as its own CI job Replace the in-place proxy reboot in e2e_ui_testing with a dedicated e2e_ui_testing_server_root_path job that boots the proxy once with SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the config gets its own job rather than killing and relaunching the live proxy. The reboot was failing deterministically: after pkill -9 and relaunch the prefixed proxy never came back up on :4000 (connection refused), so the smoke never ran. The readiness step that was supposed to surface the cause could never reach its boot-log tail because CircleCI runs steps under bash -eo pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's exit 7. Booting the proxy as the job's own background step lets any boot crash land in that step's log instead of being swallowed. The default e2e_ui_testing job is unchanged aside from dropping the reboot, prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at the root mount there via the default Playwright config. * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through (#24232) * fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through * test: mock post_call_response_headers_hook in audio speech route tests * chore(ui): remove dead App Router route stubs under (dashboard) (#30045) models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them * fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000) * fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss * fix(mcp): allow team access-group grants in OAuth authorize/token access check (#30041) * fix(mcp): honor team access-group grants in OAuth authorize/token access check * test(mcp): mock build_effective_auth_contexts in non-admin authorize tests for isolation * docs(security): require a reproduction video for vulnerability reports (#30048) (#30063) With AI models capable of automated vulnerability discovery now publicly available, we expect a large increase in report volume, much of it unverified. Requiring a video of the exploit running against a live instance raises the bar for submissions and keeps triage focused on reproducible issues. Reports without a video will be closed and reopened if one is added later. Co-authored-by: stuxf <70670632+stuxf@users.noreply.github.com> * feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796) * feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. * chore(ui): remove dead dashboard files and unused dependencies (#30047) * chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) (#30009) * fix(proxy): authorize batch files using upload target_model_names (LIT-3593) After replace_model_in_jsonl, body.model is a stripped provider id. Reverse-mapping it via resolve_model_name_from_model_id is first-match on model_list and caused false 403s when multiple deployments share the same stripped name. Use target_model_names from the unified file id instead. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593) Restores the reverse-lookup for the JSONL body.model fallback path so that legacy/pre-target_model_names managed files still map stripped provider IDs back to proxy aliases before auth. Also cleans up redundant `or None`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Revert "fix(proxy): restore resolve_model_name_from_model_id for JSONL fallback path (LIT-3593)" This reverts commit |
||
|
ryan-crabbe-berri
|
9e0d92c129
|
chore(ui): remove dead dashboard files and unused dependencies (#30047)
* chore(ui): remove dead dashboard files and unused dependencies knip flagged seven orphaned source/config files with no importers and five declared dependencies that nothing in the tree uses. Removing them shrinks the dashboard bundle's source surface and keeps the manifest honest; vite stays installed transitively via vitest, so test tooling is unaffected. * fix(ci): restore serverRootPath.config.ts referenced by SERVER_ROOT_PATH workflow The dead-code sweep removed e2e_tests/serverRootPath.config.ts, but its spec (tests/login/serverRootPathRedirect.spec.ts) and the test_server_root_path.yml workflow step still depend on it, so the redirect e2e job failed to load a config that no longer existed. |
||
|
ryan-crabbe-berri
|
248176112e
|
feat(ui): add admin flag to disable in-product UI nudges for everyone (#29796)
* feat(ui): add admin flag to disable in-product UI nudges for everyone Admins can now suppress the survey and Claude Code feedback popups for all users via a single disable_ui_nudges UI setting, instead of relying on each user dismissing them individually. * fix(ui): suppress nudges while ui settings are loading Gate nudgesDisabled on the ui-settings loading state so an admin with disable_ui_nudges on doesn't see the survey prompt flash, and the getInProductNudgesCall fetch doesn't fire, on a cold page load before the flag resolves. Falls back to showing nudges if the fetch errors. * test(ui): wrap CreateKeyPage test in QueryClientProvider page.tsx now calls useUISettings (react-query), which needs a QueryClient that layout.tsx supplies in production but the test did not. Add the provider and mock getUiSettings so the query resolves. |
||
|
tin-berri
|
d8fe091938
|
fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session (#30000)
* fix(ui/mcp): reset OAuth hook state on modal close so a prior server's token no longer leaks into the next add-server session * fix(ui/mcp): clear in-flight OAuth guard on reset and reset form/tools on modal close so nothing leaks on a parent-driven dismiss |
||
|
ryan-crabbe-berri
|
38edf241a4
|
chore(ui): remove dead App Router route stubs under (dashboard) (#30045)
models-and-endpoints, organizations, and virtual-keys each had a page.tsx route under (dashboard)/ that is not in MIGRATED_PAGES, so the sidebar and deep links never resolve to it and the route is unreachable. Each was a thin wrapper that handed the shared view empty or no-op props (empty modelData with a no-op setModelData, hardcoded empty organizations, no-op setUserRole/setUserEmail), so reaching one would render a degraded page in any case. The real wrapper belongs in the PR that flips each page into MIGRATED_PAGES, written with eyes on it and a test This continues the dead-scaffolding cleanup from #28891. The shared components these wrappers rendered (ModelsAndEndpointsView, OrganizationFilters) stay, since the legacy ?page= switch in app/page.tsx and src/components still import them |
||
|
ryan-crabbe-berri
|
6ae8a509f0
|
test(ui): data-driven App Router migration E2E smoke (default + server-root-path) (#29974)
* test(ui): add a data-driven App Router migration E2E smoke
Add a growing Playwright smoke for migrated pages: for each segment it deep-links
to the path route, asserts the URL and that the dashboard shell rendered, then
clicks off to a legacy page and asserts navigation still works. Driven by
e2e_tests/fixtures/migratedPages.ts, so adding a page is one line.
Runs in two situations against the same proxy: the default mount (npm run
e2e:migration) and a non-root SERVER_ROOT_PATH mount (npm run e2e:migration:root).
globalSetup now logs in at `${SERVER_ROOT_PATH}/ui/login` so the admin storage
state is valid under a prefix. Seeded with api-reference; append the rest as their
migrations merge.
* test(ui): support headed slow-motion + watch pauses in the migration smoke
Honor SLOWMO in the server-root-path config (the default config already did),
and add an env-gated E2E_WATCH_MS pause so a headed run lingers on each state.
Both are no-ops by default, so CI behavior is unchanged.
* test(ui): make the migration smoke a sidebar-click user journey
Rework the smoke from deep-linking to a real navigation journey: start at the
landing page, click the migrated page in the sidebar (expanding submenus for
nested items), assert the path route rendered, reload it (the check a wrong
server_root_path breaks), bounce to a legacy page and back, and — once two pages
are migrated — navigate directly between two migrated pages. Verifies via URL +
shell render, driven by the same fixture list.
* test(ui): address review on the migration smoke
Escape ROOT and segment before interpolating them into RegExp URL matchers so a
future segment containing regex metacharacters can't silently widen the match.
Make the server-root-path config fail fast when SERVER_ROOT_PATH is unset instead
of silently re-running the default mount and passing without exercising the prefix.
* test(ui): drop unused watch helper and fix stale smoke README
* test(ui): run the migration smoke under a server root path in CI
* test(ui): harden + instrument the server-root-path proxy reboot in CI
* test(ui): run the server-root-path migration smoke as its own CI job
Replace the in-place proxy reboot in e2e_ui_testing with a dedicated
e2e_ui_testing_server_root_path job that boots the proxy once with
SERVER_ROOT_PATH=/litellm, matching how every other proxy variant in the
config gets its own job rather than killing and relaunching the live proxy.
The reboot was failing deterministically: after pkill -9 and relaunch the
prefixed proxy never came back up on :4000 (connection refused), so the smoke
never ran. The readiness step that was supposed to surface the cause could
never reach its boot-log tail because CircleCI runs steps under bash -eo
pipefail and the preceding `curl -sv ... | tail` aborted the step with curl's
exit 7. Booting the proxy as the job's own background step lets any boot crash
land in that step's log instead of being swallowed.
The default e2e_ui_testing job is unchanged aside from dropping the reboot,
prefixed-readiness, and prefixed-smoke steps; the migration smoke still runs at
the root mount there via the default Playwright config.
|
||
|
tin-berri
|
51ba6e39cd
|
fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path (#29960)
* fix(ui): load MCP tool configuration tools via the OBO/passthrough-aware GET path * fix(mcp): admin-only include_disabled_tools so the settings UI shows toggled-off tools * fix(ui): repopulate MCP server edit form when server data loads after mount (OAuth return) * fix(ui): persist MCP OAuth token on save and return to the Settings tab after authorize * fix(ui): scope MCP OAuth callback to the initiating form so create and edit flows don't cross-talk * fix(ui): derive OAuth-return Settings tab via lazy state init instead of setState-in-effect * Fix MCP OAuth edit token handling --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> |
||
|
yuneng-jiang
|
bac2590b39
|
build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 (#29982)
Raise the PyJWT floor in pyproject (>=2.13.0,<3.0) and re-resolve uv.lock so the proxy installs 2.13.0 instead of 2.12.0. Bump the ws transitive-version override in the dashboard from 8.19.0 to 8.20.1 and regenerate package-lock; jsdom and openai both dedupe onto the single 8.20.1 copy. Both are routine dependency maintenance bumps to keep pinned versions current. |
||
|
milan-berri
|
f59e4ebc9e
|
fix(ui): show team projects to internal users (#28855)
Allow internal users to fetch their backend-scoped project list so the key creation project dropdown can populate for selected teams. |
||
|
Sameer Kankute
|
dfb68a23de
|
feat(galileo): add health check support for UI callback test (#29908)
* feat(galileo): add health check support for UI callback test Register galileo in /health/services so the proxy UI callback connection test works. Co-authored-by: Cursor <cursoragent@cursor.com> * feat(galileo): verify API key via /current_user health check Call Galileo's current_user endpoint so the UI callback test validates credentials against the provider. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ui): regenerate schema.d.ts for galileo health service Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): return IntegrationHealthCheckStatus from async_health_check Fixes mypy assignment error in health_services_endpoint where response was narrowed to IntegrationHealthCheckStatus from earlier branches. Co-authored-by: Cursor <cursoragent@cursor.com> * Fix Galileo logging to match Langfuse across all endpoint types. Stop skipping ingest when output is empty and log embeddings with a placeholder so embedding, speech, and other non-text responses are recorded like Langfuse. Co-authored-by: Cursor <cursoragent@cursor.com> * fix(galileo): remove unreachable health-check guard and None output sentinel The use_v2_api flag is derived from bool(api_key), so the inner GALILEO_API_KEY check inside the v2 branch could never run; collapse the credential validation into the username/password path with a combined message. _serialize_galileo_output now returns an empty string for None, so _get_galileo_input_output_content always yields a str and the post-call None coalescing guard is no longer needed. * test(galileo): cover async_health_check failure paths and empty model response Add regression tests for the Galileo health check unhealthy branches (missing project id, missing base url, missing credentials, auth failure, and request exception) and for logging a model response with no choices, which now queues an empty output instead of being skipped. --------- Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> |
||
|
Sameer Kankute
|
32c88ca74f
|
Litellm oss staging 080626 (#29932)
* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes #29665) (#29788) * feat(responses): add default no-op sign_request to BaseResponsesAPIConfig * feat(responses): call sign_request after body is final, send signed bytes when signed * feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path) * test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency * feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer * docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route * test(responses): cover fake-stream signing order and mantle bearer arg/env precedence * fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance * fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked * fix(bedrock_mantle): sign the compact Responses route too, not just create * fix(github-copilot): route per-model on /v1/responses based on model info (#29747) * feat(focus): add GCS destination for FOCUS export (#29751) * test: add failing tests for FocusGCSDestination * feat: add FocusGCSDestination reusing GCSBucketBase auth * feat: register FocusGCSDestination in factory; export from __init__ * fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config * style: apply Black formatting to gcs_destination and tests * style: apply Black formatting to factory.py * fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565) Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input request when additionalModelRequestFields: {} or system: [] are present in the payload. Both fields are optional in CommonRequestObject (total=False) and must be omitted rather than sent as empty structures. Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730) * fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible Azure OpenAI resources created via the newer "Azure AI Foundry" / Cognitive Services pathway live on `*.cognitiveservices.azure.com` subdomains, not the older `openai.azure.com`. Both are valid Azure OpenAI surfaces in production today. The OpenAI pass-through cost-tracking handler hard-codes only the older hostname in five places (four `is_openai_*_route` methods on OpenAIPassthroughLoggingHandler, plus is_openai_route on PassThroughEndpointLogging). As a result, calls from newer Azure deployments are silently classified as "not an OpenAI route", the dispatch into the cost-tracking handler is skipped, and tokens/cost never get extracted into LiteLLM_SpendLogs — the row gets written with prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'. Reproduced 2026-06-04 against a real Azure OpenAI deployment on `*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0. Fix: factor the hostname check into a single helper `_is_openai_compatible_host` listing all three recognized surfaces (api.openai.com, openai.azure.com, cognitiveservices.azure.com), and have all five call sites delegate to it. Purely additive — never weakens recognition for the originally-supported hostnames. Adds a test `test_is_openai_route_recognizes_cognitiveservices_azure_com` that exercises all four `is_openai_*_route` static methods against `*.cognitiveservices.azure.com` URLs (positive cases per route + a small cross-route negative to confirm route-specific path matching still works on the new hostname). Out of scope for this PR (separate followup): - `openai_passthrough_handler` calls chat/completions `transform_response` on Responses API payloads (`output:` not `choices:`), which throws inside the dispatch and drops the SpendLogs row entirely. Recognized + tracked separately. * ci: trigger fresh run Empty commit to re-run checks. The previous auth-and-jwt failure was a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to this PR's scope (hostname recognition in pass-through cost tracking). No code change. --------- Co-authored-by: shin-berri <shin-laptop@berri.ai> Co-authored-by: yuneng-jiang <yuneng@berri.ai> * fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812) The Responses API forces a specific function with a top-level name ({"type": "function", "name": "X"}), but _transform_tool_choice only handled the nested Chat Completions shape and fell through to returning "required" for the flat form, silently dropping the function name and degrading a forced function call to force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the "required" fallback when no name is present. * Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584) * Preserve x-anthropic-billing-header system blocks for first-party Anthropic PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for every Anthropic target. That block is how the first-party Anthropic API recognizes Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry only that block, such as the auto-mode tool-safety classifier, fail with a misleading 429 rate_limit_error; normal turns still work because they also carry the "You are Claude Code" identity block. Gate the strip behind should_strip_billing_metadata(), defaulting to False on the first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and overridden to True on the providers that reach these transforms and reject the block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek for the messages path). Behavior for those providers is unchanged. * Strip billing header on Bedrock invoke and Vertex messages pass-through Two more subclasses reach the gated strip but inherited keep-by-default. AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request, which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig (Vertex messages pass-through) calls super().transform_anthropic_messages_request. Override should_strip_billing_metadata() to True on both. Add a parametrized test asserting the flag for every first-party base (False) and provider subclass (True), covering all overrides, plus a translate_system_message regression test for the Bedrock invoke path. * fix(cache): log hashed cache keys (#29890) * fix(ui): save routing groups as list (#29889) * Revert "fix(ui): save routing groups as list (#29889)" (#29928) This reverts commit |
||
|
tin-berri
|
1528f43d4c
|
fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page (#29867)
* fix(ui): let non-creator users OAuth into OBO-mode MCP servers from the Tools page * fix(ui): clear OBO Tools-tab one-shot on navigate-back and gate on credential-status errors |
||
|
ryan-crabbe-berri
|
1afc41cb29
|
fix(ui): unify migrated-route URLs and migrate the API Reference page (#29953)
* fix(ui): unify migrated-route URLs and cut the API Reference page over to path routing Route all migrated-page navigation through one /ui-prefixed, serverRootPath-aware builder in migratedPages.ts (migratedHref/legacyPageHref/legacyKeyForPathname), replacing the three divergent base-URL constructions that lived in the dashboard layout's withBase, leftnav, and the page.tsx redirect. The previous migratedHref read NEXT_PUBLIC_BASE_URL, which no build sets, so it produced URLs without the /ui prefix the app is served under; every other internal link hardcodes /ui and this now matches that convention. Remove the sidebar's own pushState navigation so the parent (legacy root page or dashboard layout) is the single owner of navigation, fixing the double-navigate that fired when moving between a path route and a legacy ?page= route. Cut API Reference over to its path route: add api_ref -> api-reference to MIGRATED_PAGES and delete its arm from the legacy switch. Visiting /ui/?page=api_ref redirects to /ui/api-reference, the sidebar links to and highlights it, and navigating away returns to the legacy switch. * fix(ui): address review on migrated-page routing Keep the legacy hyphenated ?page=api-reference form working by mapping it to the api-reference route alongside api_ref; the old switch matched both, so a bookmark using the hyphen would otherwise fall through to the Usage default. Add legacyKeyForPathname coverage: a migrated path (with and without trailing slash) resolves to the api_ref sidebar key rather than the alias, a non-migrated path returns null, and a non-root serverRootPath prefix is stripped before matching. * fix(ui): populate serverRootPath from getUiConfig so migrated nav links keep the root path getUiConfig updated proxyBaseUrl but never called updateServerRootPath, so the module-level serverRootPath stayed at its "/" default. Under a custom server_root_path the unified migratedHref/legacyPageHref builders then dropped the prefix and the sidebar produced /ui/api-reference (404) instead of /<root>/ui/api-reference. Adds the missing updateServerRootPath call plus a regression test asserting getUiConfig sets serverRootPath and that migratedHref carries the prefix |
||
|
yuneng-jiang
|
728f057c5e
|
fix(ui): label default key type as "Full Access" on key edit page (#29870)
The key edit page showed the default key type (no allowed_routes restriction) as "Default", while the key creation form already labels the same value "Full Access". Align the edit page to the create form so the two surfaces agree on both the label and its description. |
||
|
yuneng-jiang
|
47b383dbbf
|
fix(ui): keep create guardrail modal open on outside click (#29871)
The create guardrail modal used antd's default maskClosable, so clicking
outside it dismissed the modal and reset every field the user had entered.
Setting maskClosable={false} keeps the modal open; it now closes only via
the explicit close button or Cancel, matching the other form modals in the
dashboard
|
||
|
yuneng-jiang
|
26fe26a5c0
|
fix(ui/model-hub): render provider icons on the public model hub (#29958)
The provider logo base path was relative ("../ui/assets/logos/"). With
trailingSlash enabled, the public model hub is served at /ui/model_hub_table/,
so the browser resolved the base to /ui/ui/assets/logos/ (a doubled /ui/), which
404s every icon. The authenticated hub renders inside the single-level /ui/ SPA
route where the relative path resolves correctly, so only the public hub broke.
Make the base root-absolute so it resolves at any route depth.
|
||
|
ryan-crabbe-berri
|
ff6cea4833
|
refactor(ui): single source of truth for migrated-page routing (#29949)
Consolidate the three hand-synced copies of the migrated-pages map (LEGACY_REDIRECTS in app/page.tsx, MIGRATED_PAGES in the dashboard layout, and MIGRATED_PAGES in leftnav) into one shared module, src/utils/migratedPages.ts, which also owns the migratedHref helper. Delete the unused, incomplete Sidebar2 prototype. No runtime behavior change: the map is still empty and Sidebar2 had no importers, so this is pure deduplication ahead of the per-page App Router migration. Follow-up work will unify the remaining base-URL builders (layout's withBase and page.tsx's redirect) onto migratedHref. |
||
|
Sameer Kankute
|
f5b11b72a6
|
feat(proxy): publish /v2/model/info in Swagger OpenAPI spec (#29900)
* feat(proxy): publish /v2/model/info in Swagger OpenAPI spec Expose the v2 model info endpoint in /docs by removing include_in_schema=False and documenting query parameters used by the admin UI and proxy CLI consumers. Co-authored-by: Cursor <cursoragent@cursor.com> * chore(ui): regenerate schema.d.ts for /v2/model/info OpenAPI docs Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |