mirror of
https://github.com/open-webui/open-webui.git
synced 2026-11-18 10:01:33 +00:00
Merge upstream/dev
Keep the per-worker model pool cache and take upstream's Redis pub/sub reconnect constants in the socket import.
This commit is contained in:
commit
eb798807a2
418 changed files with 114669 additions and 83531 deletions
|
|
@ -18,3 +18,6 @@ uploads
|
|||
**/*.db
|
||||
_test
|
||||
backend/data/*
|
||||
|
||||
.venv
|
||||
.git
|
||||
|
|
|
|||
|
|
@ -22,6 +22,9 @@ ENABLE_RAG_CSV_SUMMARY=false
|
|||
# Set to true to preserve backing file records, storage blobs, and per-file vectors when files are removed from knowledge bases.
|
||||
ENABLE_KNOWLEDGE_FILE_RETENTION=false
|
||||
|
||||
# Comma-separated chunk metadata keys to expose to the model alongside retrieved content.
|
||||
RAG_SOURCE_METADATA_KEYS=''
|
||||
|
||||
# Set to false to disable workspace Tools and Functions.
|
||||
ENABLE_PLUGINS=true
|
||||
|
||||
|
|
|
|||
6
.github/ISSUE_TEMPLATE/bug_report.yaml
vendored
6
.github/ISSUE_TEMPLATE/bug_report.yaml
vendored
|
|
@ -12,6 +12,8 @@ body:
|
|||
|
||||
Before submitting, search open and closed [Issues](https://github.com/open-webui/open-webui/issues) and [Discussions](https://github.com/open-webui/open-webui/discussions). The issue may already be reported or fixed on `dev`.
|
||||
|
||||
**Test on the latest release AND on `dev`, right before you submit this report, not last week.** A huge share of reports are for bugs already fixed on `dev`, sometimes weeks earlier, because the reporter only tested an old version and never rechecked. Reports that don't reproduce on latest and on current `dev` at submission time will be closed without further discussion, no exceptions.
|
||||
|
||||
Please do not open a code pull request for this report unless a maintainer asks for one, or the change is only i18n/localization. If you want to share code as reference, include it here as a local diff or patch. Actionable reproduction details are the most useful next step.
|
||||
|
||||
Security vulnerabilities must not be reported publicly. Use the [GitHub security page](https://github.com/open-webui/open-webui/security) instead.
|
||||
|
|
@ -23,12 +25,10 @@ body:
|
|||
options:
|
||||
- label: I searched open and closed issues and discussions for an existing report.
|
||||
required: true
|
||||
- label: I checked whether this is already fixed on the `dev` branch or latest source.
|
||||
- label: I reproduced this bug on the latest release AND on the current `dev` branch, right before submitting this report. I did not just check an old version or rely on a check from days ago.
|
||||
required: true
|
||||
- label: I understand that maintainers want a well-written issue before any code pull request.
|
||||
required: true
|
||||
- label: I am using the latest available version of Open WebUI for my install method.
|
||||
required: true
|
||||
- label: This is not a security vulnerability.
|
||||
required: true
|
||||
|
||||
|
|
|
|||
25
.github/pull_request_template.md
vendored
25
.github/pull_request_template.md
vendored
|
|
@ -1,31 +1,30 @@
|
|||
<!--
|
||||
Important checks for contributors:
|
||||
1. Target the `dev` branch. PRs targeting `main` will be closed.
|
||||
2. Code pull requests are not the default contribution path.
|
||||
3. Do not open a code PR as the first step. Start with a well-written Issue or Discussion unless a maintainer asked for the PR or the change is only i18n/localization.
|
||||
4. Do not delete the Contributor License Agreement section at the bottom. The CLA bot requires it.
|
||||
1. DO NOT OPEN A CODE PULL REQUEST unless a maintainer explicitly asked you to, or the change is strictly limited to i18n/localization.
|
||||
2. Target the `dev` branch. PRs targeting `main` will be closed.
|
||||
3. Do not delete the Contributor License Agreement section at the bottom. The CLA bot requires it.
|
||||
-->
|
||||
|
||||
# Pull Request
|
||||
|
||||
Thanks for wanting to improve Open WebUI. The most useful contribution is usually a clear, well-written Issue, not an unsolicited code pull request.
|
||||
**Do not open a code pull request unless a maintainer has explicitly requested it or the change is limited to i18n/localization.**
|
||||
|
||||
Open a code pull request only when a maintainer asks for one, or when the change is only i18n/localization. For real, reproducible bugs, start with a well-described [Issue](https://github.com/open-webui/open-webui/issues). For feature requests, UI/UX changes, behavior changes, architecture changes, suspected fixes, or unconfirmed approaches, start with an active [Discussion](https://github.com/open-webui/open-webui/discussions).
|
||||
The most useful way to help is to give us a clear understanding of the problem: report reproducible bugs in [Issues](https://github.com/open-webui/open-webui/issues) and share proposals in [Discussions](https://github.com/open-webui/open-webui/discussions). We use that context to evaluate solutions and refine the implementation internally, accounting for the broader codebase and ongoing work. External implementations usually require substantial reworking to fit the project's standards, and coordinating those revisions usually takes more effort than developing the solution internally. Please follow this process before investing time in a pull request. PRs opened outside these guidelines are generally closed without review.
|
||||
|
||||
Before continuing, make sure the linked Issue or Discussion explains the user-facing problem, the expected outcome, the affected workflow, and any examples, logs, screenshots, constraints, or reproduction details needed for maintainers to evaluate it.
|
||||
## Maintainer Request
|
||||
|
||||
If you have implementation notes, include them as reference in the Issue or Discussion. If you want to share code as reference, include it there as a local diff, patch, or branch note. Do not open a pull request for reference code.
|
||||
|
||||
Unsolicited PRs may be closed without review, especially when they introduce product, architecture, compatibility, dependency, or maintenance decisions that have not been discussed.
|
||||
Link the maintainer's request for this PR, or state that the change is limited to i18n/localization.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] I have read and I understand the [contribution policy](https://docs.openwebui.com/contributing/#submit-code).
|
||||
- [ ] This PR targets the `dev` branch.
|
||||
- [ ] This PR links to a well-described, confirmed Issue or active Discussion: `Closes #___` / `Relates to #___`.
|
||||
- [ ] A maintainer explicitly asked me to open this PR, or this PR only updates i18n/localization.
|
||||
- [ ] The change is one logical unit with no unrelated commits.
|
||||
- [ ] I matched nearby code patterns and avoided unnecessary new settings, abstractions, or dependencies.
|
||||
- [ ] I manually tested the changed workflow and any nearby behavior that could be affected.
|
||||
- [ ] I have not added or rewritten automated tests, fixtures, snapshots, or testing infrastructure unless a maintainer explicitly requested them.
|
||||
- [ ] I updated relevant docs, including the [Open WebUI Docs Repository](https://github.com/open-webui/docs), if needed.
|
||||
- [ ] I added screenshots for UI changes, and a recording when motion or interaction matters.
|
||||
- [ ] I reviewed any AI-generated code before submitting it.
|
||||
|
|
@ -50,9 +49,11 @@ Use one of the following prefixes:
|
|||
|
||||
Describe the change, the problem it solves, and the impact on users.
|
||||
|
||||
## Testing
|
||||
## Verification
|
||||
|
||||
List the exact manual checks you ran. Include commands, setup details, screenshots, or recordings where helpful.
|
||||
Describe how you reproduced the problem and manually checked the behavior before and after the change. Include exact steps, setup details, and relevant logs, screenshots, or recordings. Report results from relevant existing checks and anything you could not verify.
|
||||
|
||||
Do not add or rewrite automated tests unless a maintainer explicitly requests them. Tests that repeat an implementation's assumptions can pass while preserving the same mistake; maintainers determine the regression coverage needed. Do not remove, disable, or weaken existing tests to make the change pass.
|
||||
|
||||
## Changelog Entry
|
||||
|
||||
|
|
|
|||
45
.github/workflows/regression.yaml
vendored
Normal file
45
.github/workflows/regression.yaml
vendored
Normal file
|
|
@ -0,0 +1,45 @@
|
|||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Tests — run the open-webui/tests unit suite against a release candidate
|
||||
# Release pull requests go from dev into main and are titled with the version
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
name: Tests
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
branches: [main, dev]
|
||||
types: [opened, synchronize, reopened, edited]
|
||||
|
||||
# An edit must not cancel a running suite: the replacement run would skip it and still report green.
|
||||
concurrency:
|
||||
group: regression-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.event.action != 'edited' }}
|
||||
|
||||
jobs:
|
||||
# Into main only the release branch counts; into dev any version title does, so the
|
||||
# suite can be exercised outside a release. Expressions have no regex, hence the literal prefixes.
|
||||
regression:
|
||||
name: Suite
|
||||
if: >-
|
||||
(github.event.pull_request.base.ref == 'dev' ||
|
||||
github.event.pull_request.head.ref == 'dev') &&
|
||||
(github.event.action != 'edited' || github.event.changes.title != null) &&
|
||||
(startsWith(github.event.pull_request.title, '0.') ||
|
||||
startsWith(github.event.pull_request.title, '1.'))
|
||||
permissions:
|
||||
contents: read
|
||||
uses: open-webui/tests/.github/workflows/regression.yml@main
|
||||
with:
|
||||
open-webui-ref: ${{ github.event.pull_request.head.sha }}
|
||||
|
||||
# Single check to require in branch protection.
|
||||
result:
|
||||
name: Result
|
||||
needs: [regression]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions: {}
|
||||
steps:
|
||||
- name: Fail unless the suite passed or was not required
|
||||
if: needs.regression.result != 'success' && needs.regression.result != 'skipped'
|
||||
run: exit 1
|
||||
|
|
@ -34,10 +34,21 @@ Examples of unacceptable behavior include:
|
|||
- **Spamming and promotional exploitation.** Sharing irrelevant product promotions or self-promotion in the community is not allowed unless it directly contributes value to the discussion.
|
||||
- Posting low-effort, hard to read, essay-length AI generated comments or other forms of low-quality, hard to parse content that puts the burden of understanding on the reader.
|
||||
|
||||
### How We Develop the Project
|
||||
|
||||
Development is led by the maintainers, and code pull requests are reserved for work we explicitly request or exceptional contributions we choose to consider at our discretion. We use actionable reports and concrete use cases to understand problems, then evaluate, revise, and implement the appropriate approach internally. We assess each change against the project's architecture, existing behavior, quality standards, and future direction before settling on an implementation. Resolving a reported problem requires that broader context, and a working external patch usually requires substantial rewriting to meet the project's standards. Reviewing the patch, explaining the required changes, and coordinating successive revisions usually takes more effort than developing the solution internally. Fragmented commit histories, branches that have not been rebased, unresolved conflicts, and lengthy or unverified AI-generated comments add cleanup and discussion that delay the underlying work. Maintainers remain responsible for testing, documenting, supporting, and maintaining every accepted change, so we choose the approach based on the whole product and its ongoing maintenance. Clear reports, reproduction details, and relevant context give us what we need to make those decisions and develop the solution. A polished implementation, clean commit history, or completed checklist does not establish an exception to this process, and opening an issue or discussion is not an invitation to submit a PR. Wait for an explicit maintainer request before investing in a PR; unsolicited submissions are generally closed without review, and requested PRs remain subject to maintainer judgment.
|
||||
|
||||
### Feedback and Community Engagement
|
||||
|
||||
- **Constructive feedback is encouraged, but hostile or entitled behavior will result in immediate action.** If you disagree with elements of the project, we encourage you to offer meaningful improvements or fork the project if necessary. Healthy discussions and technical disagreements are welcome only when handled with professionalism.
|
||||
- **Respect contributors' time and efforts.** No one is entitled to personalized or on-demand assistance. This is a community built on collaboration and shared effort; demanding or demeaning behavior undermines that trust and will not be allowed.
|
||||
Participation should help maintainers understand a concrete problem while respecting the project's priorities and available capacity. Please follow the [issue templates](.github/ISSUE_TEMPLATE) and [pull request policy](.github/pull_request_template.md) before submitting anything.
|
||||
|
||||
- **Make reports actionable.** Search existing issues and discussions, check the latest version and whether the problem is already addressed on `dev`, and use the appropriate template. Bug reports should describe a reproducible problem, the affected workflow, expected and actual behavior, and relevant evidence. Feature requests should explain the user-facing need; broader product, UX, architecture, or maintenance questions belong in Discussions. Report security concerns privately through the [security reporting process](https://github.com/open-webui/open-webui/security).
|
||||
- **Share the problem before investing in code.** Start with an actionable issue or discussion and leave implementation planning to the maintainers. An issue or discussion alone is not an invitation to submit a PR. Please wait for an explicit request before opening one; any exception is at the maintainers' discretion. Implementation notes, local diffs, or patches may be shared as reference in the relevant issue or discussion.
|
||||
- **Respect maintainers' discretion.** Submitting an issue, proposal, or pull request does not create an obligation to respond, review, implement, or merge it. Maintainers set the project's direction and defer or close submissions based on scope, quality, maintenance cost, or available capacity. Unsolicited pull requests are generally closed without review.
|
||||
- **Keep discussion focused and concise.** Provide new information when it helps evaluate the problem. Repeated bumps, duplicate submissions, unsolicited direct messages seeking attention, or pressure for timelines place an unnecessary burden on contributors.
|
||||
- **Respect decisions and boundaries.** Technical disagreement is welcome when expressed professionally. Reopening a declined request or continuing to press for a different outcome without new, relevant information is not constructive. You are free to explore a different direction in your own fork.
|
||||
|
||||
Participants are expected to respect maintainers' decisions and the contribution process. Harassment, hostility, or repeated disregard for these boundaries will result in enforcement under this Code of Conduct.
|
||||
|
||||
### Zero Tolerance: No Warnings, Immediate Action
|
||||
|
||||
|
|
|
|||
58
Dockerfile
58
Dockerfile
|
|
@ -26,6 +26,9 @@ ARG GID=0
|
|||
######## WebUI frontend ########
|
||||
FROM --platform=$BUILDPLATFORM node:22-alpine3.20 AS build
|
||||
ARG BUILD_HASH
|
||||
ARG USE_SLIM
|
||||
ARG UID
|
||||
ARG GID
|
||||
|
||||
# Set Node.js options (heap limit Allocation failed - JavaScript heap out of memory)
|
||||
# ENV NODE_OPTIONS="--max-old-space-size=4096"
|
||||
|
|
@ -40,7 +43,14 @@ RUN npm ci --force
|
|||
|
||||
COPY . .
|
||||
ENV APP_BUILD_HASH=${BUILD_HASH}
|
||||
RUN npm run build
|
||||
RUN npm run build && \
|
||||
if [ "$USE_SLIM" = "true" ]; then find build -type f -name '*.map' -delete; fi
|
||||
|
||||
# Prepare backend ownership before the final copy so static assets occupy one layer.
|
||||
# Group 0 write access lets arbitrary OpenShift UIDs update these assets at startup.
|
||||
RUN chown -R $UID:$GID /app/backend && \
|
||||
chgrp -R 0 /app/backend/open_webui/static && \
|
||||
chmod -R g=u /app/backend/open_webui/static
|
||||
|
||||
######## WebUI backend ########
|
||||
FROM python:3.11-slim-bookworm AS base
|
||||
|
|
@ -123,24 +133,33 @@ RUN echo -n 00000000-0000-0000-0000-000000000000 > $HOME/.cache/chroma/telemetry
|
|||
# Make sure the user has access to the app and root directory
|
||||
RUN chown -R $UID:$GID /app $HOME
|
||||
|
||||
# Install common system dependencies
|
||||
# Slim cannot bundle a local model server or GPU runtime.
|
||||
RUN if [ "$USE_SLIM" = "true" ] && { [ "$USE_CUDA" = "true" ] || [ "$USE_OLLAMA" = "true" ]; }; then \
|
||||
echo "USE_SLIM cannot be combined with USE_CUDA or USE_OLLAMA" >&2; exit 1; fi
|
||||
|
||||
# Keep the slim runtime free of local document/audio processing tools.
|
||||
# Git-based tool requirements require the standard image.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
git build-essential pandoc gcc curl jq ca-certificates \
|
||||
libmariadb-dev \
|
||||
python3-dev \
|
||||
ffmpeg libsm6 libxext6 zstd \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
curl jq ca-certificates \
|
||||
&& if [ "$USE_SLIM" != "true" ]; then \
|
||||
apt-get install -y --no-install-recommends \
|
||||
git build-essential pandoc gcc libmariadb-dev ffmpeg libsm6 libxext6; \
|
||||
fi && if [ "$USE_OLLAMA" = "true" ]; then \
|
||||
apt-get install -y --no-install-recommends zstd; \
|
||||
fi && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# install python dependencies
|
||||
COPY --chown=$UID:$GID ./backend/requirements.txt ./requirements.txt
|
||||
COPY --chown=$UID:$GID ./backend/requirements*.txt ./
|
||||
|
||||
# Set UV_LINK_MODE to copy to prevent 0-byte file corruption in QEMU arm64 cross-builds
|
||||
ENV UV_LINK_MODE=copy
|
||||
|
||||
RUN set -e; \
|
||||
pip3 install --no-cache-dir uv; \
|
||||
if [ "$USE_CUDA" = "true" ]; then \
|
||||
RUN --mount=from=ghcr.io/astral-sh/uv:0.12.10,source=/uv,target=/bin/uv \
|
||||
set -e; \
|
||||
if [ "$USE_SLIM" = "true" ]; then \
|
||||
uv pip install --system -r requirements-slim.txt --no-cache-dir; \
|
||||
elif [ "$USE_CUDA" = "true" ]; then \
|
||||
# If you use CUDA the whisper and embedding model will be downloaded on first use
|
||||
# fix: pin torch<=2.9.1 - torch 2.10.0 aarch64 wheels cause SIGILL on ARM devices (RPi 4 Cortex-A72) #21349
|
||||
pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/$USE_CUDA_DOCKER_VER --no-cache-dir; \
|
||||
|
|
@ -149,7 +168,6 @@ RUN set -e; \
|
|||
python -c "import os; from sentence_transformers import SentenceTransformer; SentenceTransformer(os.environ.get('AUXILIARY_EMBEDDING_MODEL', 'TaylorAI/bge-micro-v2'), device='cpu')"; \
|
||||
python -c "import os; from faster_whisper import WhisperModel; WhisperModel(os.environ['WHISPER_MODEL'], device='cpu', compute_type='int8', download_root=os.environ['WHISPER_MODEL_DIR'])"; \
|
||||
python -c "import os; import tiktoken; tiktoken.get_encoding(os.environ['TIKTOKEN_ENCODING_NAME'])"; \
|
||||
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
|
||||
else \
|
||||
pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu --no-cache-dir; \
|
||||
uv pip install --system -r requirements.txt --no-cache-dir; \
|
||||
|
|
@ -158,7 +176,6 @@ RUN set -e; \
|
|||
python -c "import os; from sentence_transformers import SentenceTransformer; SentenceTransformer(os.environ.get('AUXILIARY_EMBEDDING_MODEL', 'TaylorAI/bge-micro-v2'), device='cpu')"; \
|
||||
python -c "import os; from faster_whisper import WhisperModel; WhisperModel(os.environ['WHISPER_MODEL'], device='cpu', compute_type='int8', download_root=os.environ['WHISPER_MODEL_DIR'])"; \
|
||||
python -c "import os; import tiktoken; tiktoken.get_encoding(os.environ['TIKTOKEN_ENCODING_NAME'])"; \
|
||||
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
|
||||
fi; \
|
||||
fi; \
|
||||
mkdir -p /app/backend/data; chown -R $UID:$GID /app/backend/data/; \
|
||||
|
|
@ -187,19 +204,8 @@ COPY --chown=$UID:$GID --from=build /app/build /app/build
|
|||
COPY --chown=$UID:$GID --from=build /app/CHANGELOG.md /app/CHANGELOG.md
|
||||
COPY --chown=$UID:$GID --from=build /app/package.json /app/package.json
|
||||
|
||||
# copy backend files
|
||||
COPY --chown=$UID:$GID ./backend .
|
||||
|
||||
# The backend rewrites its bundled static assets (favicons, splash, manifest,
|
||||
# loader.js, ...) under open_webui/static at startup. Make that directory
|
||||
# writable by an arbitrary UID -- which under OpenShift's restricted SCC is
|
||||
# always a member of GID 0 -- so those writes don't fail with EACCES and crash
|
||||
# the boot log with "[Errno 13] Permission denied". `chmod -R g=u` mirrors the
|
||||
# owner bits onto the group (the Red Hat arbitrary-UID idiom). This is applied
|
||||
# unconditionally because it targets a directory the app writes on every start;
|
||||
# the broader, opt-in USE_PERMISSION_HARDENING below covers the rest of /app.
|
||||
RUN chgrp -R 0 /app/backend/open_webui/static && \
|
||||
chmod -R g=u /app/backend/open_webui/static
|
||||
# copy backend files with the ownership and static permissions prepared above
|
||||
COPY --from=build /app/backend .
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ For more information, be sure to check out our [Open WebUI Documentation](https:
|
|||
|
||||
- 🤖 **Models & Agents**: Wrap any base model with custom instructions, tools, and knowledge to build specialized agents. Supports dynamic variables, per-user/group access control, and community preset imports via [Open WebUI Community](https://openwebui.com/).
|
||||
|
||||
- ⚡ **Agentic Execution with [Open Terminal](https://github.com/open-webui/open-terminal)**: Give your agents a terminal and filesystem to carry out multi-step tasks. Let them analyze data, run scripts, fix errors, and produce files directly in chat. Scale to teams with **[Terminals (Enterprise)](https://github.com/open-webui/terminals)** for per-user isolated environments, resource limits, and automatic lifecycle management.
|
||||
|
||||
- 📝 **Notes**: A dedicated workspace for content outside conversations. Draft with a rich editor, use AI to rewrite selected text, and attach notes to any chat for full-context injection.
|
||||
|
||||
- 📢 **Channels**: Real-time shared spaces where your team and AI models collaborate in one timeline. Tag models to draft or critique, with threads, reactions, pins, and access control.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from authlib.integrations.starlette_client import OAuth
|
|||
from pydantic import BaseModel
|
||||
|
||||
from open_webui.env import (
|
||||
USE_SLIM,
|
||||
DATA_DIR,
|
||||
DATABASE_URL,
|
||||
ENABLE_ADMIN_CHAT_ACCESS,
|
||||
|
|
@ -227,6 +228,7 @@ if CUSTOM_NAME:
|
|||
####################################
|
||||
|
||||
ENABLE_DIRECT_CONNECTIONS = os.getenv('ENABLE_DIRECT_CONNECTIONS', 'False').lower() == 'true'
|
||||
ENABLE_DIRECT_INTEGRATIONS = os.getenv('ENABLE_DIRECT_INTEGRATIONS', 'False').lower() == 'true'
|
||||
|
||||
####################################
|
||||
# OLLAMA_BASE_URL
|
||||
|
|
@ -498,12 +500,12 @@ CODE_INTERPRETER_PYODIDE_PROMPT = """
|
|||
# Vector Database
|
||||
####################################
|
||||
|
||||
VECTOR_DB = os.getenv('VECTOR_DB', 'chroma')
|
||||
VECTOR_DB = os.getenv('VECTOR_DB', 'pgvector' if USE_SLIM else 'chroma')
|
||||
|
||||
# Chroma
|
||||
CHROMA_DATA_PATH = f'{DATA_DIR}/vector_db'
|
||||
|
||||
if VECTOR_DB == 'chroma':
|
||||
if VECTOR_DB == 'chroma' and not USE_SLIM:
|
||||
import chromadb
|
||||
|
||||
CHROMA_TENANT = os.getenv('CHROMA_TENANT', chromadb.DEFAULT_TENANT)
|
||||
|
|
@ -644,7 +646,7 @@ SSL_ASSERT_FINGERPRINT = os.getenv('SSL_ASSERT_FINGERPRINT', None)
|
|||
ELASTICSEARCH_INDEX_PREFIX = os.getenv('ELASTICSEARCH_INDEX_PREFIX', 'open_webui_collections')
|
||||
# Pgvector
|
||||
PGVECTOR_DB_URL = os.getenv('PGVECTOR_DB_URL', DATABASE_URL)
|
||||
if VECTOR_DB == 'pgvector' and not PGVECTOR_DB_URL.startswith('postgres'):
|
||||
if not USE_SLIM and VECTOR_DB == 'pgvector' and not PGVECTOR_DB_URL.startswith('postgres'):
|
||||
raise ValueError(
|
||||
'Pgvector requires setting PGVECTOR_DB_URL or using Postgres with vector extension as the primary database.'
|
||||
)
|
||||
|
|
@ -739,6 +741,10 @@ else:
|
|||
except Exception:
|
||||
PGVECTOR_IVFFLAT_LISTS = 100
|
||||
|
||||
PGVECTOR_ITERATIVE_SCAN = os.getenv('PGVECTOR_ITERATIVE_SCAN', 'relaxed_order').strip().lower()
|
||||
if PGVECTOR_ITERATIVE_SCAN not in ('off', 'relaxed_order', 'strict_order'):
|
||||
PGVECTOR_ITERATIVE_SCAN = 'relaxed_order'
|
||||
|
||||
# openGauss
|
||||
OPENGAUSS_DB_URL = os.getenv('OPENGAUSS_DB_URL', DATABASE_URL)
|
||||
|
||||
|
|
@ -805,7 +811,7 @@ ORACLE_DB_POOL_MAX = int(os.getenv('ORACLE_DB_POOL_MAX', 10))
|
|||
ORACLE_DB_POOL_INCREMENT = int(os.getenv('ORACLE_DB_POOL_INCREMENT', 1))
|
||||
|
||||
|
||||
if VECTOR_DB == 'oracle23ai':
|
||||
if not USE_SLIM and VECTOR_DB == 'oracle23ai':
|
||||
if not ORACLE_DB_USER or not ORACLE_DB_PASSWORD or not ORACLE_DB_DSN:
|
||||
raise ValueError('Oracle23ai requires setting ORACLE_DB_USER, ORACLE_DB_PASSWORD, and ORACLE_DB_DSN.')
|
||||
if ORACLE_DB_USE_WALLET and (not ORACLE_WALLET_DIR or not ORACLE_WALLET_PASSWORD):
|
||||
|
|
@ -1270,6 +1276,9 @@ AZURE_AI_SEARCH_ENDPOINT = os.getenv('AZURE_AI_SEARCH_ENDPOINT', '')
|
|||
AZURE_AI_SEARCH_INDEX_NAME = os.getenv('AZURE_AI_SEARCH_INDEX_NAME', '')
|
||||
|
||||
EXA_API_KEY = os.getenv('EXA_API_KEY', '')
|
||||
EXA_MAX_CONTENT_LENGTH = int(os.environ['EXA_MAX_CONTENT_LENGTH']) if os.getenv('EXA_MAX_CONTENT_LENGTH') else None
|
||||
if EXA_MAX_CONTENT_LENGTH is not None and EXA_MAX_CONTENT_LENGTH <= 0:
|
||||
raise ValueError('EXA_MAX_CONTENT_LENGTH must be a positive integer or unset')
|
||||
|
||||
PERPLEXITY_API_KEY = os.getenv('PERPLEXITY_API_KEY', '')
|
||||
|
||||
|
|
@ -1293,6 +1302,12 @@ TAVILY_API_KEY = os.getenv('TAVILY_API_KEY', '')
|
|||
|
||||
TAVILY_EXTRACT_DEPTH = os.getenv('TAVILY_EXTRACT_DEPTH', 'basic')
|
||||
|
||||
STAAN_API_KEY = os.getenv('STAAN_API_KEY', '')
|
||||
|
||||
STAAN_MARKET = os.getenv('STAAN_MARKET', 'en-us')
|
||||
|
||||
STAAN_MAX_SNIPPETS = int(os.getenv('STAAN_MAX_SNIPPETS', '0'))
|
||||
|
||||
PLAYWRIGHT_WS_URL = os.getenv('PLAYWRIGHT_WS_URL', '')
|
||||
|
||||
PLAYWRIGHT_TIMEOUT = int(os.getenv('PLAYWRIGHT_TIMEOUT', '10000'))
|
||||
|
|
@ -1697,6 +1712,7 @@ if default_prompt_suggestions == []:
|
|||
]
|
||||
|
||||
DEFAULT_PROMPT_SUGGESTIONS = default_prompt_suggestions
|
||||
DEFAULT_PROMPT_SUGGESTIONS_I18N = {}
|
||||
|
||||
try:
|
||||
model_order_list = JSONCodec.loads(os.getenv('MODEL_ORDER_LIST', '[]'))
|
||||
|
|
@ -2162,6 +2178,7 @@ else:
|
|||
|
||||
|
||||
class BannerModel(BaseModel):
|
||||
i18n: dict[str, dict[str, str]] | None = None
|
||||
id: str
|
||||
type: str
|
||||
title: str | None = None
|
||||
|
|
@ -2834,6 +2851,7 @@ LDAP_ATTRIBUTE_FOR_GROUPS = os.getenv('LDAP_ATTRIBUTE_FOR_GROUPS', 'memberOf')
|
|||
|
||||
DEFAULT_CONFIG = {
|
||||
'direct.enable': ENABLE_DIRECT_CONNECTIONS,
|
||||
'direct.integrations.enable': ENABLE_DIRECT_INTEGRATIONS,
|
||||
'ollama.enable': ENABLE_OLLAMA_API,
|
||||
'ollama.base_urls': OLLAMA_BASE_URLS,
|
||||
'ollama.api_configs': OLLAMA_API_CONFIGS,
|
||||
|
|
@ -2998,6 +3016,7 @@ DEFAULT_CONFIG = {
|
|||
'web.search.azure_ai_search_endpoint': AZURE_AI_SEARCH_ENDPOINT,
|
||||
'web.search.azure_ai_search_index_name': AZURE_AI_SEARCH_INDEX_NAME,
|
||||
'web.search.exa_api_key': EXA_API_KEY,
|
||||
'web.search.exa_max_content_length': EXA_MAX_CONTENT_LENGTH,
|
||||
'web.search.perplexity_api_key': PERPLEXITY_API_KEY,
|
||||
'web.search.perplexity_model': PERPLEXITY_MODEL,
|
||||
'web.search.perplexity_search_context_usage': PERPLEXITY_SEARCH_CONTEXT_USAGE,
|
||||
|
|
@ -3009,6 +3028,9 @@ DEFAULT_CONFIG = {
|
|||
'web.search.sougou_api_sk': SOUGOU_API_SK,
|
||||
'web.search.tavily_api_key': TAVILY_API_KEY,
|
||||
'web.search.tavily_extract_depth': TAVILY_EXTRACT_DEPTH,
|
||||
'web.search.staan_api_key': STAAN_API_KEY,
|
||||
'web.search.staan_market': STAAN_MARKET,
|
||||
'web.search.staan_max_snippets': STAAN_MAX_SNIPPETS,
|
||||
'web.loader.playwright_ws_url': PLAYWRIGHT_WS_URL,
|
||||
'web.loader.playwright_timeout': PLAYWRIGHT_TIMEOUT,
|
||||
'web.loader.firecrawl_api_key': FIRECRAWL_API_KEY,
|
||||
|
|
@ -3095,7 +3117,9 @@ DEFAULT_CONFIG = {
|
|||
'ui.default_models': DEFAULT_MODELS,
|
||||
'ui.default_pinned_models': DEFAULT_PINNED_MODELS,
|
||||
'ui.default_interface_settings': DEFAULT_INTERFACE_SETTINGS,
|
||||
'ui.i18n': {},
|
||||
'ui.prompt_suggestions': DEFAULT_PROMPT_SUGGESTIONS,
|
||||
'ui.prompt_suggestions_i18n': DEFAULT_PROMPT_SUGGESTIONS_I18N,
|
||||
'ui.model_order_list': MODEL_ORDER_LIST,
|
||||
'models.default_metadata': DEFAULT_MODEL_METADATA,
|
||||
'models.default_params': DEFAULT_MODEL_PARAMS,
|
||||
|
|
|
|||
|
|
@ -7,7 +7,9 @@ import pkgutil
|
|||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import threading
|
||||
import traceback
|
||||
from contextlib import nullcontext
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
from uuid import uuid4
|
||||
|
|
@ -40,12 +42,13 @@ except ImportError:
|
|||
print('dotenv not installed, skipping...')
|
||||
|
||||
DOCKER = os.getenv('DOCKER', 'False').lower() == 'true'
|
||||
USE_SLIM = os.getenv('USE_SLIM_DOCKER', 'False').lower() == 'true'
|
||||
|
||||
USE_CUDA = os.getenv('USE_CUDA_DOCKER', 'false')
|
||||
DEVICE_TYPE = 'cpu'
|
||||
_cuda_error: Optional[str] = None
|
||||
|
||||
if USE_CUDA.lower() == 'true':
|
||||
if not USE_SLIM and USE_CUDA.lower() == 'true':
|
||||
try:
|
||||
import torch # noqa: E402
|
||||
|
||||
|
|
@ -57,7 +60,7 @@ if USE_CUDA.lower() == 'true':
|
|||
os.environ['USE_CUDA_DOCKER'] = 'false'
|
||||
USE_CUDA = 'false'
|
||||
|
||||
if sys.platform == 'darwin' and DEVICE_TYPE == 'cpu':
|
||||
if not USE_SLIM and sys.platform == 'darwin' and DEVICE_TYPE == 'cpu':
|
||||
try:
|
||||
import torch # noqa: E402
|
||||
|
||||
|
|
@ -66,6 +69,9 @@ if sys.platform == 'darwin' and DEVICE_TYPE == 'cpu':
|
|||
except Exception:
|
||||
pass
|
||||
|
||||
# Torch MPS inference is not thread-safe and a concurrent call kills the whole process.
|
||||
MPS_INFERENCE_LOCK = threading.Lock() if DEVICE_TYPE == 'mps' else nullcontext()
|
||||
|
||||
####################################
|
||||
# LOGGING
|
||||
####################################
|
||||
|
|
@ -245,8 +251,6 @@ if FROM_INIT_PY:
|
|||
|
||||
STATIC_DIR = Path(os.getenv('STATIC_DIR', OPEN_WEBUI_DIR / 'static'))
|
||||
|
||||
FONTS_DIR = Path(os.getenv('FONTS_DIR', OPEN_WEBUI_DIR / 'static' / 'fonts'))
|
||||
|
||||
FRONTEND_BUILD_DIR = Path(os.getenv('FRONTEND_BUILD_DIR', BASE_DIR / 'build')).resolve()
|
||||
|
||||
if FROM_INIT_PY:
|
||||
|
|
@ -367,6 +371,9 @@ ENABLE_QUERIES_CACHE = os.getenv('ENABLE_QUERIES_CACHE', 'False').lower() == 'tr
|
|||
ENABLE_ADMIN_CHAT_ACCESS = os.getenv('ENABLE_ADMIN_CHAT_ACCESS', 'True').lower() == 'true'
|
||||
RAG_SYSTEM_CONTEXT = os.getenv('RAG_SYSTEM_CONTEXT', 'False').lower() == 'true'
|
||||
|
||||
# Empty by default: chunk metadata also holds internal bookkeeping (file hashes, collection names, scores).
|
||||
RAG_SOURCE_METADATA_KEYS = [key.strip() for key in os.getenv('RAG_SOURCE_METADATA_KEYS', '').split(',') if key.strip()]
|
||||
|
||||
####################################
|
||||
# REDIS
|
||||
####################################
|
||||
|
|
@ -381,6 +388,14 @@ try:
|
|||
except ValueError:
|
||||
REDIS_RESPONSE_STREAM_TTL = 3600
|
||||
|
||||
# Seconds a task survives without a heartbeat. 0 disables expiry.
|
||||
try:
|
||||
REDIS_TASK_TTL = int(os.getenv('REDIS_TASK_TTL', '300'))
|
||||
if REDIS_TASK_TTL != 0 and REDIS_TASK_TTL < 60:
|
||||
REDIS_TASK_TTL = 300
|
||||
except ValueError:
|
||||
REDIS_TASK_TTL = 300
|
||||
|
||||
REDIS_SENTINEL_HOSTS = os.getenv('REDIS_SENTINEL_HOSTS', '')
|
||||
REDIS_SENTINEL_PORT = os.getenv('REDIS_SENTINEL_PORT', '26379')
|
||||
|
||||
|
|
@ -839,7 +854,7 @@ MINERU_MAX_MARKDOWN_BYTES = (
|
|||
# When enabled, skips pydub-based preprocessing (format conversion, compression,
|
||||
# and chunked splitting) before sending files to processing engines. Useful when
|
||||
# the upstream provider handles these steps or when ffmpeg is unavailable.
|
||||
BYPASS_PYDUB_PREPROCESSING = os.getenv('BYPASS_PYDUB_PREPROCESSING', 'False').lower() == 'true'
|
||||
BYPASS_PYDUB_PREPROCESSING = USE_SLIM or os.getenv('BYPASS_PYDUB_PREPROCESSING', 'False').lower() == 'true'
|
||||
|
||||
# When disabled (default), the OpenAI catch-all proxy endpoint (/{path:path})
|
||||
# is blocked. Enable only if you need direct passthrough to upstream OpenAI-
|
||||
|
|
@ -979,6 +994,7 @@ FORWARD_USER_INFO_HEADER_USER_NAME = os.getenv('FORWARD_USER_INFO_HEADER_USER_NA
|
|||
FORWARD_USER_INFO_HEADER_USER_ID = os.getenv('FORWARD_USER_INFO_HEADER_USER_ID', 'X-OpenWebUI-User-Id')
|
||||
FORWARD_USER_INFO_HEADER_USER_EMAIL = os.getenv('FORWARD_USER_INFO_HEADER_USER_EMAIL', 'X-OpenWebUI-User-Email')
|
||||
FORWARD_USER_INFO_HEADER_USER_ROLE = os.getenv('FORWARD_USER_INFO_HEADER_USER_ROLE', 'X-OpenWebUI-User-Role')
|
||||
FORWARD_USER_INFO_HEADER_AUTH_TYPE = os.getenv('FORWARD_USER_INFO_HEADER_AUTH_TYPE', 'X-OpenWebUI-Auth-Type')
|
||||
FORWARD_SESSION_INFO_HEADER_MESSAGE_ID = os.getenv('FORWARD_SESSION_INFO_HEADER_MESSAGE_ID', 'X-OpenWebUI-Message-Id')
|
||||
FORWARD_SESSION_INFO_HEADER_CHAT_ID = os.getenv('FORWARD_SESSION_INFO_HEADER_CHAT_ID', 'X-OpenWebUI-Chat-Id')
|
||||
|
||||
|
|
@ -1037,6 +1053,11 @@ ENABLE_CHAT_RESPONSE_BASE64_IMAGE_URL_CONVERSION = (
|
|||
)
|
||||
ENABLE_API_OUTLET_FILTERS = os.getenv('ENABLE_API_OUTLET_FILTERS', 'True').lower() == 'true'
|
||||
|
||||
# Opt in to CPython's in-place string append optimization for streamed responses.
|
||||
# Off by default for a staged rollout. Only a host already out of memory can lose
|
||||
# text here; the default path (a full copy per chunk) raises there too.
|
||||
ENABLE_CHAT_RESPONSE_STREAM_INPLACE_APPEND = os.getenv('ENABLE_CHAT_RESPONSE_STREAM_INPLACE_APPEND', 'False').lower() == 'true'
|
||||
|
||||
# When enabled, uses a hardcoded extension-to-MIME dictionary as a last-resort
|
||||
# fallback when both mimetypes.guess_type() and file.meta.content_type fail to
|
||||
# determine the content type. This can help on minimal container images (e.g.
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ from open_webui.env import (
|
|||
DATABASE_URL,
|
||||
ENABLE_DB_MIGRATIONS,
|
||||
OPEN_WEBUI_DIR,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from sqlalchemy import Dialect, MetaData, create_engine, event, types
|
||||
|
|
@ -142,6 +143,17 @@ class JSONField(types.TypeDecorator): # TEXT-backed JSON storage
|
|||
return JSONField(length=self.impl.length)
|
||||
|
||||
|
||||
if USE_SLIM:
|
||||
if make_url(DATABASE_URL).get_backend_name() not in ('sqlite', 'postgresql', 'postgres'):
|
||||
raise ValueError(
|
||||
'Slim requires SQLite or PostgreSQL for DATABASE_URL. Use the standard image for other databases.'
|
||||
)
|
||||
if DATABASE_ENABLE_IAM_TOKEN_AUTH:
|
||||
raise ValueError(
|
||||
'AWS RDS IAM authentication requires the standard image. Slim supports PostgreSQL database credentials.'
|
||||
)
|
||||
|
||||
|
||||
# Normalize SSL params from the URL once; the sync engine needs them
|
||||
# reattached in canonical libpq form for psycopg2.
|
||||
_url_without_ssl, _ssl_dict = extract_ssl_params_from_url(DATABASE_URL)
|
||||
|
|
|
|||
|
|
@ -74,7 +74,9 @@ from open_webui.config import (
|
|||
seed_registered_defaults,
|
||||
)
|
||||
from open_webui.constants import ERROR_MESSAGES, TASKS
|
||||
from open_webui.utils.recurrence import RecurrenceEvaluationTimeout
|
||||
from open_webui.env import (
|
||||
USE_SLIM,
|
||||
AIOHTTP_CLIENT_SESSION_SSL,
|
||||
AUDIT_EXCLUDED_PATHS,
|
||||
AUDIT_INCLUDED_PATHS,
|
||||
|
|
@ -106,12 +108,14 @@ from open_webui.env import (
|
|||
MAX_BODY_LOG_SIZE,
|
||||
# Redis
|
||||
REDIS_KEY_PREFIX,
|
||||
REDIS_TASK_TTL,
|
||||
REDIS_URL,
|
||||
RESET_CONFIG_ON_START,
|
||||
SAFE_MODE,
|
||||
SCIM_TOKEN,
|
||||
VERSION,
|
||||
WEBSOCKET_HEARTBEAT_INTERVAL,
|
||||
WEBSOCKET_MANAGER,
|
||||
# Admin Account Runtime Creation
|
||||
WEBUI_ADMIN_EMAIL,
|
||||
WEBUI_ADMIN_NAME,
|
||||
|
|
@ -188,6 +192,7 @@ from open_webui.socket.main import (
|
|||
get_user_id_from_session_pool,
|
||||
periodic_session_pool_cleanup,
|
||||
periodic_usage_pool_cleanup,
|
||||
redis_event_listener,
|
||||
)
|
||||
from open_webui.socket.main import (
|
||||
app as socket_app,
|
||||
|
|
@ -199,6 +204,7 @@ from open_webui.tasks import (
|
|||
list_task_ids_by_item_id,
|
||||
list_tasks,
|
||||
redis_task_command_listener,
|
||||
redis_task_heartbeat,
|
||||
stop_item_tasks,
|
||||
stop_task,
|
||||
) # Import from tasks.py
|
||||
|
|
@ -385,6 +391,11 @@ async def lifespan(app: FastAPI):
|
|||
|
||||
if app.state.redis is not None:
|
||||
app.state.redis_task_command_listener = asyncio.create_task(redis_task_command_listener(app))
|
||||
if REDIS_TASK_TTL > 0:
|
||||
app.state.redis_task_heartbeat = asyncio.create_task(redis_task_heartbeat(app))
|
||||
|
||||
if WEBSOCKET_MANAGER == 'redis':
|
||||
app.state.redis_event_listener = asyncio.create_task(redis_event_listener())
|
||||
|
||||
app.state.periodic_usage_pool_cleanup = asyncio.create_task(periodic_usage_pool_cleanup())
|
||||
app.state.periodic_session_pool_cleanup = asyncio.create_task(periodic_session_pool_cleanup())
|
||||
|
|
@ -472,6 +483,12 @@ async def lifespan(app: FastAPI):
|
|||
if hasattr(app.state, 'redis_task_command_listener'):
|
||||
app.state.redis_task_command_listener.cancel()
|
||||
|
||||
if hasattr(app.state, 'redis_task_heartbeat'):
|
||||
app.state.redis_task_heartbeat.cancel()
|
||||
|
||||
if hasattr(app.state, 'redis_event_listener'):
|
||||
app.state.redis_event_listener.cancel()
|
||||
|
||||
app.state.periodic_usage_pool_cleanup.cancel()
|
||||
app.state.periodic_session_pool_cleanup.cancel()
|
||||
app.state.scheduler_worker_loop.cancel()
|
||||
|
|
@ -494,6 +511,12 @@ app = FastAPI(
|
|||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(RecurrenceEvaluationTimeout)
|
||||
async def recurrence_timeout_handler(request: Request, exc: RecurrenceEvaluationTimeout):
|
||||
return JSONResponse(status_code=400, content={'detail': str(exc)})
|
||||
|
||||
|
||||
# Used by readiness checks to gate traffic until startup work is done.
|
||||
app.state.startup_complete = False
|
||||
|
||||
|
|
@ -1463,8 +1486,8 @@ async def chat_completion(
|
|||
async def run_initial_title_generation():
|
||||
try:
|
||||
await background_tasks_handler(title_ctx)
|
||||
except Exception as e:
|
||||
log.debug('Error generating initial chat title: %s', e)
|
||||
except Exception:
|
||||
log.exception('Error generating initial chat title')
|
||||
|
||||
asyncio.create_task(run_initial_title_generation())
|
||||
else:
|
||||
|
|
@ -1623,6 +1646,9 @@ async def chat_completion(
|
|||
|
||||
async def process_chat(request, form_data, user, metadata, model, tasks=None):
|
||||
try:
|
||||
ctx = None
|
||||
if metadata.get('assistant_message_id'):
|
||||
ctx = await build_chat_response_context(request, form_data, user, model, metadata, tasks, [])
|
||||
form_data, metadata, events = await process_chat_payload(request, form_data, user, metadata, model)
|
||||
|
||||
if await drain_approved_tool_calls(request, form_data, user, model, metadata):
|
||||
|
|
@ -1633,12 +1659,15 @@ async def chat_completion(
|
|||
# When the upstream provider returns an error (e.g. HTTP 400
|
||||
# content-filter, quota exceeded), generate_chat_completion
|
||||
# returns a JSONResponse instead of raising. Detect this and
|
||||
# raise so the except-block below emits chat:message:error +
|
||||
# chat:tasks:cancel, unblocking the frontend.
|
||||
# raise so the except-block below emits a terminal
|
||||
# chat:message:error, unblocking the frontend.
|
||||
if isinstance(response, JSONResponse) and response.status_code >= 400:
|
||||
raise Exception(get_response_error_detail(response))
|
||||
|
||||
ctx = await build_chat_response_context(request, form_data, user, model, metadata, tasks, events)
|
||||
if ctx is None:
|
||||
ctx = await build_chat_response_context(request, form_data, user, model, metadata, tasks, events)
|
||||
else:
|
||||
ctx.update(form_data=form_data, metadata=metadata, events=events)
|
||||
|
||||
return await process_chat_response(response, ctx)
|
||||
except asyncio.CancelledError:
|
||||
|
|
@ -1667,6 +1696,7 @@ async def chat_completion(
|
|||
{
|
||||
'parentId': metadata.get('user_message_id', None),
|
||||
'error': {'content': error_detail},
|
||||
'done': True,
|
||||
},
|
||||
)
|
||||
|
||||
|
|
@ -1675,12 +1705,9 @@ async def chat_completion(
|
|||
await event_emitter(
|
||||
{
|
||||
'type': 'chat:message:error',
|
||||
'data': {'error': {'content': error_detail}},
|
||||
'data': {'error': {'content': error_detail}, 'done': True},
|
||||
}
|
||||
)
|
||||
await event_emitter(
|
||||
{'type': 'chat:tasks:cancel'},
|
||||
)
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
|
|
@ -2235,6 +2262,7 @@ async def get_app_config(request: Request):
|
|||
'auth.enable_api_keys',
|
||||
'ui.enable_password_change_form',
|
||||
'direct.enable',
|
||||
'direct.integrations.enable',
|
||||
'folders.enable',
|
||||
'folders.max_file_count',
|
||||
'channels.enable',
|
||||
|
|
@ -2260,7 +2288,9 @@ async def get_app_config(request: Request):
|
|||
'ui.default_models',
|
||||
'ui.default_pinned_models',
|
||||
'ui.default_interface_settings',
|
||||
'ui.i18n',
|
||||
'ui.prompt_suggestions',
|
||||
'ui.prompt_suggestions_i18n',
|
||||
'code_execution.engine',
|
||||
'code_interpreter.engine',
|
||||
'audio.tts.engine',
|
||||
|
|
@ -2283,6 +2313,7 @@ async def get_app_config(request: Request):
|
|||
'name': app.state.WEBUI_NAME,
|
||||
'version': VERSION,
|
||||
'default_locale': str(DEFAULT_LOCALE),
|
||||
'i18n': config.get('ui.i18n') or {},
|
||||
'oauth': {
|
||||
# Hide providers (and thus the login buttons / auto-redirect) when OAuth
|
||||
# is disabled, without clearing the admin's provider configuration.
|
||||
|
|
@ -2294,6 +2325,7 @@ async def get_app_config(request: Request):
|
|||
'auto_redirect': config.get('oauth.auto_redirect'),
|
||||
},
|
||||
'features': {
|
||||
'slim': USE_SLIM,
|
||||
# --- Public: required by login/signup page pre-auth ---
|
||||
'auth': WEBUI_AUTH,
|
||||
'auth_trusted_header': bool(WEBUI_AUTH_TRUSTED_EMAIL_HEADER),
|
||||
|
|
@ -2317,6 +2349,7 @@ async def get_app_config(request: Request):
|
|||
'enable_public_active_users_count': ENABLE_PUBLIC_ACTIVE_USERS_COUNT,
|
||||
'enable_easter_eggs': ENABLE_EASTER_EGGS,
|
||||
'enable_direct_connections': config.get('direct.enable'),
|
||||
'enable_direct_integrations': config.get('direct.integrations.enable', False),
|
||||
'enable_plugins': ENABLE_PLUGINS,
|
||||
'enable_folders': config.get('folders.enable'),
|
||||
'folder_max_file_count': config.get('folders.max_file_count'),
|
||||
|
|
@ -2361,6 +2394,7 @@ async def get_app_config(request: Request):
|
|||
'default_models': config.get('ui.default_models'),
|
||||
'default_pinned_models': config.get('ui.default_pinned_models'),
|
||||
'default_prompt_suggestions': config.get('ui.prompt_suggestions'),
|
||||
'default_prompt_suggestions_i18n': config.get('ui.prompt_suggestions_i18n'),
|
||||
**({'user_count': user_count} if user_count is not None else {}),
|
||||
'code': {
|
||||
'engine': config.get('code_execution.engine'),
|
||||
|
|
@ -2575,8 +2609,8 @@ async def get_app_latest_release_version(user=Depends(get_verified_user)):
|
|||
|
||||
return {'current': VERSION, 'latest': latest_version[1:]}
|
||||
except Exception as e:
|
||||
log.debug(e)
|
||||
return {'current': VERSION, 'latest': VERSION}
|
||||
log.warning(f'Version update check failed: {e}')
|
||||
return {'current': VERSION, 'latest': None}
|
||||
|
||||
|
||||
@app.get('/api/changelog')
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ from typing import Optional
|
|||
import bcrypt
|
||||
from open_webui.internal.db import Base, JSONField, get_async_db_context
|
||||
from open_webui.models.users import User, UserModel, UserProfileImageResponse, Users
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import BaseModel, field_validator
|
||||
from sqlalchemy import Boolean, Column, String, Text, delete, select, update
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
|
@ -87,7 +87,7 @@ class SignupForm(BaseModel):
|
|||
@classmethod
|
||||
def check_profile_image_url(cls, v: str | None) -> str | None:
|
||||
if v is not None:
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
return v
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -312,6 +312,7 @@ class AutomationTable:
|
|||
rows = result.scalars().all()
|
||||
|
||||
from open_webui.utils.automations import next_run_ns
|
||||
from open_webui.utils.recurrence import RecurrenceEvaluationTimeout
|
||||
|
||||
# Batch-fetch user timezones so rescheduling respects each
|
||||
# user's local timezone instead of falling back to server time.
|
||||
|
|
@ -323,13 +324,20 @@ class AutomationTable:
|
|||
tz_result = await db.execute(select(User.id, User.timezone).where(User.id.in_(user_ids)))
|
||||
timezone_by_user_id = {uid: tz for uid, tz in tz_result.all()}
|
||||
|
||||
claimed = []
|
||||
for row in rows:
|
||||
try:
|
||||
next_run_at = await next_run_ns(row.data.get('rrule', ''), tz=timezone_by_user_id.get(row.user_id))
|
||||
except RecurrenceEvaluationTimeout:
|
||||
log.warning('Skipping automation %s: recurrence evaluation timed out', row.id)
|
||||
continue
|
||||
row.last_run_at = now_ns
|
||||
row.next_run_at = next_run_ns(row.data.get('rrule', ''), tz=timezone_by_user_id.get(row.user_id))
|
||||
row.next_run_at = next_run_at
|
||||
claimed.append(row)
|
||||
|
||||
await db.commit()
|
||||
|
||||
return [AutomationModel.model_validate(r) for r in rows]
|
||||
return [AutomationModel.model_validate(r) for r in claimed]
|
||||
|
||||
|
||||
####################
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ from open_webui.constants import ERROR_MESSAGES
|
|||
from open_webui.models.access_grants import AccessGrantModel, AccessGrants
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.users import User, UserModel, UserResponse
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
BigInteger,
|
||||
|
|
@ -179,6 +179,20 @@ class CalendarUpdateForm(BaseModel):
|
|||
access_grants: Optional[list[dict]] = None
|
||||
|
||||
|
||||
async def validate_calendar_rrule(value: Optional[str]) -> None:
|
||||
if value:
|
||||
from open_webui.utils.recurrence import rrule_interval_seconds
|
||||
|
||||
try:
|
||||
interval = await rrule_interval_seconds(value)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e)) from e
|
||||
if interval is not None and interval < MIN_CALENDAR_RRULE_INTERVAL_SECONDS:
|
||||
raise ValueError(ERROR_MESSAGES.CALENDAR_RRULE_TOO_FREQUENT)
|
||||
|
||||
|
||||
class CalendarEventForm(BaseModel):
|
||||
calendar_id: str
|
||||
title: str
|
||||
|
|
@ -193,22 +207,6 @@ class CalendarEventForm(BaseModel):
|
|||
meta: Optional[dict] = None
|
||||
attendees: Optional[list[dict]] = None
|
||||
|
||||
@field_validator('rrule')
|
||||
@classmethod
|
||||
def reject_sub_daily_rrule(cls, value: Optional[str]) -> Optional[str]:
|
||||
if value:
|
||||
from open_webui.utils.automations import rrule_interval_seconds
|
||||
|
||||
try:
|
||||
interval = rrule_interval_seconds(value)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if interval is not None and interval < MIN_CALENDAR_RRULE_INTERVAL_SECONDS:
|
||||
raise ValueError(ERROR_MESSAGES.CALENDAR_RRULE_TOO_FREQUENT)
|
||||
return value
|
||||
|
||||
|
||||
class CalendarEventUpdateForm(BaseModel):
|
||||
calendar_id: Optional[str] = None
|
||||
|
|
@ -225,22 +223,6 @@ class CalendarEventUpdateForm(BaseModel):
|
|||
is_cancelled: Optional[bool] = None
|
||||
attendees: Optional[list[dict]] = None
|
||||
|
||||
@field_validator('rrule')
|
||||
@classmethod
|
||||
def reject_sub_daily_rrule(cls, value: Optional[str]) -> Optional[str]:
|
||||
if value:
|
||||
from open_webui.utils.automations import rrule_interval_seconds
|
||||
|
||||
try:
|
||||
interval = rrule_interval_seconds(value)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if interval is not None and interval < MIN_CALENDAR_RRULE_INTERVAL_SECONDS:
|
||||
raise ValueError(ERROR_MESSAGES.CALENDAR_RRULE_TOO_FREQUENT)
|
||||
return value
|
||||
|
||||
|
||||
class RSVPForm(BaseModel):
|
||||
status: str # 'accepted' | 'declined' | 'tentative' | 'pending'
|
||||
|
|
@ -465,6 +447,7 @@ class CalendarEventTable:
|
|||
async def insert_new_event(
|
||||
self, user_id: str, form_data: CalendarEventForm, db: Optional[AsyncSession] = None
|
||||
) -> Optional[CalendarEventModel]:
|
||||
await validate_calendar_rrule(form_data.rrule)
|
||||
async with get_async_db_context(db) as db:
|
||||
now = int(time.time_ns())
|
||||
event = CalendarEvent(
|
||||
|
|
@ -695,6 +678,7 @@ class CalendarEventTable:
|
|||
async def update_event_by_id(
|
||||
self, id: str, form_data: CalendarEventUpdateForm, db: Optional[AsyncSession] = None
|
||||
) -> Optional[CalendarEventModel]:
|
||||
await validate_calendar_rrule(form_data.rrule)
|
||||
async with get_async_db_context(db) as db:
|
||||
result = await db.execute(select(CalendarEvent).filter(CalendarEvent.id == id))
|
||||
event = result.scalars().first()
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ from open_webui.models.access_grants import (
|
|||
)
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.users import User
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
|
|
@ -253,7 +253,7 @@ class ChannelWebhookForm(BaseModel):
|
|||
def check_profile_image_url(cls, v: Optional[str]) -> Optional[str]:
|
||||
if v is None:
|
||||
return v
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
|
||||
|
||||
class ChannelTable:
|
||||
|
|
|
|||
|
|
@ -258,6 +258,7 @@ class ChatForm(BaseModel):
|
|||
class ChatImportForm(ChatForm):
|
||||
meta: dict | None = {}
|
||||
pinned: bool | None = False
|
||||
archived: bool | None = False
|
||||
current_message_id: str | None = None
|
||||
created_at: int | None = None
|
||||
updated_at: int | None = None
|
||||
|
|
@ -267,11 +268,6 @@ class ChatsImportForm(BaseModel):
|
|||
chats: list[ChatImportForm]
|
||||
|
||||
|
||||
class ChatTitleMessagesForm(BaseModel):
|
||||
title: str
|
||||
messages: list[dict]
|
||||
|
||||
|
||||
class ChatTitleForm(BaseModel):
|
||||
title: str
|
||||
|
||||
|
|
@ -309,6 +305,7 @@ class ChatTitleIdResponse(BaseModel):
|
|||
last_read_at: int | None = None
|
||||
snippet: str | None = None
|
||||
active: bool = False
|
||||
archived: bool = False
|
||||
|
||||
|
||||
class SharedChatResponse(BaseModel):
|
||||
|
|
@ -649,6 +646,7 @@ class ChatTable:
|
|||
'meta': form_data.meta,
|
||||
'variables': form_data.variables or {},
|
||||
'pinned': form_data.pinned,
|
||||
'archived': form_data.archived,
|
||||
'folder_id': form_data.folder_id,
|
||||
'current_message_id': form_data.current_message_id or self.get_current_message_id(form_data.chat),
|
||||
'created_at': (form_data.created_at if form_data.created_at else int(time.time())),
|
||||
|
|
@ -1990,10 +1988,8 @@ class ChatTable:
|
|||
]
|
||||
|
||||
# Extract folder names
|
||||
folders = await Folders.search_folders_by_names(
|
||||
user_id,
|
||||
[word.replace('folder:', '') for word in search_text_words if word.startswith('folder:')],
|
||||
)
|
||||
folder_names = [word.replace('folder:', '') for word in search_text_words if word.startswith('folder:')]
|
||||
folders = await Folders.search_folders_by_names(user_id, folder_names)
|
||||
folder_ids = [folder.id for folder in folders]
|
||||
|
||||
is_pinned = None
|
||||
|
|
@ -2037,7 +2033,7 @@ class ChatTable:
|
|||
else:
|
||||
stmt = stmt.filter(Chat.share_id.is_(None))
|
||||
|
||||
if folder_ids:
|
||||
if folder_names:
|
||||
stmt = stmt.filter(Chat.folder_id.in_(folder_ids))
|
||||
|
||||
# Check if the database dialect is either 'sqlite' or 'postgresql'
|
||||
|
|
@ -2540,18 +2536,15 @@ class ChatTable:
|
|||
except Exception:
|
||||
return False
|
||||
|
||||
async def move_chats_by_user_id_and_folder_id(
|
||||
async def move_chats_by_folder_id(
|
||||
self,
|
||||
user_id: str,
|
||||
folder_id: str,
|
||||
new_folder_id: str | None,
|
||||
db: AsyncSession | None = None,
|
||||
) -> bool:
|
||||
try:
|
||||
async with get_async_db_context(db) as session:
|
||||
await session.execute(
|
||||
update(Chat).filter_by(user_id=user_id, folder_id=folder_id).values(folder_id=new_folder_id)
|
||||
)
|
||||
await session.execute(update(Chat).filter_by(folder_id=folder_id).values(folder_id=new_folder_id))
|
||||
await session.commit()
|
||||
|
||||
return True
|
||||
|
|
|
|||
|
|
@ -1069,6 +1069,26 @@ class KnowledgeTable:
|
|||
for child_id in child_ids:
|
||||
await self._delete_files_in_subtree(child_id, db=db)
|
||||
|
||||
async def get_files_by_id_and_directory_id(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
directory_id: str,
|
||||
db: Optional[AsyncSession] = None,
|
||||
) -> list[FileModel]:
|
||||
"""Get all files in a directory and its subdirectories."""
|
||||
async with get_async_db_context(db) as db:
|
||||
directory_ids = [directory_id]
|
||||
for parent_id in directory_ids:
|
||||
result = await db.execute(select(KnowledgeDirectory.id).filter_by(parent_id=parent_id))
|
||||
directory_ids.extend(result.scalars().all())
|
||||
result = await db.execute(
|
||||
select(File)
|
||||
.join(KnowledgeFile, File.id == KnowledgeFile.file_id)
|
||||
.filter(KnowledgeFile.knowledge_id == knowledge_id)
|
||||
.filter(KnowledgeFile.directory_id.in_(directory_ids))
|
||||
)
|
||||
return [FileModel.model_validate(file) for file in result.scalars().all()]
|
||||
|
||||
async def move_file_to_directory(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
|
|
|
|||
|
|
@ -10,17 +10,13 @@ from open_webui.models.access_grants import AccessGrantModel, AccessGrants
|
|||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.users import User, UserModel, UserResponse, Users
|
||||
from open_webui.utils.misc import json_text_variants
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator, model_validator
|
||||
from sqlalchemy import BigInteger, Boolean, Column, String, Text, cast, delete, func, or_, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Track invalid profile_image_url values we've already warned about so we
|
||||
# don't flood the logs on every DB read (the validator fires per-row).
|
||||
_warned_profile_urls: set[str] = set()
|
||||
|
||||
|
||||
def normalize_model_tags(tags: Any) -> list[dict[str, str]]:
|
||||
if not isinstance(tags, list):
|
||||
|
|
@ -79,26 +75,24 @@ class ModelMeta(BaseModel):
|
|||
"""Metadata for a workspace model entry (profile, description, tags, capabilities)."""
|
||||
|
||||
profile_image_url: str | None = None
|
||||
background_image_url: str | None = None
|
||||
description: str | None = Field(default=None, description='User-facing description of the model.')
|
||||
i18n: dict[str, Any] | None = None
|
||||
capabilities: dict | None = None
|
||||
knowledge: list[Any] | None = None
|
||||
|
||||
model_config = ConfigDict(extra='allow')
|
||||
|
||||
@field_validator('profile_image_url', mode='before')
|
||||
@field_validator('profile_image_url', 'background_image_url', mode='before')
|
||||
@classmethod
|
||||
def check_profile_image_url(cls, v: str | None) -> str | None:
|
||||
def check_image_url(cls, v: str | None, info: ValidationInfo) -> str | None:
|
||||
if v is None:
|
||||
return v
|
||||
try:
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v, file_only=info.field_name == 'background_image_url')
|
||||
except ValueError:
|
||||
if v not in _warned_profile_urls:
|
||||
_warned_profile_urls.add(v)
|
||||
log.warning(
|
||||
'Clearing invalid profile_image_url stored in DB (likely a legacy SVG data-URI): %.80s…',
|
||||
v,
|
||||
)
|
||||
if info.field_name == 'background_image_url':
|
||||
raise
|
||||
return None
|
||||
|
||||
@field_validator('knowledge', mode='before')
|
||||
|
|
@ -175,7 +169,7 @@ class ModelAccessListResponse(BaseModel):
|
|||
class ModelForm(BaseModel):
|
||||
model_config = ConfigDict(extra='ignore')
|
||||
|
||||
id: str
|
||||
id: str = Field(pattern=r'^\S+$')
|
||||
base_model_id: str | None = None
|
||||
name: str
|
||||
meta: ModelMeta
|
||||
|
|
@ -248,11 +242,14 @@ class ModelsTable:
|
|||
return models
|
||||
|
||||
async def get_models(
|
||||
self, writable_by_user_id: str | None = None, db: AsyncSession | None = None
|
||||
self, writable_by_user_id: str | None = None, db: AsyncSession | None = None, ids: list[str] | None = None
|
||||
) -> list[ModelUserResponse]:
|
||||
async with get_async_db_context(db) as db:
|
||||
stmt = select(Model).filter(Model.base_model_id != None)
|
||||
|
||||
if ids is not None:
|
||||
stmt = stmt.filter(Model.id.in_(ids))
|
||||
|
||||
if writable_by_user_id:
|
||||
user_group_ids = {
|
||||
group.id for group in await Groups.get_groups_by_member_id(writable_by_user_id, db=db)
|
||||
|
|
@ -290,13 +287,15 @@ class ModelsTable:
|
|||
)
|
||||
return models
|
||||
|
||||
async def get_model_owners_attaching_file(self, file_id: str, db: AsyncSession | None = None) -> dict[str, str]:
|
||||
"""Map of model id to owner id for workspace models whose knowledge attaches this file."""
|
||||
async def get_model_owner_ids_by_file_id(
|
||||
self, file_id: str, db: AsyncSession | None = None, include_background: bool = False
|
||||
) -> dict[str, str]:
|
||||
"""Return model IDs mapped to owner IDs for models referencing the file."""
|
||||
async with get_async_db_context(db) as db:
|
||||
# File ids are server-generated uuids, so the text match can only over-match.
|
||||
result = await db.execute(
|
||||
select(Model.id, Model.user_id, Model.meta).filter(
|
||||
Model.base_model_id.is_not(None), cast(Model.meta, String).like(f'%"{file_id}"%')
|
||||
Model.base_model_id.is_not(None), cast(Model.meta, String).like(f'%{file_id}%')
|
||||
)
|
||||
)
|
||||
return {
|
||||
|
|
@ -306,6 +305,7 @@ class ModelsTable:
|
|||
isinstance(item, dict) and item.get('type') == 'file' and item.get('id') == file_id
|
||||
for item in meta.get('knowledge') or []
|
||||
)
|
||||
or (include_background and meta.get('background_image_url') == f'/api/v1/files/{file_id}/content')
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -448,11 +448,13 @@ class ModelsTable:
|
|||
|
||||
return ModelListResponse(items=models, total=total)
|
||||
|
||||
async def get_model_meta_by_id(self, id: str, db: AsyncSession | None = None) -> tuple[dict, int | None]:
|
||||
"""Return (meta, updated_at) for a model, skipping access grant resolution."""
|
||||
async def get_model_meta_by_id(
|
||||
self, id: str, db: AsyncSession | None = None
|
||||
) -> tuple[dict, str, int | None] | None:
|
||||
"""Return (meta, user_id, updated_at) for a model, skipping access grant resolution."""
|
||||
try:
|
||||
async with get_async_db_context(db) as db:
|
||||
result = await db.execute(select(Model.meta, Model.updated_at).filter_by(id=id))
|
||||
result = await db.execute(select(Model.meta, Model.user_id, Model.updated_at).filter_by(id=id))
|
||||
return result.first()
|
||||
except Exception:
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -506,14 +506,16 @@ class PromptsTable:
|
|||
)
|
||||
|
||||
# Update prompt fields
|
||||
prompt.name = form_data.name
|
||||
prompt.command = form_data.command
|
||||
prompt.content = form_data.content
|
||||
prompt.data = form_data.data or prompt.data
|
||||
prompt.meta = form_data.meta or prompt.meta
|
||||
|
||||
if form_data.tags is not None:
|
||||
prompt.tags = form_data.tags
|
||||
if form_data.is_production:
|
||||
prompt.name = form_data.name
|
||||
prompt.content = form_data.content
|
||||
prompt.data = form_data.data or prompt.data
|
||||
prompt.meta = form_data.meta or prompt.meta
|
||||
|
||||
if form_data.tags is not None:
|
||||
prompt.tags = form_data.tags
|
||||
|
||||
if form_data.access_grants is not None:
|
||||
await AccessGrants.set_access_grants('prompt', prompt.id, form_data.access_grants, db=session)
|
||||
|
|
@ -531,7 +533,7 @@ class PromptsTable:
|
|||
'command': prompt.command,
|
||||
'data': form_data.data or {},
|
||||
'meta': form_data.meta or {},
|
||||
'tags': prompt.tags or [],
|
||||
'tags': form_data.tags if form_data.tags is not None else (prompt.tags or []),
|
||||
'access_grants': [grant.model_dump() for grant in current_access_grants],
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ class Skill(Base):
|
|||
|
||||
|
||||
class SkillMeta(BaseModel):
|
||||
i18n: dict[str, dict[str, str]] | None = None
|
||||
tags: Optional[list[str]] = []
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ class Tool(Base): # database table definition
|
|||
|
||||
|
||||
class ToolMeta(BaseModel):
|
||||
i18n: dict[str, dict[str, str]] | None = None
|
||||
description: str | None = None
|
||||
manifest: dict | None = {}
|
||||
has_user_valves: bool = False
|
||||
|
|
|
|||
|
|
@ -4,12 +4,18 @@ from __future__ import annotations
|
|||
|
||||
import datetime
|
||||
import time
|
||||
from typing import Optional
|
||||
from typing import Literal, Optional
|
||||
from open_webui.env import DATABASE_USER_ACTIVE_STATUS_UPDATE_INTERVAL
|
||||
from open_webui.internal.db import Base, JSONField, get_async_db_context
|
||||
from open_webui.utils.misc import throttle
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import (
|
||||
BaseModel,
|
||||
ConfigDict,
|
||||
Field,
|
||||
field_validator,
|
||||
model_validator,
|
||||
)
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
BigInteger,
|
||||
|
|
@ -36,6 +42,97 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||
####################
|
||||
|
||||
|
||||
class InterfaceTitleSettings(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
auto: bool | None = None
|
||||
|
||||
|
||||
class InterfaceImageCompressionSize(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
width: int | float | Literal[''] | None = None
|
||||
height: int | float | Literal[''] | None = None
|
||||
|
||||
|
||||
class InterfaceFloatingActionButton(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
id: str
|
||||
label: str
|
||||
input: bool
|
||||
prompt: str
|
||||
|
||||
|
||||
class InterfaceSettings(BaseModel):
|
||||
"""Fields owned by the Interface settings panel; not the entire user UI dict."""
|
||||
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
autoTags: bool | None = None
|
||||
autoFollowUps: bool | None = None
|
||||
highContrastMode: bool | None = None
|
||||
detectArtifacts: bool | None = None
|
||||
responseAutoCopy: bool | None = None
|
||||
showUsername: bool | None = None
|
||||
showUpdateToast: bool | None = None
|
||||
showChangelog: bool | None = None
|
||||
showEmojiInCall: bool | None = None
|
||||
voiceInterruption: bool | None = None
|
||||
displayMultiModelResponsesInTabs: bool | None = None
|
||||
chatFadeStreamingText: bool | None = None
|
||||
richTextInput: bool | None = None
|
||||
showFormattingToolbar: bool | None = None
|
||||
insertPromptAsRichText: bool | None = None
|
||||
promptAutocomplete: bool | None = None
|
||||
insertSuggestionPrompt: bool | None = None
|
||||
keepFollowUpPrompts: bool | None = None
|
||||
insertFollowUpPrompt: bool | None = None
|
||||
regenerateMenu: bool | None = None
|
||||
enableMessageQueue: bool | None = None
|
||||
largeTextAsFile: bool | None = None
|
||||
copyFormatted: bool | None = None
|
||||
collapseCodeBlocks: bool | None = None
|
||||
renderMarkdownInUserMessages: bool | None = None
|
||||
renderMarkdownInAssistantMessages: bool | None = None
|
||||
expandDetails: bool | None = None
|
||||
chatHoverPreview: bool | None = None
|
||||
renderMarkdownInPreviews: bool | None = None
|
||||
chatBubble: bool | None = None
|
||||
widescreenMode: bool | None = None
|
||||
splitLargeChunks: bool | None = None
|
||||
scrollOnBranchChange: bool | None = None
|
||||
scrollOnResponseGeneration: bool | None = None
|
||||
showFilesOnTerminalSelect: bool | None = None
|
||||
temporaryChatByDefault: bool | None = None
|
||||
userLocation: bool | None = None
|
||||
showChatTitleInTab: bool | None = None
|
||||
iframeSandboxAllowScripts: bool | None = None
|
||||
iframeSandboxAllowSameOrigin: bool | None = None
|
||||
iframeSandboxAllowForms: bool | None = None
|
||||
iframeSandboxAllowDownloads: bool | None = None
|
||||
terminalPreviewAllowSameOrigin: bool | None = None
|
||||
stylizedPdfExport: bool | None = None
|
||||
hapticFeedback: bool | None = None
|
||||
ctrlEnterToSend: bool | None = None
|
||||
showFloatingActionButtons: bool | None = None
|
||||
imageCompression: bool | None = None
|
||||
imageCompressionInChannels: bool | None = None
|
||||
|
||||
landingPageMode: Literal['', 'chat'] | None = None
|
||||
chatDirection: Literal['LTR', 'RTL', 'auto'] | None = None
|
||||
terminalFileDisplay: Literal['sidebar', 'inline'] | None = None
|
||||
defaultUploadContext: Literal['full', 'focused'] | None = None
|
||||
webSearch: Literal['always'] | None = None
|
||||
models: list[str] | None = None
|
||||
backgroundImageUrl: str | None = None
|
||||
fontFamily: str | None = None
|
||||
textScale: float | None = None
|
||||
title: InterfaceTitleSettings | None = None
|
||||
imageCompressionSize: InterfaceImageCompressionSize | None = None
|
||||
floatingActionButtons: list[InterfaceFloatingActionButton] | None = None
|
||||
|
||||
|
||||
class UserSettings(BaseModel):
|
||||
ui: dict | None = {}
|
||||
model_config = ConfigDict(extra='allow')
|
||||
|
|
@ -180,7 +277,7 @@ class UpdateProfileForm(BaseModel):
|
|||
@field_validator('profile_image_url')
|
||||
@classmethod
|
||||
def check_profile_image_url(cls, v: str) -> str:
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
|
||||
|
||||
class UserGroupIdsModel(UserModel):
|
||||
|
|
@ -270,7 +367,7 @@ class UserUpdateForm(BaseModel):
|
|||
def check_profile_image_url(cls, v: str | None) -> str | None:
|
||||
if v is None:
|
||||
return v
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
|
||||
|
||||
class UsersTable:
|
||||
|
|
@ -286,7 +383,7 @@ class UsersTable:
|
|||
db: AsyncSession | None = None,
|
||||
) -> UserModel | None:
|
||||
try:
|
||||
profile_image_url = validate_profile_image_url(profile_image_url)
|
||||
profile_image_url = validate_image_url(profile_image_url)
|
||||
except ValueError:
|
||||
profile_image_url = '/user.png'
|
||||
|
||||
|
|
@ -657,7 +754,7 @@ class UsersTable:
|
|||
db: AsyncSession | None = None,
|
||||
) -> UserModel | None:
|
||||
try:
|
||||
profile_image_url = validate_profile_image_url(profile_image_url)
|
||||
profile_image_url = validate_image_url(profile_image_url)
|
||||
except ValueError:
|
||||
profile_image_url = '/user.png'
|
||||
|
||||
|
|
@ -706,7 +803,9 @@ class UsersTable:
|
|||
return None
|
||||
scim = dict(user.scim or {})
|
||||
scim[provider] = {'external_id': external_id}
|
||||
user.scim = scim
|
||||
if scim != user.scim:
|
||||
user.scim = scim
|
||||
user.updated_at = int(time.time())
|
||||
await session.commit()
|
||||
return UserModel.model_validate(user)
|
||||
|
||||
|
|
@ -729,7 +828,18 @@ class UsersTable:
|
|||
if not user:
|
||||
return None
|
||||
user_settings = dict(user.settings or {})
|
||||
updated = dict(updated)
|
||||
ui_settings = updated.pop('ui', None)
|
||||
user_settings.update(updated)
|
||||
if ui_settings is not None:
|
||||
# UI updates are field-level patches: omission keeps a value; null resets it.
|
||||
current_ui_settings = dict(user_settings.get('ui') or {})
|
||||
for key, value in ui_settings.items():
|
||||
if value is None:
|
||||
current_ui_settings.pop(key, None)
|
||||
else:
|
||||
current_ui_settings[key] = value
|
||||
user_settings['ui'] = current_ui_settings
|
||||
user.settings = user_settings
|
||||
await session.commit()
|
||||
return UserModel.model_validate(user)
|
||||
|
|
|
|||
121
backend/open_webui/retrieval/loaders/local.py
Normal file
121
backend/open_webui/retrieval/loaders/local.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
from langchain_core.documents import Document
|
||||
|
||||
|
||||
class TextLoader:
|
||||
def __init__(self, file_path, encoding=None):
|
||||
self.file_path = str(file_path)
|
||||
self.encoding = encoding
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
try:
|
||||
text = Path(self.file_path).read_text(encoding=self.encoding)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Error loading {self.file_path}') from e
|
||||
return [
|
||||
Document(
|
||||
page_content=text,
|
||||
metadata={'source': self.file_path},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class HTMLLoader(TextLoader):
|
||||
def load(self) -> list[Document]:
|
||||
with open(self.file_path, encoding=self.encoding) as file:
|
||||
soup = BeautifulSoup(file, 'lxml')
|
||||
return [
|
||||
Document(
|
||||
page_content=soup.get_text(),
|
||||
metadata={'source': self.file_path, 'title': str(soup.title.string) if soup.title else ''},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class DocxLoader(TextLoader):
|
||||
def load(self) -> list[Document]:
|
||||
import docx2txt
|
||||
|
||||
return [
|
||||
Document(
|
||||
page_content=docx2txt.process(Path(self.file_path).expanduser()),
|
||||
metadata={'source': self.file_path},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class UnstructuredLoader:
|
||||
def __init__(self, file_path, file_format, mode='single', **kwargs):
|
||||
# Match the optional-package check; format dependencies are loaded when parsing.
|
||||
import_module('unstructured')
|
||||
self.file_path = file_path
|
||||
self.file_format = file_format
|
||||
self.mode = mode
|
||||
self.kwargs = kwargs
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
file_format = self.file_format
|
||||
if file_format in ('doc', 'ppt', 'pptx'):
|
||||
from unstructured.file_utils.filetype import detect_filetype
|
||||
|
||||
legacy_format = 'doc' if file_format == 'doc' else 'ppt'
|
||||
try:
|
||||
import_module('magic')
|
||||
except ImportError:
|
||||
is_legacy = Path(self.file_path).suffix == f'.{legacy_format}'
|
||||
else:
|
||||
is_legacy = detect_filetype(self.file_path).name.lower() == legacy_format
|
||||
file_format = legacy_format if is_legacy else legacy_format + 'x'
|
||||
elif file_format == 'msg':
|
||||
from unstructured.file_utils.filetype import detect_filetype
|
||||
|
||||
detected = detect_filetype(self.file_path).name
|
||||
if detected not in ('EML', 'MSG'):
|
||||
raise ValueError(f'Unsupported email file type: {detected}')
|
||||
file_format = 'email' if detected == 'EML' else 'msg'
|
||||
|
||||
module = import_module(f'unstructured.partition.{file_format}')
|
||||
elements = getattr(module, f'partition_{file_format}')(filename=self.file_path, **self.kwargs)
|
||||
metadata = {'source': str(self.file_path)}
|
||||
if self.mode == 'elements':
|
||||
return [
|
||||
Document(
|
||||
page_content=str(element),
|
||||
metadata={
|
||||
**metadata,
|
||||
**element.metadata.to_dict(),
|
||||
'category': element.category,
|
||||
'element_id': element.id,
|
||||
},
|
||||
)
|
||||
for element in elements
|
||||
]
|
||||
return [Document(page_content='\n\n'.join(map(str, elements)), metadata=metadata)]
|
||||
|
||||
|
||||
class DocumentIntelligenceLoader:
|
||||
def __init__(self, file_path, api_endpoint, api_key=None, azure_credential=None, api_model='prebuilt-layout'):
|
||||
if (api_key is None) == (azure_credential is None):
|
||||
raise ValueError('Provide exactly one of api_key or azure_credential.')
|
||||
self.file_path = file_path
|
||||
self.api_endpoint = api_endpoint
|
||||
self.api_key = api_key
|
||||
self.azure_credential = azure_credential
|
||||
self.api_model = api_model
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
from azure.ai.documentintelligence import DocumentIntelligenceClient
|
||||
from azure.core.credentials import AzureKeyCredential
|
||||
|
||||
credential = self.azure_credential if self.azure_credential is not None else AzureKeyCredential(self.api_key)
|
||||
with DocumentIntelligenceClient(self.api_endpoint, credential) as client, open(self.file_path, 'rb') as file:
|
||||
result = client.begin_analyze_document(
|
||||
self.api_model,
|
||||
body=file,
|
||||
content_type='application/octet-stream',
|
||||
output_content_format='markdown',
|
||||
).result()
|
||||
return [Document(page_content=result.content, metadata=result.as_dict())]
|
||||
|
|
@ -7,27 +7,29 @@ import zipfile
|
|||
|
||||
import ftfy
|
||||
import requests
|
||||
from fastapi import HTTPException
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from langchain_community.document_loaders import (
|
||||
AzureAIDocumentIntelligenceLoader,
|
||||
BSHTMLLoader,
|
||||
CSVLoader,
|
||||
Docx2txtLoader,
|
||||
PyPDFLoader,
|
||||
TextLoader,
|
||||
)
|
||||
from langchain_core.documents import Document
|
||||
from open_webui.env import (
|
||||
AIOHTTP_CLIENT_SESSION_SSL,
|
||||
GLOBAL_LOG_LEVEL,
|
||||
USE_SLIM,
|
||||
MINERU_MAX_MARKDOWN_BYTES,
|
||||
REQUESTS_VERIFY,
|
||||
)
|
||||
from open_webui.retrieval.loaders.datalab_marker import DatalabMarkerLoader
|
||||
from open_webui.retrieval.loaders.external_document import ExternalDocumentLoader
|
||||
from open_webui.retrieval.loaders.local import (
|
||||
DocumentIntelligenceLoader,
|
||||
DocxLoader,
|
||||
HTMLLoader,
|
||||
TextLoader,
|
||||
UnstructuredLoader,
|
||||
)
|
||||
from open_webui.retrieval.loaders.mineru import MinerULoader
|
||||
from open_webui.retrieval.loaders.mistral import MistralLoader
|
||||
from open_webui.retrieval.loaders.paddleocr_vl import PADDLEOCR_VL_SUPPORTED_EXTENSIONS, PaddleOCRVLLoader
|
||||
from open_webui.retrieval.loaders.pdf import PDFLoader
|
||||
from open_webui.utils.headers import get_user_groups_for_custom_headers
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
|
||||
|
|
@ -50,6 +52,7 @@ known_source_ext = [
|
|||
'h',
|
||||
'c',
|
||||
'cs',
|
||||
'ino',
|
||||
'sql',
|
||||
'log',
|
||||
'ini',
|
||||
|
|
@ -88,6 +91,7 @@ known_source_ext = [
|
|||
'yaml',
|
||||
'yml',
|
||||
'toml',
|
||||
'svg',
|
||||
]
|
||||
|
||||
known_archive_ext = {'docx', 'epub', 'odt', 'pptx', 'xlsx'}
|
||||
|
|
@ -160,7 +164,21 @@ class CSVLoaderWithSummary:
|
|||
self.encoding = encoding
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
docs = CSVLoader(self.file_path, encoding=self.encoding).load()
|
||||
docs = []
|
||||
try:
|
||||
with open(self.file_path, newline='', encoding=self.encoding) as file:
|
||||
for index, row in enumerate(csv.DictReader(file)):
|
||||
fields = []
|
||||
for key, value in row.items():
|
||||
if isinstance(value, str):
|
||||
value = value.strip()
|
||||
elif isinstance(value, list):
|
||||
value = ','.join(v.strip() for v in value)
|
||||
fields.append(f'{key.strip() if key is not None else key}: {value}')
|
||||
content = '\n'.join(fields)
|
||||
docs.append(Document(page_content=content, metadata={'source': self.file_path, 'row': index}))
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Error loading {self.file_path}') from e
|
||||
if os.getenv('ENABLE_RAG_CSV_SUMMARY', 'False').lower() == 'true':
|
||||
summary = get_csv_summary(self.filename, self.file_path, self.encoding)
|
||||
if summary:
|
||||
|
|
@ -215,7 +233,7 @@ class TikaLoader:
|
|||
if self.extract_images == True:
|
||||
headers['X-Tika-PDFextractInlineImages'] = 'true'
|
||||
|
||||
endpoint_path = 'tika/json/text' if self.server_version == '4' else 'tika/text'
|
||||
endpoint_path = 'tika/json/md' if self.server_version == '4' else 'tika/text'
|
||||
content_key = 'tk:content' if self.server_version == '4' else 'X-TIKA:content'
|
||||
endpoint = f'{self.url.rstrip("/")}/{endpoint_path}'
|
||||
|
||||
|
|
@ -273,8 +291,17 @@ class DoclingLoader:
|
|||
)
|
||||
if r.ok:
|
||||
result = r.json()
|
||||
# Docling reports failed and skipped conversions inside HTTP 200 responses.
|
||||
conversion_status = result.get('status')
|
||||
if conversion_status in ['failure', 'skipped']:
|
||||
error_details = (
|
||||
'; '.join(filter(None, (error.get('error_message') for error in result.get('errors', []))))
|
||||
or 'no error message provided'
|
||||
)
|
||||
raise Exception(f'Error calling Docling: conversion status {conversion_status} - {error_details}')
|
||||
|
||||
document_data = result.get('document', {})
|
||||
md_content = document_data.get('md_content', '')
|
||||
md_content = document_data.get('md_content') or ''
|
||||
text = md_content or '<No text content found>'
|
||||
|
||||
metadata = {'Content-Type': self.mime_type} if self.mime_type else {}
|
||||
|
|
@ -313,7 +340,11 @@ class Loader:
|
|||
def load(self, filename: str, file_content_type: str, file_path: str) -> list[Document]:
|
||||
loader = self._get_loader(filename, file_content_type, file_path)
|
||||
docs = loader.load()
|
||||
return [Document(page_content=ftfy.fix_text(doc.page_content), metadata=doc.metadata) for doc in docs]
|
||||
# ftfy's auto mode unescapes entities on every line before the first literal '<', rewriting the document.
|
||||
return [
|
||||
Document(page_content=ftfy.fix_text(doc.page_content, unescape_html=False), metadata=doc.metadata)
|
||||
for doc in docs
|
||||
]
|
||||
|
||||
async def aload(self, filename: str, file_content_type: str, file_path: str) -> list[Document]:
|
||||
"""
|
||||
|
|
@ -613,14 +644,14 @@ class Loader:
|
|||
)
|
||||
):
|
||||
if self.kwargs.get('DOCUMENT_INTELLIGENCE_KEY') != '':
|
||||
loader = AzureAIDocumentIntelligenceLoader(
|
||||
loader = DocumentIntelligenceLoader(
|
||||
file_path=file_path,
|
||||
api_endpoint=self.kwargs.get('DOCUMENT_INTELLIGENCE_ENDPOINT'),
|
||||
api_key=self.kwargs.get('DOCUMENT_INTELLIGENCE_KEY'),
|
||||
api_model=self.kwargs.get('DOCUMENT_INTELLIGENCE_MODEL'),
|
||||
)
|
||||
else:
|
||||
loader = AzureAIDocumentIntelligenceLoader(
|
||||
loader = DocumentIntelligenceLoader(
|
||||
file_path=file_path,
|
||||
api_endpoint=self.kwargs.get('DOCUMENT_INTELLIGENCE_ENDPOINT'),
|
||||
azure_credential=DefaultAzureCredential(),
|
||||
|
|
@ -666,8 +697,21 @@ class Loader:
|
|||
file_path=file_path,
|
||||
)
|
||||
else:
|
||||
if USE_SLIM:
|
||||
if file_ext == 'csv':
|
||||
return CSVLoaderWithSummary(file_path, filename, self._detect_text_encoding(file_path))
|
||||
if file_ext in ['htm', 'html']:
|
||||
return HTMLLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
if file_ext in ['txt', 'md', 'markdown', 'rst', 'xml'] or self._is_text_file(
|
||||
file_ext, file_content_type
|
||||
):
|
||||
return TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
raise HTTPException(
|
||||
503,
|
||||
'This file type requires an external document extractor in slim. Configure one that supports it.',
|
||||
)
|
||||
if file_ext == 'pdf':
|
||||
loader = PyPDFLoader(
|
||||
loader = PDFLoader(
|
||||
file_path,
|
||||
extract_images=self.kwargs.get('PDF_EXTRACT_IMAGES'),
|
||||
mode=self.kwargs.get('PDF_LOADER_MODE', 'page'),
|
||||
|
|
@ -680,9 +724,7 @@ class Loader:
|
|||
)
|
||||
elif file_ext == 'rst':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredRSTLoader
|
||||
|
||||
loader = UnstructuredRSTLoader(file_path, mode='elements')
|
||||
loader = UnstructuredLoader(file_path, 'rst', mode='elements')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -692,9 +734,7 @@ class Loader:
|
|||
loader = TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
elif file_ext == 'xml':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredXMLLoader
|
||||
|
||||
loader = UnstructuredXMLLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'xml')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -703,14 +743,12 @@ class Loader:
|
|||
)
|
||||
loader = TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
elif file_ext in ['htm', 'html']:
|
||||
loader = BSHTMLLoader(file_path, open_encoding='unicode_escape')
|
||||
loader = HTMLLoader(file_path, encoding='unicode_escape')
|
||||
elif file_ext == 'md':
|
||||
loader = TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
elif file_content_type == 'application/epub+zip':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredEPubLoader
|
||||
|
||||
loader = UnstructuredEPubLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'epub')
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .epub files requires the 'unstructured' package. "
|
||||
|
|
@ -720,12 +758,10 @@ class Loader:
|
|||
file_content_type == 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
|
||||
or file_ext == 'docx'
|
||||
):
|
||||
loader = Docx2txtLoader(file_path)
|
||||
loader = DocxLoader(file_path)
|
||||
elif file_ext == 'doc' or file_content_type == 'application/msword':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredWordDocumentLoader
|
||||
|
||||
loader = UnstructuredWordDocumentLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'doc')
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .doc files requires the 'unstructured' package. "
|
||||
|
|
@ -736,9 +772,7 @@ class Loader:
|
|||
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
|
||||
] or file_ext in ['xls', 'xlsx']:
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredExcelLoader
|
||||
|
||||
loader = UnstructuredExcelLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'xlsx')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -751,9 +785,7 @@ class Loader:
|
|||
'application/vnd.openxmlformats-officedocument.presentationml.presentation',
|
||||
] or file_ext in ['ppt', 'pptx']:
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredPowerPointLoader
|
||||
|
||||
loader = UnstructuredPowerPointLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'ppt' if file_ext == 'ppt' else 'pptx')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -763,12 +795,8 @@ class Loader:
|
|||
loader = PptxLoader(file_path)
|
||||
elif file_ext == 'msg':
|
||||
try:
|
||||
from langchain_community.document_loaders import (
|
||||
UnstructuredEmailLoader,
|
||||
)
|
||||
|
||||
# unstructured parses .msg via python-oxmsg; avoids extract_msg's beautifulsoup4<4.14 conflict
|
||||
loader = UnstructuredEmailLoader(file_path, process_attachments=False)
|
||||
loader = UnstructuredLoader(file_path, 'msg', process_attachments=False)
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .msg files requires the 'unstructured' package. "
|
||||
|
|
@ -776,9 +804,7 @@ class Loader:
|
|||
)
|
||||
elif file_ext == 'odt':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredODTLoader
|
||||
|
||||
loader = UnstructuredODTLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'odt')
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .odt files requires the 'unstructured' package. "
|
||||
|
|
|
|||
101
backend/open_webui/retrieval/loaders/pdf.py
Normal file
101
backend/open_webui/retrieval/loaders/pdf.py
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
import datetime as dt
|
||||
import io
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from langchain_core.document_loaders import BaseLoader
|
||||
from langchain_core.documents import Document
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PDFLoader(BaseLoader):
|
||||
def __init__(self, file_path, *, extract_images=False, mode='page'):
|
||||
if mode not in ('single', 'page'):
|
||||
raise ValueError("PDF mode must be 'single' or 'page'")
|
||||
self.file_path = str(Path(file_path).expanduser())
|
||||
self.extract_images = extract_images
|
||||
self.mode = mode
|
||||
self.ocr = None
|
||||
|
||||
def lazy_load(self):
|
||||
from pypdf import PdfReader
|
||||
|
||||
with open(self.file_path, 'rb') as file:
|
||||
reader = PdfReader(file)
|
||||
metadata = {'producer': 'PyPDF', 'creator': 'PyPDF', 'creationdate': ''}
|
||||
for key, value in (reader.metadata or {}).items():
|
||||
key = key.removeprefix('/').lower()
|
||||
value = value if type(value) in (str, int) else str(value)
|
||||
if key in ('creationdate', 'moddate') and isinstance(value, str):
|
||||
try:
|
||||
value = dt.datetime.strptime(value.replace("'", ''), 'D:%Y%m%d%H%M%S%z').isoformat()
|
||||
except ValueError:
|
||||
pass
|
||||
metadata[key] = (
|
||||
value.strip()
|
||||
if isinstance(value, str) and key not in ('creationdate', 'moddate', 'page_count', 'file_path')
|
||||
else value
|
||||
)
|
||||
metadata.update(source=self.file_path, total_pages=len(reader.pages))
|
||||
labels = reader.page_labels if self.mode == 'page' else None
|
||||
texts = []
|
||||
for index, page in enumerate(reader.pages):
|
||||
text = page.extract_text()
|
||||
if self.extract_images:
|
||||
image_text = self._extract_images(page)
|
||||
if image_text:
|
||||
text = self._merge_image_text(text, image_text)
|
||||
text = text.strip()
|
||||
if self.mode == 'page':
|
||||
yield Document(page_content=text, metadata={**metadata, 'page': index, 'page_label': labels[index]})
|
||||
else:
|
||||
texts.append(text)
|
||||
if self.mode == 'single':
|
||||
yield Document(page_content='\n\f'.join(texts), metadata=metadata)
|
||||
|
||||
@staticmethod
|
||||
def _merge_image_text(text, image_text):
|
||||
# Insert before the final paragraphs/footer where possible, matching existing chunks.
|
||||
position, separator = len(text), '\n\n'
|
||||
for _ in range(2):
|
||||
for delimiter in ('\n\n\n', '\n\n'):
|
||||
found = text.rfind(delimiter, 0, position)
|
||||
if found >= 0:
|
||||
position, separator = found, delimiter
|
||||
break
|
||||
else:
|
||||
break
|
||||
return text[:position] + separator + image_text + text[position:]
|
||||
|
||||
def _extract_images(self, page):
|
||||
import numpy as np
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
if '/Resources' not in page or '/XObject' not in page['/Resources']:
|
||||
return ''
|
||||
texts = []
|
||||
xobjects = page['/Resources']['/XObject']
|
||||
for name in xobjects:
|
||||
try:
|
||||
stream = xobjects[name]
|
||||
if stream.get('/Subtype') != '/Image':
|
||||
continue
|
||||
try:
|
||||
# Encoded images, including CMYK JPEGs, can go straight to Pillow.
|
||||
image = Image.open(io.BytesIO(stream.get_data()))
|
||||
except UnidentifiedImageError:
|
||||
image = stream.decode_as_image()
|
||||
pixels = np.array(image.convert('RGB'))
|
||||
except Exception as e:
|
||||
log.warning('Skipping unreadable PDF image %s: %s', name, e)
|
||||
continue
|
||||
|
||||
if self.ocr is None:
|
||||
from rapidocr import RapidOCR
|
||||
|
||||
self.ocr = RapidOCR()
|
||||
result = self.ocr(pixels)
|
||||
if result and result.txts:
|
||||
texts.append('\n'.join(result.txts).strip())
|
||||
return '\n\n' + '\n'.join(filter(None, texts)) + '\n\n' if any(texts) else ''
|
||||
|
|
@ -10,13 +10,13 @@ from typing import Awaitable, Optional, Union
|
|||
from urllib.parse import quote
|
||||
|
||||
import aiohttp
|
||||
import numpy as np
|
||||
import requests
|
||||
from huggingface_hub import snapshot_download
|
||||
from fastapi import HTTPException
|
||||
from langchain_classic.retrievers import (
|
||||
ContextualCompressionRetriever,
|
||||
EnsembleRetriever,
|
||||
)
|
||||
from langchain_community.retrievers import BM25Retriever
|
||||
from langchain_core.documents import Document
|
||||
from open_webui.config import (
|
||||
RAG_EMBEDDING_CONTENT_PREFIX,
|
||||
|
|
@ -32,7 +32,10 @@ from open_webui.env import (
|
|||
BYPASS_RETRIEVAL_ACCESS_CONTROL,
|
||||
ENABLE_FORWARD_USER_INFO_HEADERS,
|
||||
ENABLE_RETRIEVAL_UNSCOPED_COLLECTIONS,
|
||||
MPS_INFERENCE_LOCK,
|
||||
OFFLINE_MODE,
|
||||
RAG_SOURCE_METADATA_KEYS,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.models.access_grants import AccessGrants
|
||||
from open_webui.models.chats import Chats
|
||||
|
|
@ -45,7 +48,7 @@ from open_webui.models.users import UserModel
|
|||
from open_webui.retrieval.loaders.youtube import YoutubeLoader
|
||||
from open_webui.retrieval.vector.async_client import ASYNC_VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.external import retrieve_external_knowledge
|
||||
from open_webui.retrieval.vector.factory import VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import get_vector_db_client
|
||||
from open_webui.retrieval.vector.main import GetResult, SearchResult
|
||||
from open_webui.retrieval.web.utils import get_web_loader
|
||||
from open_webui.utils.access_control.files import get_owner_accessible_folder_files, has_access_to_file
|
||||
|
|
@ -62,6 +65,15 @@ from langchain_core.callbacks import CallbackManagerForRetrieverRun
|
|||
from langchain_core.retrievers import BaseRetriever
|
||||
|
||||
|
||||
class BM25Retriever(BaseRetriever):
|
||||
docs: list[Document]
|
||||
vectorizer: Any
|
||||
k: int
|
||||
|
||||
def _get_relevant_documents(self, query: str, *, run_manager: CallbackManagerForRetrieverRun) -> list[Document]:
|
||||
return self.vectorizer.get_top_n(query.split(), self.docs, n=self.k)
|
||||
|
||||
|
||||
def is_youtube_url(url: str) -> bool:
|
||||
youtube_regex = r'^(https?://)?(www\.)?(youtube\.com|youtu\.be)/.+$'
|
||||
return re.match(youtube_regex, url) is not None
|
||||
|
|
@ -330,27 +342,23 @@ class VectorSearchRetriever(BaseRetriever):
|
|||
|
||||
|
||||
def query_doc(collection_name: str, query_embedding: list[float], k: int, user: UserModel = None):
|
||||
try:
|
||||
log.debug('query_doc:doc %s', collection_name)
|
||||
result = VECTOR_DB_CLIENT.search(
|
||||
collection_name=collection_name,
|
||||
vectors=[query_embedding],
|
||||
limit=k,
|
||||
)
|
||||
log.debug('query_doc:doc %s', collection_name)
|
||||
result = get_vector_db_client().search(
|
||||
collection_name=collection_name,
|
||||
vectors=[query_embedding],
|
||||
limit=k,
|
||||
)
|
||||
|
||||
if result:
|
||||
log.info('query_doc:result %s %s', result.ids, result.metadatas)
|
||||
if result:
|
||||
log.info('query_doc:result %s %s', result.ids, result.metadatas)
|
||||
|
||||
return result
|
||||
except Exception as e:
|
||||
log.exception(f'Error querying doc {collection_name} with limit {k}: {e}')
|
||||
raise e
|
||||
return result
|
||||
|
||||
|
||||
def get_doc(collection_name: str, user: UserModel = None):
|
||||
try:
|
||||
log.debug('get_doc:doc %s', collection_name)
|
||||
result = VECTOR_DB_CLIENT.get(collection_name=collection_name)
|
||||
result = get_vector_db_client().get(collection_name=collection_name)
|
||||
|
||||
if result:
|
||||
log.info('query_doc:result %s %s', result.ids, result.metadatas)
|
||||
|
|
@ -495,122 +503,116 @@ async def query_doc_with_hybrid_search(
|
|||
enable_enriched_texts: bool = False,
|
||||
native_hybrid_search: bool = True,
|
||||
) -> dict:
|
||||
try:
|
||||
if native_hybrid_search and not enable_enriched_texts:
|
||||
native_result = await query_doc_with_native_hybrid_search(
|
||||
collection_name=collection_name,
|
||||
query=query,
|
||||
embedding_function=embedding_function,
|
||||
k=k,
|
||||
reranking_function=reranking_function,
|
||||
k_reranker=k_reranker,
|
||||
r=r,
|
||||
hybrid_bm25_weight=hybrid_bm25_weight,
|
||||
)
|
||||
if native_result is not None:
|
||||
return native_result
|
||||
|
||||
if collection_result is None:
|
||||
collection_result = await ASYNC_VECTOR_DB_CLIENT.get(collection_name=collection_name)
|
||||
|
||||
# First check if collection_result has the required attributes
|
||||
if (
|
||||
not collection_result
|
||||
or not hasattr(collection_result, 'documents')
|
||||
or not hasattr(collection_result, 'metadatas')
|
||||
):
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
# Now safely check the documents content after confirming attributes exist
|
||||
if (
|
||||
not collection_result.documents
|
||||
or len(collection_result.documents) == 0
|
||||
or not collection_result.documents[0]
|
||||
):
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
log.debug('query_doc_with_hybrid_search:doc %s', collection_name)
|
||||
|
||||
original_texts = collection_result.documents[0]
|
||||
bm25_metadatas = [
|
||||
{**meta, CHUNK_HASH_KEY: _content_hash(original_texts[idx])}
|
||||
for idx, meta in enumerate(collection_result.metadatas[0])
|
||||
]
|
||||
|
||||
bm25_texts = get_enriched_texts(collection_result) if enable_enriched_texts else original_texts
|
||||
|
||||
bm25_retriever = BM25Retriever.from_texts(
|
||||
texts=bm25_texts,
|
||||
metadatas=bm25_metadatas,
|
||||
)
|
||||
bm25_retriever.k = k
|
||||
|
||||
vector_search_retriever = VectorSearchRetriever(
|
||||
if native_hybrid_search and not enable_enriched_texts:
|
||||
native_result = await query_doc_with_native_hybrid_search(
|
||||
collection_name=collection_name,
|
||||
query=query,
|
||||
embedding_function=embedding_function,
|
||||
top_k=k,
|
||||
)
|
||||
|
||||
# Use CHUNK_HASH_KEY for dedup so enriched BM25 texts don't defeat RRF
|
||||
if hybrid_bm25_weight <= 0:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[vector_search_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
elif hybrid_bm25_weight >= 1:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
else:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever, vector_search_retriever],
|
||||
weights=[hybrid_bm25_weight, 1.0 - hybrid_bm25_weight],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
|
||||
compressor = RerankCompressor(
|
||||
embedding_function=embedding_function,
|
||||
top_n=k_reranker,
|
||||
k=k,
|
||||
reranking_function=reranking_function,
|
||||
r_score=r,
|
||||
k_reranker=k_reranker,
|
||||
r=r,
|
||||
hybrid_bm25_weight=hybrid_bm25_weight,
|
||||
)
|
||||
if native_result is not None:
|
||||
return native_result
|
||||
|
||||
if collection_result is None:
|
||||
collection_result = await ASYNC_VECTOR_DB_CLIENT.get(collection_name=collection_name)
|
||||
|
||||
# First check if collection_result has the required attributes
|
||||
if (
|
||||
not collection_result
|
||||
or not hasattr(collection_result, 'documents')
|
||||
or not hasattr(collection_result, 'metadatas')
|
||||
):
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
# Now safely check the documents content after confirming attributes exist
|
||||
if not collection_result.documents or len(collection_result.documents) == 0 or not collection_result.documents[0]:
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
log.debug('query_doc_with_hybrid_search:doc %s', collection_name)
|
||||
|
||||
original_texts = collection_result.documents[0]
|
||||
bm25_metadatas = [
|
||||
{**meta, CHUNK_HASH_KEY: _content_hash(original_texts[idx])}
|
||||
for idx, meta in enumerate(collection_result.metadatas[0])
|
||||
]
|
||||
|
||||
bm25_texts = get_enriched_texts(collection_result) if enable_enriched_texts else original_texts
|
||||
|
||||
from rank_bm25 import BM25Okapi
|
||||
|
||||
bm25_retriever = BM25Retriever(
|
||||
docs=[Document(page_content=text, metadata=meta) for text, meta in zip(bm25_texts, bm25_metadatas)],
|
||||
vectorizer=BM25Okapi([text.split() for text in bm25_texts]),
|
||||
k=k,
|
||||
)
|
||||
|
||||
vector_search_retriever = VectorSearchRetriever(
|
||||
collection_name=collection_name,
|
||||
embedding_function=embedding_function,
|
||||
top_k=k,
|
||||
)
|
||||
|
||||
# Use CHUNK_HASH_KEY for dedup so enriched BM25 texts don't defeat RRF
|
||||
if hybrid_bm25_weight <= 0:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[vector_search_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
elif hybrid_bm25_weight >= 1:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
else:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever, vector_search_retriever],
|
||||
weights=[hybrid_bm25_weight, 1.0 - hybrid_bm25_weight],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
|
||||
compression_retriever = ContextualCompressionRetriever(
|
||||
base_compressor=compressor, base_retriever=ensemble_retriever
|
||||
)
|
||||
compressor = RerankCompressor(
|
||||
embedding_function=embedding_function,
|
||||
top_n=k_reranker,
|
||||
reranking_function=reranking_function,
|
||||
r_score=r,
|
||||
)
|
||||
|
||||
result = await compression_retriever.ainvoke(query)
|
||||
compression_retriever = ContextualCompressionRetriever(
|
||||
base_compressor=compressor, base_retriever=ensemble_retriever
|
||||
)
|
||||
|
||||
distances = [d.metadata.get('score') for d in result]
|
||||
documents = [d.page_content for d in result]
|
||||
metadatas = [d.metadata for d in result]
|
||||
result = await compression_retriever.ainvoke(query)
|
||||
|
||||
# retrieve only min(k, k_reranker) items, sort and cut by distance if k < k_reranker
|
||||
if k < k_reranker:
|
||||
sorted_items = sorted(zip(distances, documents, metadatas), key=lambda x: x[0], reverse=True)
|
||||
sorted_items = sorted_items[:k]
|
||||
distances = [d.metadata.get('score') for d in result]
|
||||
documents = [d.page_content for d in result]
|
||||
metadatas = [d.metadata for d in result]
|
||||
|
||||
if sorted_items:
|
||||
distances, documents, metadatas = map(list, zip(*sorted_items))
|
||||
else:
|
||||
distances, documents, metadatas = [], [], []
|
||||
# retrieve only min(k, k_reranker) items, sort and cut by distance if k < k_reranker
|
||||
if k < k_reranker:
|
||||
sorted_items = sorted(zip(distances, documents, metadatas), key=lambda x: x[0], reverse=True)
|
||||
sorted_items = sorted_items[:k]
|
||||
|
||||
result = {
|
||||
'distances': [distances],
|
||||
'documents': [documents],
|
||||
'metadatas': [metadatas],
|
||||
}
|
||||
if sorted_items:
|
||||
distances, documents, metadatas = map(list, zip(*sorted_items))
|
||||
else:
|
||||
distances, documents, metadatas = [], [], []
|
||||
|
||||
log.info('query_doc_with_hybrid_search:result %s %s', result['metadatas'], result['distances'])
|
||||
return result
|
||||
except Exception as e:
|
||||
log.exception(f'Error querying doc {collection_name} with hybrid search: {e}')
|
||||
raise e
|
||||
result = {
|
||||
'distances': [distances],
|
||||
'documents': [documents],
|
||||
'metadatas': [metadatas],
|
||||
}
|
||||
|
||||
log.info('query_doc_with_hybrid_search:result %s %s', result['metadatas'], result['distances'])
|
||||
return result
|
||||
|
||||
|
||||
def merge_get_results(get_results: list[dict]) -> dict:
|
||||
|
|
@ -731,7 +733,8 @@ async def query_collection(
|
|||
log.debug('Hybrid search failed, falling back to vector search: %s', e)
|
||||
|
||||
results = []
|
||||
error = False
|
||||
last_error = None
|
||||
failed_collection_names = set()
|
||||
|
||||
def process_query_collection(collection_name, query_embedding):
|
||||
try:
|
||||
|
|
@ -742,11 +745,10 @@ async def query_collection(
|
|||
query_embedding=query_embedding,
|
||||
)
|
||||
if result is not None:
|
||||
return result.model_dump(), None
|
||||
return None, None
|
||||
return result.model_dump(), None, collection_name
|
||||
return None, None, collection_name
|
||||
except Exception as e:
|
||||
log.exception(f'Error when querying the collection: {e}')
|
||||
return None, e
|
||||
return None, e, collection_name
|
||||
|
||||
# Sanitize: filter out None/empty queries to prevent embedding crashes
|
||||
# (e.g. when get_last_user_message returns None)
|
||||
|
|
@ -767,14 +769,20 @@ async def query_collection(
|
|||
]
|
||||
)
|
||||
|
||||
for result, err in task_results:
|
||||
for result, err, collection_name in task_results:
|
||||
if err is not None:
|
||||
error = True
|
||||
last_error = err
|
||||
failed_collection_names.add(collection_name)
|
||||
elif result is not None:
|
||||
results.append(result)
|
||||
|
||||
if error and not results:
|
||||
log.warning('All collection queries failed. No results returned.')
|
||||
if failed_collection_names:
|
||||
log.error(
|
||||
'query_collection: %s collection(s) had failing queries: %s',
|
||||
len(failed_collection_names),
|
||||
', '.join(sorted(failed_collection_names)),
|
||||
exc_info=last_error,
|
||||
)
|
||||
|
||||
return merge_and_sort_query_results(results, k=k)
|
||||
|
||||
|
|
@ -791,7 +799,8 @@ async def query_collection_with_hybrid_search(
|
|||
enable_enriched_texts: bool = False,
|
||||
) -> dict:
|
||||
results = []
|
||||
error = False
|
||||
last_error = None
|
||||
failed_collection_names = set()
|
||||
|
||||
if not enable_enriched_texts:
|
||||
|
||||
|
|
@ -850,10 +859,9 @@ async def query_collection_with_hybrid_search(
|
|||
enable_enriched_texts=enable_enriched_texts,
|
||||
native_hybrid_search=False,
|
||||
)
|
||||
return result, None
|
||||
return result, None, collection_name
|
||||
except Exception as e:
|
||||
log.exception(f'Error when querying the collection with hybrid_search: {e}')
|
||||
return None, e
|
||||
return None, e, collection_name
|
||||
|
||||
# Prepare tasks for all collections and queries
|
||||
# Avoid running any tasks for collections that failed to fetch data (have assigned None)
|
||||
|
|
@ -867,13 +875,22 @@ async def query_collection_with_hybrid_search(
|
|||
# Run all queries in parallel using asyncio.gather
|
||||
task_results = await asyncio.gather(*[process_query(collection_name, query) for collection_name, query in tasks])
|
||||
|
||||
for result, err in task_results:
|
||||
for result, err, collection_name in task_results:
|
||||
if err is not None:
|
||||
error = True
|
||||
last_error = err
|
||||
failed_collection_names.add(collection_name)
|
||||
elif result is not None:
|
||||
results.append(result)
|
||||
|
||||
if error and not results:
|
||||
if failed_collection_names:
|
||||
log.error(
|
||||
'query_collection_with_hybrid_search: %s collection(s) had failing queries: %s',
|
||||
len(failed_collection_names),
|
||||
', '.join(sorted(failed_collection_names)),
|
||||
exc_info=last_error,
|
||||
)
|
||||
|
||||
if failed_collection_names and not results:
|
||||
raise Exception('Hybrid search failed for all collections. Using Non-hybrid search as fallback.')
|
||||
|
||||
return merge_and_sort_query_results(results, k=k)
|
||||
|
|
@ -1109,6 +1126,8 @@ def get_embedding_function(
|
|||
if embedding_engine == '':
|
||||
# Sentence transformers: CPU-bound sync operation
|
||||
async def async_embedding_function(query, prefix=None, user=None):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external embedding engine (openai, ollama, azure_openai).')
|
||||
# Deferred so a missing local model degrades RAG instead of crashing boot.
|
||||
if embedding_function is None:
|
||||
raise ValueError(
|
||||
|
|
@ -1116,17 +1135,16 @@ def get_embedding_function(
|
|||
'SentenceTransformer model name, or configure an external '
|
||||
'RAG_EMBEDDING_ENGINE (ollama, openai, azure_openai).'
|
||||
)
|
||||
return await asyncio.to_thread(
|
||||
(
|
||||
lambda query, prefix=None: embedding_function.encode(
|
||||
|
||||
def encode():
|
||||
with MPS_INFERENCE_LOCK:
|
||||
return embedding_function.encode(
|
||||
query,
|
||||
batch_size=int(embedding_batch_size),
|
||||
**({'prompt': prefix} if prefix else {}),
|
||||
).tolist()
|
||||
),
|
||||
query,
|
||||
prefix,
|
||||
)
|
||||
|
||||
return await asyncio.to_thread(encode)
|
||||
|
||||
return async_embedding_function
|
||||
elif embedding_engine in ['ollama', 'openai', 'azure_openai']:
|
||||
|
|
@ -1243,6 +1261,14 @@ async def generate_embeddings(
|
|||
|
||||
|
||||
def get_reranking_function(reranking_engine, reranking_model, reranking_function, reranking_batch_size=32):
|
||||
if USE_SLIM and reranking_model and reranking_engine != 'external':
|
||||
|
||||
def unavailable(query, documents, user=None):
|
||||
raise HTTPException(
|
||||
503, 'Configure an external reranker, or clear the reranking model to use cosine scoring.'
|
||||
)
|
||||
|
||||
return unavailable
|
||||
if reranking_function is None:
|
||||
return None
|
||||
if reranking_engine == 'external':
|
||||
|
|
@ -1250,9 +1276,14 @@ def get_reranking_function(reranking_engine, reranking_model, reranking_function
|
|||
[(query, doc.page_content) for doc in documents], user=user
|
||||
)
|
||||
else:
|
||||
return lambda query, documents, user=None: reranking_function.predict(
|
||||
[(query, doc.page_content) for doc in documents], batch_size=int(reranking_batch_size)
|
||||
)
|
||||
|
||||
def predict(query, documents, user=None):
|
||||
with MPS_INFERENCE_LOCK:
|
||||
return reranking_function.predict(
|
||||
[(query, doc.page_content) for doc in documents], batch_size=int(reranking_batch_size)
|
||||
)
|
||||
|
||||
return predict
|
||||
|
||||
|
||||
# UUIDs, SHA-256 digests, and prefixed variants thereof all fit [A-Za-z0-9_-].
|
||||
|
|
@ -1331,6 +1362,11 @@ async def filter_accessible_collections(
|
|||
return validated
|
||||
|
||||
|
||||
def filter_source_metadata(metadata: dict) -> dict:
|
||||
"""Keep only the chunk metadata keys the operator allowed the model to see."""
|
||||
return {key: metadata[key] for key in RAG_SOURCE_METADATA_KEYS if metadata.get(key) is not None}
|
||||
|
||||
|
||||
async def get_sources_from_items(
|
||||
request,
|
||||
items,
|
||||
|
|
@ -1683,6 +1719,8 @@ async def get_sources_from_items(
|
|||
|
||||
|
||||
def get_model_path(model: str, update_model: bool = False):
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
# Construct huggingface_hub kwargs with local_files_only to return the snapshot path
|
||||
cache_dir = os.getenv('SENTENCE_TRANSFORMERS_HOME')
|
||||
|
||||
|
|
@ -1728,6 +1766,17 @@ from langchain_core.callbacks import Callbacks
|
|||
from langchain_core.documents import BaseDocumentCompressor, Document
|
||||
|
||||
|
||||
def cosine_similarity(query, documents) -> np.ndarray:
|
||||
"""Score one query against documents without loading a model runtime."""
|
||||
if len(documents) == 0:
|
||||
return np.array([], dtype=float)
|
||||
query = np.asarray(query, dtype=float).reshape(-1)
|
||||
documents = np.asarray(documents, dtype=float)
|
||||
query = query / max(np.linalg.norm(query), 1e-12)
|
||||
documents = documents / np.maximum(np.linalg.norm(documents, axis=1, keepdims=True), 1e-12)
|
||||
return documents @ query
|
||||
|
||||
|
||||
class RerankCompressor(BaseDocumentCompressor):
|
||||
embedding_function: Any
|
||||
top_n: int
|
||||
|
|
@ -1763,18 +1812,18 @@ class RerankCompressor(BaseDocumentCompressor):
|
|||
query: str,
|
||||
callbacks: Callbacks | None = None,
|
||||
) -> Sequence[Document]:
|
||||
if not documents:
|
||||
return []
|
||||
reranking = self.reranking_function is not None
|
||||
|
||||
scores = None
|
||||
if reranking:
|
||||
scores = await asyncio.to_thread(self.reranking_function, query, documents)
|
||||
else:
|
||||
from sentence_transformers import util as st_util
|
||||
|
||||
query_embedding = await self.embedding_function(query, RAG_EMBEDDING_QUERY_PREFIX)
|
||||
doc_texts = [doc.page_content for doc in documents]
|
||||
document_embedding = await self.embedding_function(doc_texts, RAG_EMBEDDING_CONTENT_PREFIX)
|
||||
scores = st_util.cos_sim(query_embedding, document_embedding)[0]
|
||||
scores = cosine_similarity(query_embedding, document_embedding)
|
||||
|
||||
if scores is not None:
|
||||
docs_with_scores = list(
|
||||
|
|
|
|||
|
|
@ -15,9 +15,8 @@ transparently dispatches each call to a worker thread via
|
|||
`asyncio.to_thread`. Async callers can `await ASYNC_VECTOR_DB_CLIENT.x(...)`
|
||||
in place of `VECTOR_DB_CLIENT.x(...)` and the loop stays responsive.
|
||||
|
||||
The original `VECTOR_DB_CLIENT` is unchanged, so callers already running
|
||||
inside `run_in_threadpool` (e.g. `save_docs_to_vector_db`) are not
|
||||
affected.
|
||||
Client initialization and calls run in the worker thread. Synchronous callers
|
||||
already inside `run_in_threadpool` use `get_vector_db_client()` directly.
|
||||
|
||||
Thread-safety expectations
|
||||
--------------------------
|
||||
|
|
@ -55,7 +54,7 @@ from __future__ import annotations
|
|||
import asyncio
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
from open_webui.retrieval.vector.factory import VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import get_vector_db_client
|
||||
from open_webui.retrieval.vector.main import (
|
||||
GetResult,
|
||||
SearchResult,
|
||||
|
|
@ -73,30 +72,30 @@ class AsyncVectorDBClient:
|
|||
typically swallowed by surrounding ``try/except``).
|
||||
"""
|
||||
|
||||
def __init__(self, sync_client: VectorDBBase) -> None:
|
||||
def __init__(self, sync_client: Optional[VectorDBBase] = None) -> None:
|
||||
self._sync = sync_client
|
||||
|
||||
@property
|
||||
def sync(self) -> VectorDBBase:
|
||||
"""Escape hatch for code that must call the sync client directly
|
||||
(e.g. already inside a worker thread)."""
|
||||
return self._sync
|
||||
return self._sync if self._sync is not None else get_vector_db_client()
|
||||
|
||||
@property
|
||||
def supports_hybrid_search(self) -> bool:
|
||||
return type(self._sync).hybrid_search is not VectorDBBase.hybrid_search
|
||||
return type(self.sync).hybrid_search is not VectorDBBase.hybrid_search
|
||||
|
||||
async def has_collection(self, collection_name: str) -> bool:
|
||||
return await asyncio.to_thread(self._sync.has_collection, collection_name)
|
||||
return await asyncio.to_thread(lambda: self.sync.has_collection(collection_name))
|
||||
|
||||
async def delete_collection(self, collection_name: str) -> None:
|
||||
return await asyncio.to_thread(self._sync.delete_collection, collection_name)
|
||||
return await asyncio.to_thread(lambda: self.sync.delete_collection(collection_name))
|
||||
|
||||
async def insert(self, collection_name: str, items: List[VectorItem]) -> None:
|
||||
return await asyncio.to_thread(self._sync.insert, collection_name, items)
|
||||
return await asyncio.to_thread(lambda: self.sync.insert(collection_name, items))
|
||||
|
||||
async def upsert(self, collection_name: str, items: List[VectorItem]) -> None:
|
||||
return await asyncio.to_thread(self._sync.upsert, collection_name, items)
|
||||
return await asyncio.to_thread(lambda: self.sync.upsert(collection_name, items))
|
||||
|
||||
async def search(
|
||||
self,
|
||||
|
|
@ -105,7 +104,7 @@ class AsyncVectorDBClient:
|
|||
filter: Optional[Dict] = None,
|
||||
limit: int = 10,
|
||||
) -> Optional[SearchResult]:
|
||||
return await asyncio.to_thread(self._sync.search, collection_name, vectors, filter, limit)
|
||||
return await asyncio.to_thread(lambda: self.sync.search(collection_name, vectors, filter, limit))
|
||||
|
||||
async def hybrid_search(
|
||||
self,
|
||||
|
|
@ -117,13 +116,7 @@ class AsyncVectorDBClient:
|
|||
hybrid_bm25_weight: float = 0.5,
|
||||
) -> Optional[SearchResult]:
|
||||
return await asyncio.to_thread(
|
||||
self._sync.hybrid_search,
|
||||
collection_name,
|
||||
query,
|
||||
vectors,
|
||||
filter,
|
||||
limit,
|
||||
hybrid_bm25_weight,
|
||||
lambda: self.sync.hybrid_search(collection_name, query, vectors, filter, limit, hybrid_bm25_weight)
|
||||
)
|
||||
|
||||
async def query(
|
||||
|
|
@ -132,10 +125,10 @@ class AsyncVectorDBClient:
|
|||
filter: Dict,
|
||||
limit: Optional[int] = None,
|
||||
) -> Optional[GetResult]:
|
||||
return await asyncio.to_thread(self._sync.query, collection_name, filter, limit)
|
||||
return await asyncio.to_thread(lambda: self.sync.query(collection_name, filter, limit))
|
||||
|
||||
async def get(self, collection_name: str) -> Optional[GetResult]:
|
||||
return await asyncio.to_thread(self._sync.get, collection_name)
|
||||
return await asyncio.to_thread(lambda: self.sync.get(collection_name))
|
||||
|
||||
async def delete(
|
||||
self,
|
||||
|
|
@ -143,10 +136,10 @@ class AsyncVectorDBClient:
|
|||
ids: Optional[List[str]] = None,
|
||||
filter: Optional[Dict] = None,
|
||||
) -> None:
|
||||
return await asyncio.to_thread(self._sync.delete, collection_name, ids, filter)
|
||||
return await asyncio.to_thread(lambda: self.sync.delete(collection_name, ids, filter))
|
||||
|
||||
async def reset(self) -> None:
|
||||
return await asyncio.to_thread(self._sync.reset)
|
||||
return await asyncio.to_thread(lambda: self.sync.reset())
|
||||
|
||||
|
||||
ASYNC_VECTOR_DB_CLIENT = AsyncVectorDBClient(VECTOR_DB_CLIENT)
|
||||
ASYNC_VECTOR_DB_CLIENT = AsyncVectorDBClient()
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@ from open_webui.config import (
|
|||
CHROMA_HTTP_SSL,
|
||||
CHROMA_TENANT,
|
||||
)
|
||||
from open_webui.env import USE_SLIM
|
||||
from fastapi import HTTPException
|
||||
from open_webui.retrieval.vector.main import (
|
||||
GetResult,
|
||||
SearchResult,
|
||||
|
|
@ -29,6 +31,8 @@ log = logging.getLogger(__name__)
|
|||
|
||||
class ChromaClient(VectorDBBase):
|
||||
def __init__(self):
|
||||
if USE_SLIM and not CHROMA_HTTP_HOST:
|
||||
raise HTTPException(503, 'Configure CHROMA_HTTP_HOST: embedded Chroma is unavailable in slim.')
|
||||
settings_dict = {
|
||||
'allow_reset': True,
|
||||
'anonymized_telemetry': False,
|
||||
|
|
@ -58,7 +62,7 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def has_collection(self, collection_name: str) -> bool:
|
||||
try:
|
||||
self.client.get_collection(name=collection_name)
|
||||
self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
return True
|
||||
except NotFoundError:
|
||||
return False
|
||||
|
|
@ -76,7 +80,7 @@ class ChromaClient(VectorDBBase):
|
|||
) -> Optional[SearchResult]:
|
||||
# Search for the nearest neighbor items based on the vectors and return 'limit' number of results.
|
||||
try:
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
result = collection.query(
|
||||
query_embeddings=vectors,
|
||||
|
|
@ -105,7 +109,7 @@ class ChromaClient(VectorDBBase):
|
|||
def query(self, collection_name: str, filter: dict, limit: Optional[int] = None) -> Optional[GetResult]:
|
||||
# Query the items from the collection based on the filter.
|
||||
try:
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
result = collection.get(
|
||||
where=filter,
|
||||
|
|
@ -125,7 +129,7 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def get(self, collection_name: str) -> Optional[GetResult]:
|
||||
# Get all the items in the collection.
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
result = collection.get()
|
||||
return GetResult(
|
||||
|
|
@ -139,7 +143,9 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def insert(self, collection_name: str, items: list[VectorItem]):
|
||||
# Insert the items into the collection, if the collection does not exist, it will be created.
|
||||
collection = self.client.get_or_create_collection(name=collection_name, metadata={'hnsw:space': 'cosine'})
|
||||
collection = self.client.get_or_create_collection(
|
||||
name=collection_name, metadata={'hnsw:space': 'cosine'}, embedding_function=None
|
||||
)
|
||||
|
||||
ids = [item['id'] for item in items]
|
||||
documents = [item['text'] for item in items]
|
||||
|
|
@ -157,7 +163,9 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def upsert(self, collection_name: str, items: list[VectorItem]):
|
||||
# Update the items in the collection, if the items are not present, insert them. If the collection does not exist, it will be created.
|
||||
collection = self.client.get_or_create_collection(name=collection_name, metadata={'hnsw:space': 'cosine'})
|
||||
collection = self.client.get_or_create_collection(
|
||||
name=collection_name, metadata={'hnsw:space': 'cosine'}, embedding_function=None
|
||||
)
|
||||
|
||||
ids = [item['id'] for item in items]
|
||||
documents = [item['text'] for item in items]
|
||||
|
|
@ -174,7 +182,7 @@ class ChromaClient(VectorDBBase):
|
|||
):
|
||||
# Delete the items from the collection based on the ids.
|
||||
try:
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
if ids:
|
||||
collection.delete(ids=ids)
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import process_metadata
|
||||
from pymilvus import DataType
|
||||
from pymilvus import MilvusClient as Client
|
||||
from pymilvus.exceptions import MilvusException
|
||||
|
|
@ -191,7 +192,7 @@ class MilvusClient(VectorDBBase):
|
|||
'id': item['id'],
|
||||
'vector': item['vector'],
|
||||
'text': text,
|
||||
'metadata': item['metadata'],
|
||||
'metadata': process_metadata(item['metadata']),
|
||||
RESOURCE_ID_FIELD: resource_id,
|
||||
}
|
||||
)
|
||||
|
|
|
|||
|
|
@ -311,6 +311,7 @@ class OpenGaussClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
@ -333,6 +334,7 @@ class OpenGaussClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions, process_metadata
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -400,7 +400,7 @@ class Oracle23aiClient(VectorDBBase):
|
|||
Returns:
|
||||
str: JSON representation of metadata
|
||||
"""
|
||||
return json.dumps(metadata, default=self._decimal_handler) if metadata else '{}'
|
||||
return json.dumps(process_metadata(metadata), default=self._decimal_handler) if metadata else '{}'
|
||||
|
||||
def _json_to_metadata(self, json_str: str) -> Dict:
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from open_webui.config import (
|
|||
PGVECTOR_HNSW_M,
|
||||
PGVECTOR_INDEX_METHOD,
|
||||
PGVECTOR_INITIALIZE_MAX_VECTOR_LENGTH,
|
||||
PGVECTOR_ITERATIVE_SCAN,
|
||||
PGVECTOR_IVFFLAT_LISTS,
|
||||
PGVECTOR_PGCRYPTO,
|
||||
PGVECTOR_PGCRYPTO_KEY,
|
||||
|
|
@ -154,6 +155,7 @@ class PgvectorClient(VectorDBBase):
|
|||
index_method, index_options = self._vector_index_configuration()
|
||||
self._ensure_vector_index(index_method, index_options)
|
||||
self._ensure_text_search_index()
|
||||
self.iterative_scan_sql = self._iterative_scan_setting(index_method)
|
||||
|
||||
self.session.execute(
|
||||
text(
|
||||
|
|
@ -223,6 +225,9 @@ class PgvectorClient(VectorDBBase):
|
|||
)
|
||||
|
||||
if not existing_index_def:
|
||||
if index_method == 'ivfflat' and not self._has_enough_ivfflat_training_rows():
|
||||
return
|
||||
|
||||
index_sql = (
|
||||
f'CREATE INDEX IF NOT EXISTS {index_name} '
|
||||
f'ON document_chunk USING {index_method} (vector {VECTOR_OPCLASS})'
|
||||
|
|
@ -237,6 +242,38 @@ class PgvectorClient(VectorDBBase):
|
|||
f' {index_options}' if index_options else '',
|
||||
)
|
||||
|
||||
def _has_enough_ivfflat_training_rows(self) -> bool:
|
||||
# ivfflat samples 50 rows per list to place its centroids, so recall stays poor until the table holds that many
|
||||
min_training_rows = 50 * PGVECTOR_IVFFLAT_LISTS
|
||||
row_count = self.session.execute(
|
||||
text('SELECT count(*) FROM (SELECT 1 FROM document_chunk LIMIT :min_training_rows) AS sample'),
|
||||
{'min_training_rows': min_training_rows},
|
||||
).scalar()
|
||||
|
||||
if row_count < min_training_rows:
|
||||
log.info(
|
||||
"Deferring vector index 'idx_document_chunk_vector' until document_chunk holds %s rows to cluster on, "
|
||||
'it has %s. Searches run as an exact scan until then.',
|
||||
min_training_rows,
|
||||
row_count,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
def _iterative_scan_setting(self, index_method: str) -> Optional[str]:
|
||||
if PGVECTOR_ITERATIVE_SCAN == 'off':
|
||||
return None
|
||||
|
||||
version = self.session.execute(text("SELECT extversion FROM pg_extension WHERE extname = 'vector'")).scalar()
|
||||
version_parts = [int(part) for part in (version or '').split('.') if part.isdigit()]
|
||||
if version_parts[:2] < [0, 8]:
|
||||
log.info('Iterative scan needs pgvector 0.8 or newer, the server has %s.', version or 'none')
|
||||
return None
|
||||
|
||||
# ivfflat only accepts relaxed_order
|
||||
mode = 'relaxed_order' if index_method == 'ivfflat' else PGVECTOR_ITERATIVE_SCAN
|
||||
return f'SET LOCAL {index_method}.iterative_scan = {mode}'
|
||||
|
||||
def _ensure_text_search_index(self) -> None:
|
||||
if PGVECTOR_PGCRYPTO:
|
||||
return
|
||||
|
|
@ -503,6 +540,9 @@ class PgvectorClient(VectorDBBase):
|
|||
.order_by(query_vectors.c.qid, subq.c.distance)
|
||||
)
|
||||
|
||||
if self.iterative_scan_sql:
|
||||
self.session.execute(text(self.iterative_scan_sql))
|
||||
|
||||
result_proxy = self.session.execute(stmt)
|
||||
results = result_proxy.all()
|
||||
|
||||
|
|
@ -512,6 +552,7 @@ class PgvectorClient(VectorDBBase):
|
|||
metadatas = [[] for _ in range(num_queries)]
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return SearchResult(
|
||||
ids=ids,
|
||||
distances=distances,
|
||||
|
|
@ -631,6 +672,7 @@ class PgvectorClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
@ -670,6 +712,7 @@ class PgvectorClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions, process_metadata
|
||||
from qdrant_client import QdrantClient as Qclient
|
||||
from qdrant_client.http.models import PointStruct
|
||||
from qdrant_client.models import models
|
||||
|
|
@ -136,7 +136,7 @@ class QdrantClient(VectorDBBase):
|
|||
PointStruct(
|
||||
id=item['id'],
|
||||
vector=item['vector'],
|
||||
payload={'text': item['text'], 'metadata': item['metadata']},
|
||||
payload={'text': item['text'], 'metadata': process_metadata(item['metadata'])},
|
||||
)
|
||||
for item in items
|
||||
]
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions, process_metadata
|
||||
from qdrant_client import QdrantClient as Qclient
|
||||
from qdrant_client.http.exceptions import UnexpectedResponse
|
||||
from qdrant_client.http.models import PointStruct
|
||||
|
|
@ -182,7 +182,7 @@ class QdrantClient(VectorDBBase):
|
|||
vector=item['vector'],
|
||||
payload={
|
||||
'text': item['text'],
|
||||
'metadata': item['metadata'],
|
||||
'metadata': process_metadata(item['metadata']),
|
||||
TENANT_ID_FIELD: tenant_id,
|
||||
},
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,12 @@
|
|||
from threading import Lock
|
||||
|
||||
from fastapi import HTTPException
|
||||
from open_webui.config import (
|
||||
ENABLE_MILVUS_MULTITENANCY_MODE,
|
||||
ENABLE_QDRANT_MULTITENANCY_MODE,
|
||||
VECTOR_DB,
|
||||
)
|
||||
from open_webui.env import USE_SLIM
|
||||
from open_webui.retrieval.vector.main import VectorDBBase
|
||||
from open_webui.retrieval.vector.type import VectorType
|
||||
|
||||
|
|
@ -13,6 +17,11 @@ class Vector:
|
|||
"""
|
||||
get vector db instance by vector type
|
||||
"""
|
||||
if USE_SLIM and vector_type != VectorType.PGVECTOR:
|
||||
raise HTTPException(
|
||||
503,
|
||||
'Slim requires PostgreSQL/pgvector for vector storage. Set VECTOR_DB=pgvector and PGVECTOR_DB_URL, or use the standard image.',
|
||||
)
|
||||
match vector_type:
|
||||
case VectorType.MILVUS:
|
||||
if ENABLE_MILVUS_MULTITENANCY_MODE:
|
||||
|
|
@ -88,4 +97,27 @@ class Vector:
|
|||
raise ValueError(f'Unsupported vector type: {vector_type}')
|
||||
|
||||
|
||||
VECTOR_DB_CLIENT = Vector.get_vector(VECTOR_DB)
|
||||
VECTOR_DB_CLIENT = None if USE_SLIM else Vector.get_vector(VECTOR_DB)
|
||||
_vector_client_lock = Lock()
|
||||
|
||||
|
||||
def get_vector_db_client() -> VectorDBBase:
|
||||
"""Initialize slim's remote client on first use so chat can start without it."""
|
||||
global VECTOR_DB_CLIENT
|
||||
if VECTOR_DB_CLIENT is not None:
|
||||
return VECTOR_DB_CLIENT
|
||||
with _vector_client_lock:
|
||||
if VECTOR_DB_CLIENT is None:
|
||||
from open_webui import config
|
||||
|
||||
if VECTOR_DB == VectorType.PGVECTOR and not config.PGVECTOR_DB_URL.startswith('postgres'):
|
||||
raise HTTPException(503, 'Configure PGVECTOR_DB_URL for remote vector storage.')
|
||||
try:
|
||||
VECTOR_DB_CLIENT = Vector.get_vector(VECTOR_DB)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
503, f'Unable to connect to configured vector database ({VECTOR_DB}): {exc}'
|
||||
) from exc
|
||||
return VECTOR_DB_CLIENT
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ from __future__ import annotations
|
|||
import logging
|
||||
import urllib.request
|
||||
|
||||
from ddgs import DDGS
|
||||
from open_webui.env import USE_SLIM
|
||||
from open_webui.retrieval.web.main import SearchResult, get_filtered_results
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -26,6 +26,13 @@ def search_duckduckgo(
|
|||
Returns:
|
||||
list[SearchResult]: A list of search results
|
||||
"""
|
||||
if USE_SLIM:
|
||||
raise ValueError(
|
||||
'DDGS is unavailable in slim. Configure another web search provider in Admin Settings > Web Search.'
|
||||
)
|
||||
|
||||
from ddgs import DDGS
|
||||
|
||||
# The ddgs library (primp-based) does not auto-detect proxy env vars.
|
||||
# Resolve via stdlib getproxies() — same pattern as the other loaders.
|
||||
env_proxies = urllib.request.getproxies()
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
from open_webui.retrieval.web.main import SearchResult
|
||||
|
|
@ -10,18 +8,12 @@ log = logging.getLogger(__name__)
|
|||
EXA_API_BASE = 'https://api.exa.ai'
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExaResult:
|
||||
url: str
|
||||
title: str
|
||||
text: str
|
||||
|
||||
|
||||
def search_exa(
|
||||
api_key: str,
|
||||
query: str,
|
||||
count: int,
|
||||
filter_list: Optional[list[str]] = None,
|
||||
filter_list: list[str] | None = None,
|
||||
max_content_length: int | None = None,
|
||||
) -> list[SearchResult]:
|
||||
"""Search using Exa Search API and return the results as a list of SearchResult objects.
|
||||
|
||||
|
|
@ -29,7 +21,8 @@ def search_exa(
|
|||
api_key (str): A Exa Search API key
|
||||
query (str): The query to search for
|
||||
count (int): Number of results to return
|
||||
filter_list (Optional[list[str]]): List of domains to filter results by
|
||||
filter_list (list[str] | None): List of domains to filter results by
|
||||
max_content_length (int | None): Maximum characters per result; None leaves text unlimited.
|
||||
"""
|
||||
log.info('Searching with Exa for query: %s', query)
|
||||
|
||||
|
|
@ -39,7 +32,7 @@ def search_exa(
|
|||
'query': query,
|
||||
'numResults': count or 5,
|
||||
'includeDomains': filter_list,
|
||||
'contents': {'text': True, 'highlights': True},
|
||||
'contents': {'text': {'maxCharacters': max_content_length} if max_content_length is not None else True},
|
||||
'type': 'auto', # Use the auto search type (keyword or neural)
|
||||
}
|
||||
|
||||
|
|
@ -48,22 +41,13 @@ def search_exa(
|
|||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
results = []
|
||||
for result in data['results']:
|
||||
results.append(
|
||||
ExaResult(
|
||||
url=result['url'],
|
||||
title=result['title'],
|
||||
text=result['text'],
|
||||
)
|
||||
)
|
||||
|
||||
results = data['results']
|
||||
log.info('Found %s results', len(results))
|
||||
return [
|
||||
SearchResult(
|
||||
link=result.url,
|
||||
title=result.title,
|
||||
snippet=result.text,
|
||||
link=result['url'],
|
||||
title=result['title'],
|
||||
snippet=(result.get('text') or '')[:max_content_length],
|
||||
)
|
||||
for result in results
|
||||
]
|
||||
|
|
|
|||
60
backend/open_webui/retrieval/web/staan.py
Normal file
60
backend/open_webui/retrieval/web/staan.py
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import requests
|
||||
from open_webui.retrieval.web.main import SearchResult, get_filtered_results
|
||||
|
||||
|
||||
def search_staan(
|
||||
api_key: str,
|
||||
query: str,
|
||||
count: int,
|
||||
filter_list: list[str] | None = None,
|
||||
market: str | None = None,
|
||||
max_snippets: int | None = None,
|
||||
) -> list[SearchResult]:
|
||||
"""Search using Staan's Web Search API and return the results as a list of SearchResult objects.
|
||||
|
||||
Args:
|
||||
api_key (str): A Staan API key
|
||||
query (str): The query to search for
|
||||
count (int): The maximum number of results to return
|
||||
filter_list (list[str] | None): The domains to allow or block
|
||||
market (str | None): The market to search in, e.g. 'en-us'
|
||||
max_snippets (int | None): The maximum extra snippets to request per result
|
||||
|
||||
Returns:
|
||||
A list of SearchResult objects.
|
||||
"""
|
||||
url = 'https://api.staan.ai/v2/search/web'
|
||||
headers = {
|
||||
'Accept': 'application/json',
|
||||
'Authorization': f'Bearer {api_key}',
|
||||
}
|
||||
params = {'q': query, 'market': market}
|
||||
|
||||
if max_snippets:
|
||||
params['extra_snippets'] = 'true'
|
||||
params['max_snippets'] = max_snippets
|
||||
|
||||
response = requests.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
|
||||
results = response.json().get('web', {}).get('results', [])
|
||||
if filter_list:
|
||||
results = get_filtered_results(results, filter_list)
|
||||
|
||||
return [
|
||||
SearchResult(
|
||||
link=result.get('url', ''),
|
||||
title=result.get('title'),
|
||||
snippet=_build_snippet(result),
|
||||
)
|
||||
for result in results[:count]
|
||||
]
|
||||
|
||||
|
||||
def _build_snippet(result: dict) -> str:
|
||||
"""Combine the snippet and the extra snippets list into a single string."""
|
||||
parts = [result.get('snippet')]
|
||||
parts.extend(extra.get('chunk') for extra in result.get('extra_snippets', []))
|
||||
return '\n\n'.join(part for part in parts if part)
|
||||
|
|
@ -8,6 +8,7 @@ import time
|
|||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta
|
||||
from importlib import import_module
|
||||
from typing import (
|
||||
Any,
|
||||
AsyncIterator,
|
||||
|
|
@ -29,9 +30,9 @@ import urllib3.connection
|
|||
import urllib3.connectionpool
|
||||
import validators
|
||||
from requests.adapters import HTTPAdapter
|
||||
from fastapi import HTTPException
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from langchain_community.document_loaders import PlaywrightURLLoader, WebBaseLoader
|
||||
from langchain_community.document_loaders.base import BaseLoader
|
||||
from langchain_core.document_loaders import BaseLoader
|
||||
from langchain_core.documents import Document
|
||||
from open_webui.config import (
|
||||
ENABLE_LOCAL_WEB_FETCH,
|
||||
|
|
@ -58,6 +59,7 @@ from open_webui.env import (
|
|||
AIOHTTP_CLIENT_SSL_CERT_FILE,
|
||||
AIOHTTP_CLIENT_TIMEOUT,
|
||||
USER_AGENT,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.retrieval.loaders.external_web import ExternalWebLoader
|
||||
from open_webui.retrieval.loaders.microsoft_web_iq import MicrosoftWebIQLoader
|
||||
|
|
@ -128,7 +130,7 @@ def _assert_addresses_allowed(addresses: Sequence[str]) -> None:
|
|||
|
||||
def validate_url(url: Union[str, Sequence[str]]):
|
||||
if isinstance(url, str):
|
||||
if isinstance(validators.url(url), validators.ValidationError):
|
||||
if isinstance(validators.url(url, simple_host=ENABLE_LOCAL_WEB_FETCH), validators.ValidationError):
|
||||
raise ValueError(ERROR_MESSAGES.INVALID_URL)
|
||||
|
||||
# Reject parser-confusing chars: urlparse and requests/aiohttp split
|
||||
|
|
@ -302,6 +304,9 @@ _DROPPED_REQUEST_HEADERS = {'accept-encoding', 'connection', 'content-length', '
|
|||
# The clients hand us a decoded body, so the sender's framing no longer describes it.
|
||||
_DROPPED_RESPONSE_HEADERS = {'connection', 'content-encoding', 'content-length', 'transfer-encoding'}
|
||||
|
||||
# The Playwright loader only reads the page HTML, which none of these feed.
|
||||
_DROPPED_RESOURCE_TYPES = {'font', 'image', 'media'}
|
||||
|
||||
|
||||
def _forwardable_request_headers(headers: Dict[str, str]) -> Dict[str, str]:
|
||||
return {name: value for name, value in headers.items() if name.lower() not in _DROPPED_REQUEST_HEADERS}
|
||||
|
|
@ -643,7 +648,7 @@ class SafeMicrosoftWebIQLoader(BaseLoader, RateLimitMixin, URLProcessingMixin):
|
|||
raise e
|
||||
|
||||
|
||||
class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessingMixin):
|
||||
class SafePlaywrightURLLoader(BaseLoader, RateLimitMixin, URLProcessingMixin):
|
||||
"""Load HTML pages safely with Playwright, supporting SSL verification, rate limiting, and remote browser connection.
|
||||
|
||||
Attributes:
|
||||
|
|
@ -674,6 +679,13 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
playwright_timeout: Optional[int] = 10000,
|
||||
):
|
||||
"""Initialize with additional safety parameters and remote browser support."""
|
||||
if USE_SLIM:
|
||||
raise HTTPException(
|
||||
503, 'Playwright is unavailable in slim. Use basic HTTP fetching or an external web loader.'
|
||||
)
|
||||
|
||||
for package in ('playwright', 'unstructured'):
|
||||
import_module(package)
|
||||
|
||||
proxy_server = proxy.get('server') if proxy else None
|
||||
if trust_env and not proxy_server:
|
||||
|
|
@ -685,14 +697,11 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
else:
|
||||
proxy = {'server': env_proxy_server}
|
||||
|
||||
# We'll set headless to False if using playwright_ws_url since it's handled by the remote browser
|
||||
super().__init__(
|
||||
urls=web_paths,
|
||||
continue_on_failure=continue_on_failure,
|
||||
headless=headless if playwright_ws_url is None else False,
|
||||
remove_selectors=remove_selectors,
|
||||
proxy=proxy,
|
||||
)
|
||||
self.urls = web_paths
|
||||
self.continue_on_failure = continue_on_failure
|
||||
self.headless = headless if playwright_ws_url is None else False
|
||||
self.remove_selectors = remove_selectors or []
|
||||
self.proxy = proxy
|
||||
self.verify_ssl = verify_ssl
|
||||
self.requests_per_second = requests_per_second
|
||||
self.last_request_time = None
|
||||
|
|
@ -700,6 +709,12 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
self.trust_env = trust_env
|
||||
self.playwright_timeout = playwright_timeout
|
||||
|
||||
@staticmethod
|
||||
def _extract_html(html):
|
||||
from unstructured.partition.html import partition_html
|
||||
|
||||
return '\n\n'.join(str(element) for element in partition_html(text=html))
|
||||
|
||||
def _request_timeout(self) -> float:
|
||||
# per-hop budget, since page.goto's timeout cannot reach into our own fetch and 0 disables
|
||||
# it. aiohttp treats it as a total where requests only caps each read, so sync runs looser.
|
||||
|
|
@ -719,6 +734,9 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
|
||||
def _intercept_navigation_sync(self, route, session):
|
||||
req = route.request
|
||||
if req.resource_type in _DROPPED_RESOURCE_TYPES:
|
||||
route.abort()
|
||||
return
|
||||
|
||||
hop_cookies: List[Tuple[str, str]] = []
|
||||
|
||||
|
|
@ -773,6 +791,9 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
|
||||
async def _intercept_navigation(self, route, session):
|
||||
req = route.request
|
||||
if req.resource_type in _DROPPED_RESOURCE_TYPES:
|
||||
await route.abort()
|
||||
return
|
||||
|
||||
hop_cookies: List[Tuple[str, str]] = []
|
||||
|
||||
|
|
@ -847,12 +868,18 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
browser.new_page(service_workers='block') as page,
|
||||
):
|
||||
page.route('**/*', lambda route: self._intercept_navigation_sync(route, session))
|
||||
page.route_web_socket('**/*', lambda ws_route: ws_route.close())
|
||||
# sync close() hangs the dispatcher; a no-op handler still never connects to the server
|
||||
page.route_web_socket('**/*', lambda ws_route: None)
|
||||
response = page.goto(url, timeout=self.playwright_timeout)
|
||||
if response is None:
|
||||
raise ValueError(f'page.goto() returned None for url {url}')
|
||||
|
||||
text = self.evaluator.evaluate(page, browser, response)
|
||||
for selector in self.remove_selectors:
|
||||
for element in page.locator(selector).all():
|
||||
if element.is_visible():
|
||||
element.evaluate('element => element.remove()')
|
||||
text = self._extract_html(page.content())
|
||||
page.unroute_all(behavior='ignoreErrors')
|
||||
metadata = {'source': url}
|
||||
yield Document(page_content=text, metadata=metadata)
|
||||
except Exception as e:
|
||||
|
|
@ -887,7 +914,12 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
if response is None:
|
||||
raise ValueError(f'page.goto() returned None for url {url}')
|
||||
|
||||
text = await self.evaluator.evaluate_async(page, browser, response)
|
||||
for selector in self.remove_selectors:
|
||||
for element in await page.locator(selector).all():
|
||||
if await element.is_visible():
|
||||
await element.evaluate('element => element.remove()')
|
||||
text = await asyncio.to_thread(self._extract_html, await page.content())
|
||||
await page.unroute_all(behavior='ignoreErrors')
|
||||
metadata = {'source': url}
|
||||
yield Document(page_content=text, metadata=metadata)
|
||||
except Exception as e:
|
||||
|
|
@ -897,42 +929,58 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
raise e
|
||||
|
||||
|
||||
class SafeWebBaseLoader(WebBaseLoader):
|
||||
"""WebBaseLoader with enhanced error handling for URLs."""
|
||||
class SafeWebBaseLoader(BaseLoader):
|
||||
"""Fetch pages with connect-time address checks and bounded concurrency."""
|
||||
|
||||
def __init__(self, trust_env: bool = False, *args, **kwargs):
|
||||
"""Initialize SafeWebBaseLoader
|
||||
Args:
|
||||
trust_env (bool, optional): set to True if using proxy to make web requests, for example
|
||||
using http(s)_proxy environment variables. Defaults to False.
|
||||
"""
|
||||
# lxml parses scraped pages far faster than the html.parser default
|
||||
kwargs.setdefault('default_parser', 'lxml')
|
||||
super().__init__(*args, **kwargs)
|
||||
def __init__(
|
||||
self,
|
||||
web_paths,
|
||||
verify_ssl=True,
|
||||
trust_env=False,
|
||||
requests_per_second=2,
|
||||
continue_on_failure=False,
|
||||
requests_kwargs=None,
|
||||
raise_for_status=False,
|
||||
default_parser='lxml',
|
||||
bs_kwargs=None,
|
||||
bs_get_text_kwargs=None,
|
||||
):
|
||||
self.web_paths = list(web_paths)
|
||||
self.trust_env = trust_env
|
||||
|
||||
# Propagate USER_AGENT env var so that both the sync _scrape() and
|
||||
# async _fetch() paths present a real UA instead of python-requests/2.x
|
||||
# which gets blocked by Cloudflare, Wikipedia, and similar bot-detection.
|
||||
# _fetch() forwards self.session.headers to the aiohttp session, so
|
||||
# setting it here covers both code-paths.
|
||||
if USER_AGENT:
|
||||
self.session.headers['User-Agent'] = USER_AGENT
|
||||
|
||||
# Prevent redirect-based SSRF on the synchronous _scrape() path.
|
||||
# validate_url() is called once on the originally-submitted URL, but the
|
||||
# parent WebBaseLoader's _scrape() invokes self.session.get(url, **self.requests_kwargs)
|
||||
# which by default follows redirects. Without the override below, an attacker
|
||||
# can submit a public URL that 302-redirects to an internal address (RFC1918,
|
||||
# 127.0.0.1, 169.254.169.254, etc.) and the redirected target is fetched without
|
||||
# re-validation. Matches the policy enforced on the async _fetch() path below.
|
||||
self.requests_kwargs = {
|
||||
**(self.requests_kwargs or {}),
|
||||
'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS,
|
||||
self.requests_per_second = requests_per_second
|
||||
self.continue_on_failure = continue_on_failure
|
||||
self.requests_kwargs = {**(requests_kwargs or {}), 'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS}
|
||||
self.raise_for_status = raise_for_status
|
||||
self.default_parser = default_parser
|
||||
self.bs_kwargs = bs_kwargs or {}
|
||||
self.bs_get_text_kwargs = bs_get_text_kwargs or {}
|
||||
# Preserve the synchronous loader's environment-proxy behavior.
|
||||
self.session = get_ssrf_safe_requests_session()
|
||||
self.session.verify = verify_ssl
|
||||
self.session.headers = {
|
||||
'User-Agent': USER_AGENT or 'DefaultLangchainUserAgent',
|
||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
|
||||
'Accept-Language': 'en-US,en;q=0.5',
|
||||
'Referer': 'https://www.google.com/',
|
||||
'DNT': '1',
|
||||
'Connection': 'keep-alive',
|
||||
'Upgrade-Insecure-Requests': '1',
|
||||
}
|
||||
|
||||
self.session.mount('http://', _SSRFSafeAdapter())
|
||||
self.session.mount('https://', _SSRFSafeAdapter())
|
||||
async def fetch_all(self, urls):
|
||||
semaphore = asyncio.Semaphore(self.requests_per_second)
|
||||
|
||||
async def fetch(url):
|
||||
async with semaphore:
|
||||
try:
|
||||
return await self._fetch(url)
|
||||
except Exception as e:
|
||||
if not self.continue_on_failure:
|
||||
raise
|
||||
log.warning('Error fetching %s: %s', url, e)
|
||||
return ''
|
||||
|
||||
return await asyncio.gather(*(fetch(url) for url in urls))
|
||||
|
||||
async def _fetch(self, url: str, retries: int = 3, cooldown: int = 2, backoff: float = 1.5) -> str:
|
||||
connector = _SSRFSafeConnector()
|
||||
|
|
@ -948,10 +996,10 @@ class SafeWebBaseLoader(WebBaseLoader):
|
|||
else:
|
||||
kwargs['ssl'] = AIOHTTP_CLIENT_SESSION_SSL
|
||||
|
||||
async with session.get(
|
||||
url,
|
||||
**(self.requests_kwargs | kwargs),
|
||||
) as response:
|
||||
options = self.requests_kwargs | kwargs
|
||||
if isinstance(options.get('timeout'), (int, float)):
|
||||
options['timeout'] = aiohttp.ClientTimeout(total=options['timeout'])
|
||||
async with session.get(url, **options) as response:
|
||||
if self.raise_for_status:
|
||||
response.raise_for_status()
|
||||
return await response.text()
|
||||
|
|
@ -963,38 +1011,24 @@ class SafeWebBaseLoader(WebBaseLoader):
|
|||
await asyncio.sleep(cooldown * backoff**i)
|
||||
raise ValueError('retry count exceeded')
|
||||
|
||||
def _unpack_fetch_results(self, results: Any, urls: List[str], parser: Union[str, None] = None) -> List[Any]:
|
||||
"""Unpack fetch results into BeautifulSoup objects."""
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
final_results = []
|
||||
for i, result in enumerate(results):
|
||||
url = urls[i]
|
||||
url_parser = parser
|
||||
if url_parser is None:
|
||||
url_parser = 'xml' if url.endswith('.xml') else self.default_parser
|
||||
self._check_parser(url_parser)
|
||||
final_results.append(BeautifulSoup(result, url_parser, **self.bs_kwargs))
|
||||
return final_results
|
||||
|
||||
def lazy_load(self) -> Iterator[Document]:
|
||||
"""Lazy load text from the url(s) in web_path with error handling."""
|
||||
for path in self.web_paths:
|
||||
try:
|
||||
soup = self._scrape(path, bs_kwargs=self.bs_kwargs)
|
||||
text = soup.get_text(**self.bs_get_text_kwargs)
|
||||
|
||||
# Build metadata
|
||||
metadata = extract_metadata(soup, path)
|
||||
|
||||
yield Document(page_content=text, metadata=metadata)
|
||||
with self.session.get(path, **self.requests_kwargs) as response:
|
||||
if self.raise_for_status:
|
||||
response.raise_for_status()
|
||||
response.encoding = response.apparent_encoding
|
||||
yield self._document_from_html(response.text, path)
|
||||
except Exception as e:
|
||||
# Log the error and continue with the next URL
|
||||
log.exception(f'Error loading {path}: {e}')
|
||||
|
||||
def _document_from_html(self, html: str, url: str) -> Document:
|
||||
"""Build one Document."""
|
||||
soup = self._unpack_fetch_results([html], [url])[0]
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
parser = 'xml' if url.endswith('.xml') else self.default_parser
|
||||
soup = BeautifulSoup(html, parser, **self.bs_kwargs)
|
||||
return Document(
|
||||
page_content=soup.get_text(**self.bs_get_text_kwargs),
|
||||
metadata=extract_metadata(soup, url),
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@ from open_webui.env import (
|
|||
DEVICE_TYPE,
|
||||
ENABLE_FORWARD_USER_INFO_HEADERS,
|
||||
ENV,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.models.config import Config
|
||||
|
|
@ -58,9 +59,10 @@ from open_webui.utils.session_pool import get_session
|
|||
from pydantic import BaseModel
|
||||
|
||||
# pydub needs stdlib audioop (gone in 3.13); keep requires-python capped < 3.13
|
||||
from pydub import AudioSegment
|
||||
from pydub.silence import split_on_silence
|
||||
from pydub.utils import mediainfo
|
||||
if not USE_SLIM:
|
||||
from pydub import AudioSegment
|
||||
from pydub.silence import split_on_silence
|
||||
from pydub.utils import mediainfo
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
router = APIRouter()
|
||||
|
|
@ -213,6 +215,8 @@ def transcode_audio_to_mp3(audio_data: bytes, content_type_header: str, output_p
|
|||
|
||||
|
||||
def set_faster_whisper_model(model: str, auto_update: bool = False):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external speech-to-text engine. Local Whisper is unavailable in slim.')
|
||||
whisper_model = None
|
||||
if model:
|
||||
from faster_whisper import WhisperModel
|
||||
|
|
@ -285,14 +289,20 @@ async def get_audio_config(request: Request, user=Depends(get_admin_user)):
|
|||
|
||||
@router.post('/config/update')
|
||||
async def update_audio_config(request: Request, form_data: AudioConfigUpdateForm, user=Depends(get_admin_user)):
|
||||
if USE_SLIM:
|
||||
current = await Config.get_many('audio.stt.engine', 'audio.tts.engine')
|
||||
if form_data.stt.ENGINE == '' and current.get('audio.stt.engine') != '':
|
||||
raise HTTPException(400, 'Local Whisper is unavailable in slim. Select an external speech-to-text engine.')
|
||||
if form_data.tts.ENGINE == 'transformers' and current.get('audio.tts.engine') != 'transformers':
|
||||
raise HTTPException(400, 'Local TTS is unavailable in slim. Select an external text-to-speech engine.')
|
||||
await Config.upsert(
|
||||
{
|
||||
**config_updates(form_data.tts.model_dump(), TTS_CONFIG_KEYS),
|
||||
**config_updates(form_data.stt.model_dump(), STT_CONFIG_KEYS),
|
||||
**config_updates(form_data.tts.model_dump(exclude_unset=True), TTS_CONFIG_KEYS),
|
||||
**config_updates(form_data.stt.model_dump(exclude_unset=True), STT_CONFIG_KEYS),
|
||||
}
|
||||
)
|
||||
|
||||
if form_data.stt.ENGINE == '':
|
||||
if form_data.stt.ENGINE == '' and not USE_SLIM:
|
||||
request.app.state.faster_whisper_model = await asyncio.to_thread(
|
||||
set_faster_whisper_model, form_data.stt.WHISPER_MODEL, WHISPER_MODEL_AUTO_UPDATE
|
||||
)
|
||||
|
|
@ -314,6 +324,8 @@ async def update_audio_config(request: Request, form_data: AudioConfigUpdateForm
|
|||
|
||||
|
||||
def load_speech_pipeline(request):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external text-to-speech engine. Local TTS is unavailable in slim.')
|
||||
from datasets import load_dataset
|
||||
from transformers import pipeline
|
||||
|
||||
|
|
@ -328,7 +340,9 @@ def load_speech_pipeline(request):
|
|||
|
||||
async def _raise_tts_error(exc: Exception, r=None) -> None:
|
||||
"""Raise a standardised HTTPException from a TTS provider failure."""
|
||||
code = r.status if r is not None else 500
|
||||
if isinstance(exc, HTTPException):
|
||||
raise exc
|
||||
code = r.status if r is not None and r.status >= 400 else 500
|
||||
# LICENSE covers this Open WebUI error identifier.
|
||||
# Do not alter, remove, obscure, or replace it except as LICENSE permits:
|
||||
# https://docs.openwebui.com/license.
|
||||
|
|
@ -351,8 +365,29 @@ async def _write_tts_cache(
|
|||
audio: bytes,
|
||||
body_path: Path,
|
||||
payload: dict,
|
||||
content_type: str = 'audio/mpeg',
|
||||
) -> None:
|
||||
"""Persist audio + request metadata to the speech cache."""
|
||||
if USE_SLIM:
|
||||
mime_type = content_type.split(';')[0].strip().lower()
|
||||
if mime_type not in {
|
||||
'audio/mpeg',
|
||||
'audio/mp3',
|
||||
'audio/wav',
|
||||
'audio/x-wav',
|
||||
'audio/ogg',
|
||||
'audio/opus',
|
||||
'audio/webm',
|
||||
'audio/flac',
|
||||
'audio/aac',
|
||||
'audio/mp4',
|
||||
}:
|
||||
raise HTTPException(
|
||||
502,
|
||||
f'TTS returned unsupported format {mime_type}. Configure the provider to return MP3, WAV, Ogg, or another browser-playable audio format.',
|
||||
)
|
||||
async with aiofiles.open(file_path.with_suffix('.mime'), 'w') as f:
|
||||
await f.write(content_type)
|
||||
async with aiofiles.open(file_path, 'wb') as f:
|
||||
await f.write(audio)
|
||||
async with aiofiles.open(body_path, 'w') as f:
|
||||
|
|
@ -389,6 +424,10 @@ async def _tts_openai(request, payload, file_path, file_body_path, user):
|
|||
audio_data = await r.read()
|
||||
content_type = r.headers.get('Content-Type', 'audio/mpeg')
|
||||
|
||||
if USE_SLIM:
|
||||
await _write_tts_cache(file_path, audio_data, file_body_path, payload, content_type)
|
||||
return FileResponse(file_path, media_type=content_type)
|
||||
|
||||
if not await asyncio.to_thread(transcode_audio_to_mp3, audio_data, content_type, file_path):
|
||||
async with aiofiles.open(file_path, 'wb') as f:
|
||||
await f.write(audio_data)
|
||||
|
|
@ -430,8 +469,9 @@ async def _tts_elevenlabs(request, payload, file_path, file_body_path, user):
|
|||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload)
|
||||
return FileResponse(file_path)
|
||||
content_type = r.headers.get('Content-Type', 'audio/mpeg')
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload, content_type)
|
||||
return FileResponse(file_path, media_type=content_type if USE_SLIM else None)
|
||||
except Exception as exc:
|
||||
log.exception(exc)
|
||||
await _raise_tts_error(exc, r)
|
||||
|
|
@ -465,8 +505,9 @@ async def _tts_azure(request, payload, file_path, file_body_path, user):
|
|||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload)
|
||||
return FileResponse(file_path)
|
||||
content_type = r.headers.get('Content-Type', 'audio/mpeg')
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload, content_type)
|
||||
return FileResponse(file_path, media_type=content_type if USE_SLIM else None)
|
||||
except Exception as exc:
|
||||
log.exception(exc)
|
||||
await _raise_tts_error(exc, r)
|
||||
|
|
@ -474,6 +515,8 @@ async def _tts_azure(request, payload, file_path, file_body_path, user):
|
|||
|
||||
async def _tts_transformers(request, payload, file_path, file_body_path, user):
|
||||
"""Generate speech via the local HuggingFace SpeechT5 pipeline (thread-offloaded)."""
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external text-to-speech engine. Local TTS is unavailable in slim.')
|
||||
import soundfile as sf
|
||||
import torch
|
||||
|
||||
|
|
@ -558,6 +601,8 @@ _TTS_ENGINES = {
|
|||
@router.post('/speech')
|
||||
async def speech(request: Request, user=Depends(get_verified_user)):
|
||||
engine = await Config.get('audio.tts.engine')
|
||||
if USE_SLIM and engine in ('', 'transformers'):
|
||||
raise HTTPException(503, 'Configure an external text-to-speech engine.')
|
||||
if engine == '':
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
|
|
@ -572,7 +617,10 @@ async def speech(request: Request, user=Depends(get_verified_user)):
|
|||
|
||||
body = await request.body()
|
||||
name = hashlib.sha256(
|
||||
body + str(engine).encode('utf-8') + str(await Config.get('audio.tts.model')).encode('utf-8')
|
||||
body
|
||||
+ str(engine).encode('utf-8')
|
||||
+ str(await Config.get('audio.tts.model')).encode('utf-8')
|
||||
+ (b':slim' if USE_SLIM else b'')
|
||||
).hexdigest()
|
||||
|
||||
file_path = SPEECH_CACHE_DIR.joinpath(f'{name}.mp3')
|
||||
|
|
@ -587,7 +635,11 @@ async def speech(request: Request, user=Depends(get_verified_user)):
|
|||
subject_id=name,
|
||||
data={'engine': engine, 'cached': True},
|
||||
)
|
||||
return FileResponse(file_path)
|
||||
content_type = None
|
||||
if USE_SLIM:
|
||||
async with aiofiles.open(file_path.with_suffix('.mime')) as f:
|
||||
content_type = await f.read()
|
||||
return FileResponse(file_path, media_type=content_type)
|
||||
|
||||
try:
|
||||
payload = JSONCodec.loads(body)
|
||||
|
|
@ -960,7 +1012,11 @@ async def _transcribe_mistral(request, file_path, filename, metadata, file_dir,
|
|||
session = await get_session()
|
||||
if use_chat_completions:
|
||||
audio_file_to_use = file_path
|
||||
if is_audio_conversion_required(file_path):
|
||||
if USE_SLIM and Path(filename).suffix.lower() not in ('.mp3', '.wav'):
|
||||
raise HTTPException(
|
||||
400, 'Mistral chat transcription requires MP3 or WAV in slim; local conversion is unavailable.'
|
||||
)
|
||||
if not BYPASS_PYDUB_PREPROCESSING and is_audio_conversion_required(file_path):
|
||||
log.debug('Converting audio to mp3 for chat completions API')
|
||||
converted_path = await asyncio.to_thread(convert_audio_to_mp3, file_path)
|
||||
if converted_path:
|
||||
|
|
|
|||
|
|
@ -77,8 +77,7 @@ from open_webui.utils.auth import (
|
|||
from open_webui.utils.groups import apply_default_group_assignment
|
||||
from open_webui.utils.misc import parse_duration, validate_email_format
|
||||
from open_webui.utils.rate_limit import RateLimiter
|
||||
from open_webui.utils.redis import get_redis_client
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, StrictStr, field_validator
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
|
@ -88,12 +87,11 @@ log = logging.getLogger(__name__)
|
|||
|
||||
# Forgive us our failed attempts, as we forgive those
|
||||
# who exceed their allotted rate against this gate.
|
||||
signin_rate_limiter = RateLimiter(redis_client=get_redis_client(), limit=5 * 3, window=60 * 3)
|
||||
signin_rate_limiter = RateLimiter(limit=5 * 3, window=60 * 3)
|
||||
# Best-effort throttle only: there is no caller identity before the provider answers,
|
||||
# and deployments may derive request.client from proxy headers.
|
||||
token_exchange_rate_limiter = (
|
||||
RateLimiter(
|
||||
redis_client=get_redis_client(),
|
||||
limit=OAUTH_TOKEN_EXCHANGE_RATE_LIMIT,
|
||||
window=OAUTH_TOKEN_EXCHANGE_RATE_LIMIT_WINDOW,
|
||||
)
|
||||
|
|
@ -106,6 +104,7 @@ ADMIN_CONFIG_KEYS = {
|
|||
'SHOW_ADMIN_DETAILS': 'auth.admin.show',
|
||||
'ADMIN_EMAIL': 'auth.admin.email',
|
||||
'WEBUI_URL': 'webui.url',
|
||||
'ENABLE_LOGIN_FORM': 'ui.enable_login_form',
|
||||
'ENABLE_SIGNUP': 'ui.enable_signup',
|
||||
'ENABLE_API_KEYS': 'auth.enable_api_keys',
|
||||
'ENABLE_API_KEYS_ENDPOINT_RESTRICTIONS': 'auth.api_key.endpoint_restrictions',
|
||||
|
|
@ -113,6 +112,7 @@ ADMIN_CONFIG_KEYS = {
|
|||
'DEFAULT_USER_ROLE': 'ui.default_user_role',
|
||||
'DEFAULT_GROUP_ID': 'ui.default_group_id',
|
||||
'DEFAULT_INTERFACE_SETTINGS': 'ui.default_interface_settings',
|
||||
'I18N': 'ui.i18n',
|
||||
'JWT_EXPIRES_IN': 'auth.jwt_expiry',
|
||||
'ENABLE_COMMUNITY_SHARING': 'ui.enable_community_sharing',
|
||||
'ENABLE_MESSAGE_RATING': 'ui.enable_message_rating',
|
||||
|
|
@ -814,7 +814,7 @@ async def signin(
|
|||
db=db,
|
||||
)
|
||||
else:
|
||||
if signin_rate_limiter.is_limited(form_data.email.lower()):
|
||||
if await signin_rate_limiter.is_limited(request.app.state.redis, form_data.email.lower()):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
detail=ERROR_MESSAGES.RATE_LIMIT_EXCEEDED,
|
||||
|
|
@ -1209,6 +1209,7 @@ class AdminConfig(BaseModel):
|
|||
SHOW_ADMIN_DETAILS: bool
|
||||
ADMIN_EMAIL: str | None = None
|
||||
WEBUI_URL: str
|
||||
ENABLE_LOGIN_FORM: bool = True
|
||||
ENABLE_SIGNUP: bool
|
||||
ENABLE_API_KEYS: bool
|
||||
ENABLE_API_KEYS_ENDPOINT_RESTRICTIONS: bool
|
||||
|
|
@ -1216,6 +1217,7 @@ class AdminConfig(BaseModel):
|
|||
DEFAULT_USER_ROLE: str
|
||||
DEFAULT_GROUP_ID: str
|
||||
DEFAULT_INTERFACE_SETTINGS: dict | None = None
|
||||
I18N: dict[str, dict[str, StrictStr]] | None = None
|
||||
JWT_EXPIRES_IN: str
|
||||
ENABLE_COMMUNITY_SHARING: bool
|
||||
ENABLE_MESSAGE_RATING: bool
|
||||
|
|
@ -1236,10 +1238,40 @@ class AdminConfig(BaseModel):
|
|||
PENDING_USER_OVERLAY_CONTENT: str | None = None
|
||||
RESPONSE_WATERMARK: str | None = None
|
||||
|
||||
@field_validator('I18N')
|
||||
@classmethod
|
||||
def validate_i18n(cls, value):
|
||||
if value is None:
|
||||
raise ValueError('I18N must be a dictionary')
|
||||
unsafe_keys = {'__proto__', 'prototype', 'constructor'}
|
||||
|
||||
def placeholders(text):
|
||||
return {match.strip() for match in re.findall(r'\{\{\s*-?\s*([^},]+)(?:,[^}]+)?\s*\}\}', text)}
|
||||
|
||||
cleaned = {}
|
||||
for locale, entries in value.items():
|
||||
if not locale.strip() or locale in unsafe_keys:
|
||||
raise ValueError(f'Invalid language: {locale}')
|
||||
translations = {}
|
||||
for key, text in entries.items():
|
||||
if not key.strip() or key in unsafe_keys:
|
||||
raise ValueError(f'Invalid translation key: {key}')
|
||||
if text.strip():
|
||||
if placeholders(key) != placeholders(text):
|
||||
raise ValueError(f'Interpolation placeholders do not match: {locale}: {key}')
|
||||
translations[key] = text
|
||||
if translations:
|
||||
cleaned[locale] = translations
|
||||
return cleaned
|
||||
|
||||
|
||||
@router.post('/admin/config')
|
||||
async def update_admin_config(request: Request, form_data: AdminConfig, user=Depends(get_admin_user)):
|
||||
updates = config_updates(form_data.model_dump(), ADMIN_CONFIG_KEYS)
|
||||
if 'ENABLE_LOGIN_FORM' not in form_data.model_fields_set:
|
||||
updates.pop('ui.enable_login_form', None)
|
||||
if 'I18N' not in form_data.model_fields_set:
|
||||
updates.pop('ui.i18n', None)
|
||||
updates['ui.default_interface_settings'] = form_data.DEFAULT_INTERFACE_SETTINGS or {}
|
||||
updates['folders.max_file_count'] = int(form_data.FOLDER_MAX_FILE_COUNT) if form_data.FOLDER_MAX_FILE_COUNT else ''
|
||||
updates['automations.max_count'] = int(form_data.AUTOMATION_MAX_COUNT) if form_data.AUTOMATION_MAX_COUNT else ''
|
||||
|
|
@ -1603,8 +1635,8 @@ async def token_exchange(
|
|||
detail='Token exchange is disabled',
|
||||
)
|
||||
|
||||
if token_exchange_rate_limiter and token_exchange_rate_limiter.is_limited(
|
||||
request.client.host if request.client else 'unknown'
|
||||
if token_exchange_rate_limiter and await token_exchange_rate_limiter.is_limited(
|
||||
request.app.state.redis, request.client.host if request.client else 'unknown'
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
|
|
@ -1720,6 +1752,7 @@ async def token_exchange(
|
|||
user=user,
|
||||
user_data=user_data,
|
||||
provider=provider,
|
||||
access_token=form_data.token,
|
||||
db=db,
|
||||
)
|
||||
if await Config.get('oauth.enable_group_mapping'):
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ async def check_automation_limits(request, user, rrule_str: str, db, is_create:
|
|||
if min_interval:
|
||||
min_interval = int(min_interval)
|
||||
if min_interval > 0:
|
||||
interval = rrule_interval_seconds(rrule_str)
|
||||
interval = await rrule_interval_seconds(rrule_str)
|
||||
if interval is not None and interval < min_interval:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
|
|
@ -150,7 +150,7 @@ async def enrich_automation(automation: AutomationModel, db: AsyncSession, tz: s
|
|||
return AutomationResponse(
|
||||
**automation.model_dump(),
|
||||
last_run=last_run,
|
||||
next_runs=next_n_runs_ns(automation.data['rrule'], tz=tz),
|
||||
next_runs=await next_n_runs_ns(automation.data['rrule'], tz=tz),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -216,7 +216,7 @@ async def create_new_automation(
|
|||
await check_automation_folder_access(form_data.folder_id, user, db)
|
||||
await check_automation_channel_access(form_data, user, db)
|
||||
try:
|
||||
validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
await validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
|
|
@ -226,7 +226,7 @@ async def create_new_automation(
|
|||
await check_automation_limits(request, user, form_data.data.rrule, db, is_create=True)
|
||||
|
||||
tz = user.timezone
|
||||
automation = await Automations.insert(user.id, form_data, next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
automation = await Automations.insert(user.id, form_data, await next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
response = await enrich_automation(automation, db, tz=tz)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -276,7 +276,7 @@ async def update_automation_by_id(
|
|||
await check_automation_channel_access(form_data, user, db)
|
||||
|
||||
try:
|
||||
validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
await validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
|
|
@ -286,7 +286,7 @@ async def update_automation_by_id(
|
|||
await check_automation_limits(request, user, form_data.data.rrule, db, is_create=False)
|
||||
|
||||
tz = user.timezone
|
||||
updated = await Automations.update_by_id(id, form_data, next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
updated = await Automations.update_by_id(id, form_data, await next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
response = await enrich_automation(updated, db, tz=tz)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -313,7 +313,7 @@ async def toggle_automation_by_id(
|
|||
await check_automations_permission(request, user)
|
||||
automation = await Automations.get_by_id(id, db=db)
|
||||
check_automation_access(automation, user)
|
||||
toggled = await Automations.toggle(id, next_run_ns(automation.data['rrule'], tz=user.timezone), db=db)
|
||||
toggled = await Automations.toggle(id, await next_run_ns(automation.data['rrule'], tz=user.timezone), db=db)
|
||||
response = await enrich_automation(toggled, db, tz=user.timezone)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
|
|||
|
|
@ -273,7 +273,10 @@ async def get_events(
|
|||
async def create_event(request: Request, form_data: CalendarEventForm, user: UserModel = Depends(get_verified_user)):
|
||||
await check_calendar_permission(request, user)
|
||||
await _check_calendar_access(form_data.calendar_id, user, 'write')
|
||||
event = await CalendarEvents.insert_new_event(user.id, form_data)
|
||||
try:
|
||||
event = await CalendarEvents.insert_new_event(user.id, form_data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=422, detail=str(e)) from e
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.CALENDAR_EVENT_CREATED,
|
||||
|
|
@ -325,7 +328,10 @@ async def update_event(
|
|||
if form_data.calendar_id is not None and form_data.calendar_id != event.calendar_id:
|
||||
await _check_calendar_access(form_data.calendar_id, user, 'write')
|
||||
|
||||
updated = await CalendarEvents.update_event_by_id(event_id, form_data)
|
||||
try:
|
||||
updated = await CalendarEvents.update_event_by_id(event_id, form_data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=422, detail=str(e)) from e
|
||||
if not updated:
|
||||
raise HTTPException(status_code=500, detail='Failed to update')
|
||||
await publish_event(
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ from fastapi.responses import FileResponse, Response, StreamingResponse
|
|||
from open_webui.config import ENABLE_ADMIN_CHAT_ACCESS, ENABLE_ADMIN_EXPORT
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.env import STATIC_DIR
|
||||
from open_webui.env import ENABLE_PROFILE_IMAGE_URL_FORWARDING, STATIC_DIR
|
||||
from open_webui.internal.db import get_async_session
|
||||
from open_webui.models.access_grants import AccessGrants, has_public_read_access_grant, has_public_write_access_grant
|
||||
from open_webui.models.config import Config
|
||||
|
|
@ -1102,13 +1102,12 @@ async def model_response_handler(request, channel, message, user, db=None):
|
|||
# Resolve model config (same path automations use)
|
||||
from open_webui.utils.automations import _resolve_model_defaults
|
||||
|
||||
tool_ids, features, filter_ids, _ = await _resolve_model_defaults(request.app, model_id)
|
||||
|
||||
# Build full form_data — same shape as frontend POST.
|
||||
# The channel: prefix routes pipeline events to the
|
||||
# channel emitter in socket/main.py instead of the
|
||||
# default chat emitter.
|
||||
form_data = {
|
||||
**await _resolve_model_defaults(request.app, model_id),
|
||||
'model': model_id,
|
||||
'messages': [
|
||||
system_message,
|
||||
|
|
@ -1122,12 +1121,6 @@ async def model_response_handler(request, channel, message, user, db=None):
|
|||
}
|
||||
if files:
|
||||
form_data['files'] = files
|
||||
if tool_ids:
|
||||
form_data['tool_ids'] = tool_ids
|
||||
if features:
|
||||
form_data['features'] = features
|
||||
if filter_ids:
|
||||
form_data['filter_ids'] = filter_ids
|
||||
|
||||
# Call the full chat completion pipeline — streaming,
|
||||
# tools, filters, RAG — everything. The pipeline runs as
|
||||
|
|
@ -1823,9 +1816,15 @@ async def delete_message_by_id(
|
|||
|
||||
|
||||
@router.get('/webhooks/{webhook_id}/profile/image')
|
||||
async def get_webhook_profile_image(webhook_id: str, user=Depends(get_verified_user)):
|
||||
async def get_webhook_profile_image(
|
||||
request: Request,
|
||||
webhook_id: str,
|
||||
user=Depends(get_verified_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
"""Get webhook profile image by webhook ID."""
|
||||
webhook = await Channels.get_webhook_by_id(webhook_id)
|
||||
await check_channels_access(request, user)
|
||||
webhook = await Channels.get_webhook_by_id(webhook_id, db=db)
|
||||
if not webhook:
|
||||
# Return default favicon if webhook not found
|
||||
# LICENSE covers this Open WebUI fallback logo.
|
||||
|
|
@ -1833,13 +1832,26 @@ async def get_webhook_profile_image(webhook_id: str, user=Depends(get_verified_u
|
|||
# https://docs.openwebui.com/license.
|
||||
return FileResponse(f'{STATIC_DIR}/favicon.png')
|
||||
|
||||
channel = await Channels.get_channel_by_id(webhook.channel_id, db=db)
|
||||
if not channel:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=ERROR_MESSAGES.NOT_FOUND)
|
||||
|
||||
if channel.type in ['group', 'dm']:
|
||||
if not await Channels.is_user_channel_member(channel.id, user.id, db=db):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.DEFAULT())
|
||||
else:
|
||||
if user.role != 'admin' and not await channel_has_access(user.id, channel, permission='read', db=db):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.DEFAULT())
|
||||
|
||||
if webhook.profile_image_url:
|
||||
# Check if it's url or base64
|
||||
if webhook.profile_image_url.startswith('http'):
|
||||
return Response(
|
||||
status_code=status.HTTP_302_FOUND,
|
||||
headers={'Location': webhook.profile_image_url},
|
||||
)
|
||||
if ENABLE_PROFILE_IMAGE_URL_FORWARDING:
|
||||
return Response(
|
||||
status_code=status.HTTP_302_FOUND,
|
||||
headers={'Location': webhook.profile_image_url},
|
||||
)
|
||||
# When forwarding is disabled, fall through to the default image to prevent client-side IP/UA/Referer leaks.
|
||||
elif webhook.profile_image_url.startswith('data:image'):
|
||||
try:
|
||||
header, base64_data = webhook.profile_image_url.split(',', 1)
|
||||
|
|
|
|||
|
|
@ -888,6 +888,7 @@ async def search_user_chats(
|
|||
created_at=chat.created_at,
|
||||
last_read_at=chat.last_read_at,
|
||||
snippet=chat_search_snippet(chat.chat, search_text),
|
||||
archived=chat.archived,
|
||||
)
|
||||
)
|
||||
|
||||
|
|
@ -1592,7 +1593,7 @@ async def delete_chat_by_id(
|
|||
# Cancel any in-flight LLM tasks (streaming, title/tags generation) before
|
||||
# deleting the chat to prevent orphaned requests.
|
||||
await stop_item_tasks(request.app.state.redis, id)
|
||||
await Chats.delete_orphan_tags_for_user(chat.meta.get('tags', []), user.id, threshold=1, db=db)
|
||||
await Chats.delete_orphan_tags_for_user(chat.meta.get('tags', []), chat.user_id, threshold=1, db=db)
|
||||
|
||||
# Cascade to internal child chats spawned from this one.
|
||||
for child_id in await Chats.get_internal_chat_ids_by_parent_id(id, chat.user_id):
|
||||
|
|
@ -1690,15 +1691,6 @@ async def fork_chat_by_id(
|
|||
|
||||
history = (chat.chat or {}).get('history') or {}
|
||||
messages_map = await Chats.get_messages_map_by_chat_id(id) or history.get('messages') or {}
|
||||
if any(
|
||||
message.get('role') == 'assistant' and message.get('done') is False
|
||||
for message in messages_map.values()
|
||||
if isinstance(message, dict)
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail='Wait for the current response to finish before forking.',
|
||||
)
|
||||
|
||||
source_message_id = (
|
||||
(form_data.message_id if form_data else None) or chat.current_message_id or history.get('currentId')
|
||||
|
|
@ -1715,6 +1707,22 @@ async def fork_chat_by_id(
|
|||
detail=detail,
|
||||
) from exc
|
||||
|
||||
# An unfinished message is stale unless it is awaiting tool approval
|
||||
for message in fork_history['messages'].values():
|
||||
if message.get('role') != 'assistant' or message.get('done') is not False:
|
||||
continue
|
||||
|
||||
output = message.get('output')
|
||||
if isinstance(output, list) and any(
|
||||
isinstance(item, dict)
|
||||
and item.get('type') == 'function_call'
|
||||
and item.get('status') in {'pending', 'queued', 'requires_approval'}
|
||||
for item in output
|
||||
):
|
||||
continue
|
||||
|
||||
message['done'] = True
|
||||
|
||||
updated_chat = {**(chat.chat or {})}
|
||||
updated_chat.pop('currentId', None)
|
||||
updated_chat.update(
|
||||
|
|
@ -1732,10 +1740,15 @@ async def fork_chat_by_id(
|
|||
'forked_from_message_id': source_message_id,
|
||||
}
|
||||
|
||||
# The source chat's folder may no longer be writable by the caller.
|
||||
folder_id = chat.folder_id
|
||||
if folder_id is not None and not await has_folder_write_access(user.id, folder_id, db=db):
|
||||
folder_id = None
|
||||
|
||||
fork = await Chats.insert_new_chat(
|
||||
str(uuid4()),
|
||||
user.id,
|
||||
ChatForm(chat=updated_chat, folder_id=chat.folder_id),
|
||||
ChatForm(chat=updated_chat, folder_id=folder_id),
|
||||
db=db,
|
||||
internal_meta=meta,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
from typing import Optional
|
||||
from typing import Any, Optional
|
||||
|
||||
import aiohttp
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
|
|
@ -40,6 +39,7 @@ log = logging.getLogger(__name__)
|
|||
|
||||
CONNECTIONS_CONFIG_KEYS = {
|
||||
'ENABLE_DIRECT_CONNECTIONS': 'direct.enable',
|
||||
'ENABLE_DIRECT_INTEGRATIONS': 'direct.integrations.enable',
|
||||
'ENABLE_BASE_MODELS_CACHE': 'models.base_models_cache',
|
||||
}
|
||||
CODE_EXECUTION_CONFIG_KEYS = {
|
||||
|
|
@ -132,6 +132,7 @@ async def get_config_namespace(namespace: str, user=Depends(get_admin_user)):
|
|||
|
||||
class ConnectionsConfigForm(BaseModel):
|
||||
ENABLE_DIRECT_CONNECTIONS: bool
|
||||
ENABLE_DIRECT_INTEGRATIONS: bool = False
|
||||
ENABLE_BASE_MODELS_CACHE: bool
|
||||
|
||||
|
||||
|
|
@ -146,7 +147,7 @@ async def set_connections_config(
|
|||
form_data: ConnectionsConfigForm,
|
||||
user=Depends(get_admin_user),
|
||||
):
|
||||
await Config.upsert(config_updates(form_data.model_dump(), CONNECTIONS_CONFIG_KEYS))
|
||||
await Config.upsert(config_updates(form_data.model_dump(exclude_unset=True), CONNECTIONS_CONFIG_KEYS))
|
||||
values = await get_config_values(CONNECTIONS_CONFIG_KEYS)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -218,6 +219,7 @@ class ToolServerConnection(BaseModel):
|
|||
path: str
|
||||
type: str | None = 'openapi' # openapi, mcp
|
||||
auth_type: str | None
|
||||
forward_cookies: bool = False
|
||||
headers: dict | str | None = None
|
||||
key: str | None
|
||||
config: dict | None
|
||||
|
|
@ -307,6 +309,7 @@ class TerminalServerConnection(BaseModel):
|
|||
|
||||
key: str | None = ''
|
||||
auth_type: str | None = 'bearer'
|
||||
forward_cookies: bool = False
|
||||
|
||||
config: dict | None = None
|
||||
|
||||
|
|
@ -808,17 +811,24 @@ class PromptSuggestion(BaseModel):
|
|||
|
||||
class SetDefaultSuggestionsForm(BaseModel):
|
||||
suggestions: list[PromptSuggestion]
|
||||
i18n: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@router.post('/suggestions', response_model=list[PromptSuggestion])
|
||||
@router.post('/suggestions', response_model=dict)
|
||||
async def set_default_suggestions(
|
||||
request: Request,
|
||||
form_data: SetDefaultSuggestionsForm,
|
||||
user=Depends(get_admin_user),
|
||||
):
|
||||
data = form_data.model_dump()
|
||||
await Config.upsert({'ui.prompt_suggestions': data['suggestions']})
|
||||
await Config.upsert(
|
||||
{
|
||||
'ui.prompt_suggestions': data['suggestions'],
|
||||
'ui.prompt_suggestions_i18n': data.get('i18n') or {},
|
||||
}
|
||||
)
|
||||
suggestions = await Config.get('ui.prompt_suggestions')
|
||||
suggestions_i18n = await Config.get('ui.prompt_suggestions_i18n')
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.CONFIG_SUGGESTIONS_UPDATED,
|
||||
|
|
@ -827,7 +837,7 @@ async def set_default_suggestions(
|
|||
subject_type='config',
|
||||
data={'count': len(suggestions or [])},
|
||||
)
|
||||
return suggestions
|
||||
return {'suggestions': suggestions, 'i18n': suggestions_i18n}
|
||||
|
||||
|
||||
############################
|
||||
|
|
|
|||
|
|
@ -4,7 +4,10 @@ from typing import Optional
|
|||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.env import MPS_INFERENCE_LOCK
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.env import USE_SLIM
|
||||
from open_webui.retrieval.utils import cosine_similarity
|
||||
from open_webui.internal.db import get_async_session
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.models.feedbacks import (
|
||||
|
|
@ -68,6 +71,8 @@ _embedding_model = None
|
|||
|
||||
def _get_embedding_model():
|
||||
global _embedding_model
|
||||
if USE_SLIM:
|
||||
return None
|
||||
if _embedding_model is None:
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
|
@ -179,8 +184,9 @@ def _compute_similarities(feedbacks: list[LeaderboardFeedbackData], query: str)
|
|||
return {}
|
||||
|
||||
try:
|
||||
tag_embeddings = embedding_model.encode(all_tags)
|
||||
query_embedding = embedding_model.encode([query])[0]
|
||||
with MPS_INFERENCE_LOCK:
|
||||
tag_embeddings = embedding_model.encode(all_tags)
|
||||
query_embedding = embedding_model.encode([query])[0]
|
||||
except Exception as e:
|
||||
log.error(f'Embedding error: {e}')
|
||||
return {}
|
||||
|
|
@ -215,6 +221,7 @@ class LeaderboardResponse(BaseModel):
|
|||
|
||||
@router.get('/leaderboard', response_model=LeaderboardResponse)
|
||||
async def get_leaderboard(
|
||||
request: Request,
|
||||
query: Optional[str] = None,
|
||||
user=Depends(get_admin_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
|
|
@ -224,7 +231,17 @@ async def get_leaderboard(
|
|||
|
||||
similarities = None
|
||||
if query and query.strip():
|
||||
similarities = await run_in_threadpool(_compute_similarities, feedbacks, query.strip())
|
||||
if USE_SLIM:
|
||||
tags = list({tag for feedback in feedbacks for tag in (feedback.data or {}).get('tags', [])})
|
||||
embeddings = await request.app.state.EMBEDDING_FUNCTION([query.strip(), *tags], user=user)
|
||||
scores = cosine_similarity(embeddings[0], embeddings[1:])
|
||||
tag_scores = dict(zip(tags, scores.tolist()))
|
||||
similarities = {
|
||||
feedback.id: max((tag_scores.get(tag, 0) for tag in (feedback.data or {}).get('tags', [])), default=0)
|
||||
for feedback in feedbacks
|
||||
}
|
||||
else:
|
||||
similarities = await run_in_threadpool(_compute_similarities, feedbacks, query.strip())
|
||||
|
||||
elo_stats = _calculate_elo(feedbacks, similarities)
|
||||
tags_by_model = _get_top_tags(feedbacks)
|
||||
|
|
|
|||
|
|
@ -461,7 +461,11 @@ async def upload_file_handler(
|
|||
log.exception(e)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=ERROR_MESSAGES.DEFAULT('Error uploading file'),
|
||||
detail=(
|
||||
ERROR_MESSAGES.EMPTY_CONTENT
|
||||
if isinstance(e, ValueError) and e.args == (ERROR_MESSAGES.EMPTY_CONTENT,)
|
||||
else ERROR_MESSAGES.DEFAULT('Error uploading file')
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -401,7 +401,7 @@ async def update_folder_parent_id_by_id(
|
|||
form_data.parent_id, user.id, folder.name, db=db
|
||||
)
|
||||
|
||||
if existing_folder:
|
||||
if existing_folder and existing_folder.id != id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=ERROR_MESSAGES.DEFAULT('Folder already exists'),
|
||||
|
|
@ -683,7 +683,7 @@ async def delete_folder_by_id(
|
|||
folder_owner_id = folder.user_id
|
||||
|
||||
folder_ids = await Folders.get_folder_ids_by_id_and_user_id_in_subtree(id, folder_owner_id, db=db)
|
||||
if await Chats.count_chats_by_folder_ids_and_user_id(folder_ids, folder_owner_id, db=db):
|
||||
if delete_contents and await Chats.count_chats_by_folder_ids_and_user_id(folder_ids, folder_owner_id, db=db):
|
||||
chat_delete_permission = await has_permission(
|
||||
user.id, 'chat.delete', await Config.get('user.permissions'), db=db
|
||||
)
|
||||
|
|
@ -704,8 +704,8 @@ async def delete_folder_by_id(
|
|||
for folder_id in folder_ids:
|
||||
if delete_contents:
|
||||
await Chats.delete_chats_by_user_id_and_folder_id(folder_owner_id, folder_id, db=db)
|
||||
else:
|
||||
await Chats.move_chats_by_user_id_and_folder_id(folder_owner_id, folder_id, None, db=db)
|
||||
|
||||
await Chats.move_chats_by_folder_id(folder_id, None, db=db)
|
||||
|
||||
# Clean up access grants for this folder
|
||||
await AccessGrants.revoke_all_access('folder', folder_id, db=db)
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ from open_webui.models.functions import (
|
|||
)
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.plugin import (
|
||||
get_function_contents_cache,
|
||||
get_functions_cache,
|
||||
get_function_module_from_cache,
|
||||
load_function_module_by_id,
|
||||
|
|
@ -440,6 +441,8 @@ async def delete_function_by_id(
|
|||
if result:
|
||||
FUNCTIONS = get_functions_cache(request)
|
||||
FUNCTIONS.pop(id, None)
|
||||
FUNCTION_CONTENTS = get_function_contents_cache(request)
|
||||
FUNCTION_CONTENTS.pop(id, None)
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.FUNCTION_DELETED,
|
||||
|
|
|
|||
|
|
@ -329,38 +329,34 @@ def get_automatic1111_api_auth(image_config):
|
|||
return f'Basic {auth1111_base64_encoded_string}'
|
||||
|
||||
|
||||
@router.get('/config/url/verify')
|
||||
async def verify_url(request: Request, user=Depends(get_admin_user)):
|
||||
image_config = await get_image_config()
|
||||
if image_config.IMAGE_GENERATION_ENGINE == 'automatic1111':
|
||||
try:
|
||||
session = await get_session()
|
||||
async with session.get(
|
||||
url=f'{image_config.AUTOMATIC1111_BASE_URL}/sdapi/v1/options',
|
||||
headers={'authorization': get_automatic1111_api_auth(image_config)},
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
return True
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail=ERROR_MESSAGES.INVALID_URL)
|
||||
elif image_config.IMAGE_GENERATION_ENGINE == 'comfyui':
|
||||
headers = None
|
||||
if image_config.COMFYUI_API_KEY:
|
||||
headers = {'Authorization': f'Bearer {image_config.COMFYUI_API_KEY}'}
|
||||
try:
|
||||
session = await get_session()
|
||||
async with session.get(
|
||||
url=f'{image_config.COMFYUI_BASE_URL}/object_info',
|
||||
headers=headers,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
return True
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail=ERROR_MESSAGES.INVALID_URL)
|
||||
class ConnectionVerificationForm(BaseModel):
|
||||
engine: str
|
||||
url: str
|
||||
key: str | None = None
|
||||
|
||||
|
||||
@router.post('/verify')
|
||||
async def verify_connection(form_data: ConnectionVerificationForm, user=Depends(get_admin_user)):
|
||||
url = form_data.url.rstrip('/')
|
||||
headers = {}
|
||||
if form_data.engine == 'automatic1111':
|
||||
url = f'{url}/sdapi/v1/options'
|
||||
if form_data.key is not None:
|
||||
headers['Authorization'] = f'Basic {base64.b64encode(form_data.key.encode("utf-8")).decode("utf-8")}'
|
||||
elif form_data.engine == 'comfyui':
|
||||
url = f'{url}/object_info'
|
||||
if form_data.key:
|
||||
headers['Authorization'] = f'Bearer {form_data.key}'
|
||||
else:
|
||||
return True
|
||||
raise HTTPException(status_code=400, detail='Unsupported image engine')
|
||||
|
||||
try:
|
||||
session = await get_session()
|
||||
async with session.get(url=url, headers=headers, ssl=AIOHTTP_CLIENT_SESSION_SSL) as r:
|
||||
r.raise_for_status()
|
||||
return True
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail=ERROR_MESSAGES.INVALID_URL)
|
||||
|
||||
|
||||
@router.get('/models')
|
||||
|
|
@ -436,7 +432,10 @@ async def get_models(request: Request, user=Depends(get_verified_user)):
|
|||
)
|
||||
)
|
||||
except Exception as e:
|
||||
log.exception(f'Failed to list image generation models: {e}')
|
||||
log.error(
|
||||
f'Failed to list image generation models: {str(e) or type(e).__name__}',
|
||||
exc_info=not isinstance(e, (aiohttp.ClientConnectionError, TimeoutError)),
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=ERROR_MESSAGES.DEFAULT(e, 'Failed to retrieve image generation models'),
|
||||
|
|
@ -668,7 +667,9 @@ async def image_generations(
|
|||
if image_url := image.get('url', None):
|
||||
image_data, content_type = await get_image_data(
|
||||
image_url,
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'},
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'}
|
||||
if _is_same_origin(image_url, image_config.IMAGES_OPENAI_API_BASE_URL)
|
||||
else None,
|
||||
)
|
||||
else:
|
||||
image_data, content_type = await get_image_data(image['b64_json'])
|
||||
|
|
@ -919,11 +920,8 @@ async def image_edits(
|
|||
|
||||
if data.startswith('http://') or data.startswith('https://'):
|
||||
parsed = urlparse(data)
|
||||
if (
|
||||
parsed.netloc == urlparse(str(request.base_url)).netloc
|
||||
and parsed.path.startswith('/api/v1/files/')
|
||||
and '/content' in parsed.path
|
||||
):
|
||||
# Fetching /api/v1/files/{id}/content over the network would be unauthenticated.
|
||||
if parsed.path.startswith('/api/v1/files/') and '/content' in parsed.path:
|
||||
return await load_url_image(parsed.path)
|
||||
|
||||
# Validate URL to prevent SSRF attacks against local/private networks.
|
||||
|
|
@ -1047,7 +1045,9 @@ async def image_edits(
|
|||
if image_url := image.get('url', None):
|
||||
image_data, content_type = await get_image_data(
|
||||
image_url,
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'},
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'}
|
||||
if _is_same_origin(image_url, image_config.IMAGES_EDIT_OPENAI_API_BASE_URL)
|
||||
else None,
|
||||
)
|
||||
else:
|
||||
image_data, content_type = await get_image_data(image['b64_json'])
|
||||
|
|
|
|||
|
|
@ -1642,13 +1642,9 @@ async def remove_file_from_knowledge_by_id(
|
|||
|
||||
# Remove content from the vector database
|
||||
try:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(
|
||||
collection_name=knowledge.id, filter={'file_id': form_data.file_id}
|
||||
) # Remove by file_id first
|
||||
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(
|
||||
collection_name=knowledge.id, filter={'hash': file.hash}
|
||||
) # Remove by hash as well in case of duplicates
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=knowledge.id, filter={'file_id': form_data.file_id})
|
||||
if file.hash:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=knowledge.id, filter={'hash': file.hash})
|
||||
except Exception as e:
|
||||
log.debug('This was most likely caused by bypassing embedding processing')
|
||||
log.debug(e)
|
||||
|
|
@ -1988,7 +1984,8 @@ async def sync_knowledge_cleanup(
|
|||
|
||||
try:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'file_id': file_id})
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'hash': file.hash})
|
||||
if file.hash:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'hash': file.hash})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
|
@ -2321,6 +2318,9 @@ async def delete_knowledge_directory(
|
|||
detail=ERROR_MESSAGES.NOT_FOUND,
|
||||
)
|
||||
|
||||
# Collect before delete_directory drops the KnowledgeFile rows
|
||||
files = [] if move_files else await Knowledges.get_files_by_id_and_directory_id(id, dir_id, db=db)
|
||||
|
||||
success = await Knowledges.delete_directory(
|
||||
directory_id=dir_id,
|
||||
move_files_to_parent=move_files,
|
||||
|
|
@ -2331,6 +2331,23 @@ async def delete_knowledge_directory(
|
|||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail='Failed to delete directory.',
|
||||
)
|
||||
|
||||
for file in files:
|
||||
try:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'file_id': file.id})
|
||||
if file.hash:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'hash': file.hash})
|
||||
except Exception as e:
|
||||
log.debug('This was most likely caused by bypassing embedding processing')
|
||||
log.debug(e)
|
||||
|
||||
if (
|
||||
not ENABLE_KNOWLEDGE_FILE_RETENTION
|
||||
and not await Knowledges.get_knowledges_by_file_id(file.id, db=db)
|
||||
and (file.user_id == user.id or user.role == 'admin')
|
||||
):
|
||||
await delete_file_resource(file, db)
|
||||
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.KNOWLEDGE_DIRECTORY_DELETED,
|
||||
|
|
|
|||
|
|
@ -241,6 +241,7 @@ async def update_memories(
|
|||
|
||||
operations = validate_memory_operations(form_data)
|
||||
metadata = getattr(request.state, 'metadata', {}) or {}
|
||||
model = metadata.get('model')
|
||||
source = form_data.source or 'tool'
|
||||
for operation in operations:
|
||||
if operation.get('action') in {'add', 'replace', 'move'}:
|
||||
|
|
@ -248,7 +249,7 @@ async def update_memories(
|
|||
'created_by': source,
|
||||
'chat_id': metadata.get('chat_id'),
|
||||
'message_id': metadata.get('message_id'),
|
||||
'model': metadata.get('model'),
|
||||
'model': model.get('id') if isinstance(model, dict) else None,
|
||||
}
|
||||
|
||||
try:
|
||||
|
|
@ -263,7 +264,7 @@ async def update_memories(
|
|||
for result in results:
|
||||
memory = result.get('memory')
|
||||
if isinstance(memory, MemoryModel):
|
||||
result = {**result, 'memory': memory.model_dump()}
|
||||
result = {**result, 'memory': memory.model_dump(exclude={'meta'})}
|
||||
if result.get('status') in {'created', 'updated'}:
|
||||
vector = await request.app.state.EMBEDDING_FUNCTION(
|
||||
memory_vector_text(memory.content, memory.path),
|
||||
|
|
@ -438,7 +439,7 @@ async def read_memory_path(
|
|||
)
|
||||
return {
|
||||
**result,
|
||||
'memories': [memory.model_dump() for memory in result['memories']],
|
||||
'memories': [memory.model_dump(exclude={'meta'}) for memory in result['memories']],
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ from fastapi import (
|
|||
APIRouter,
|
||||
Depends,
|
||||
HTTPException,
|
||||
Query,
|
||||
Request,
|
||||
Response,
|
||||
status,
|
||||
|
|
@ -19,11 +20,16 @@ from fastapi import (
|
|||
from fastapi.responses import RedirectResponse, StreamingResponse
|
||||
from open_webui.config import BYPASS_ADMIN_ACCESS_CONTROL
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.env import ENABLE_PROFILE_IMAGE_URL_FORWARDING, PROFILE_IMAGE_ALLOWED_MIME_TYPES
|
||||
from open_webui.env import (
|
||||
BYPASS_MODEL_ACCESS_CONTROL,
|
||||
ENABLE_PROFILE_IMAGE_URL_FORWARDING,
|
||||
PROFILE_IMAGE_ALLOWED_MIME_TYPES,
|
||||
)
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.internal.db import get_async_session
|
||||
from open_webui.models.access_grants import AccessGrants
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.models.files import Files
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.models import (
|
||||
ModelAccessListResponse,
|
||||
|
|
@ -36,12 +42,14 @@ from open_webui.models.models import (
|
|||
ModelResponse,
|
||||
Models,
|
||||
)
|
||||
from open_webui.utils.access_control import filter_allowed_access_grants, has_permission
|
||||
from open_webui.storage.provider import Storage
|
||||
from open_webui.utils.access_control import filter_allowed_access_grants, has_access, has_permission
|
||||
from open_webui.utils.access_control.files import has_access_to_file
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.chat_variables import get_chat_variables_schema
|
||||
from open_webui.utils.models import get_all_models
|
||||
from pydantic import BaseModel
|
||||
from open_webui.utils.validate import BACKGROUND_IMAGE_MAX_BYTES, validate_background_image
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -89,7 +97,28 @@ def _safe_static_redirect_path(url: str) -> str | None:
|
|||
|
||||
|
||||
def is_valid_model_id(model_id: str) -> bool:
|
||||
return model_id and len(model_id) <= 256
|
||||
return model_id and len(model_id) <= 256 and not any(char.isspace() for char in model_id)
|
||||
|
||||
|
||||
async def _verify_background_image(url: str | None, user, db, previous_url: str | None = None) -> None:
|
||||
if not url or url == previous_url:
|
||||
return
|
||||
file_id = url.split('/')[-2]
|
||||
file = await Files.get_file_by_id(file_id, db=db)
|
||||
if not file or not (
|
||||
user.role == 'admin' or file.user_id == user.id or await has_access_to_file(file_id, 'read', user, db=db)
|
||||
):
|
||||
raise HTTPException(status_code=403, detail='Background image is not accessible.')
|
||||
try:
|
||||
path = await asyncio.to_thread(Storage.get_file, file.path)
|
||||
with open(path, 'rb') as image:
|
||||
data = await asyncio.to_thread(image.read, BACKGROUND_IMAGE_MAX_BYTES + 1)
|
||||
content_type = await asyncio.to_thread(validate_background_image, data)
|
||||
except (ValueError, OSError) as error:
|
||||
raise HTTPException(status_code=400, detail=str(error)) from error
|
||||
if (file.meta or {}).get('content_type') != content_type:
|
||||
if not await Files.update_file_metadata_by_id(file_id, {'content_type': content_type}, db=db):
|
||||
raise HTTPException(status_code=500, detail='Could not validate background image.')
|
||||
|
||||
|
||||
async def _verify_knowledge_file_access(
|
||||
|
|
@ -211,6 +240,11 @@ async def get_models(
|
|||
###########################
|
||||
|
||||
|
||||
@router.get('/all', response_model=list[ModelResponse])
|
||||
async def get_all_model_records(user=Depends(get_admin_user), db: AsyncSession = Depends(get_async_session)):
|
||||
return await Models.get_all_models(db=db)
|
||||
|
||||
|
||||
@router.get('/base/tags', response_model=list[str])
|
||||
async def get_base_model_tags(user=Depends(get_admin_user), db: AsyncSession = Depends(get_async_session)):
|
||||
tags = await Models.get_all_tags(user_id=user.id, is_admin=True, is_base_model=True, db=db)
|
||||
|
|
@ -306,6 +340,8 @@ async def create_new_model(
|
|||
db,
|
||||
)
|
||||
|
||||
await _verify_background_image(form_data.meta.background_image_url, user, db)
|
||||
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
|
|
@ -336,9 +372,14 @@ async def create_new_model(
|
|||
############################
|
||||
|
||||
|
||||
@router.get('/export', response_model=list[ModelModel])
|
||||
class ModelExportResponse(ModelModel):
|
||||
background_image_data: str | None = None
|
||||
|
||||
|
||||
@router.get('/export', response_model=list[ModelExportResponse])
|
||||
async def export_models(
|
||||
request: Request,
|
||||
ids: list[str] | None = Query(None),
|
||||
user=Depends(get_verified_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
|
|
@ -354,9 +395,36 @@ async def export_models(
|
|||
)
|
||||
|
||||
if user.role == 'admin' and BYPASS_ADMIN_ACCESS_CONTROL:
|
||||
return await Models.get_models(db=db)
|
||||
models = await Models.get_models(db=db, ids=ids)
|
||||
else:
|
||||
return await Models.get_models(writable_by_user_id=user.id, db=db)
|
||||
models = await Models.get_models(writable_by_user_id=user.id, db=db, ids=ids)
|
||||
if ids is not None:
|
||||
requested = set(ids)
|
||||
if requested != {model.id for model in models}:
|
||||
raise HTTPException(status_code=403, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
exported = []
|
||||
for model in models:
|
||||
data = model.model_dump()
|
||||
url = model.meta.background_image_url
|
||||
if url:
|
||||
try:
|
||||
file = await Files.get_file_by_id(url.split('/')[-2], db=db)
|
||||
if not file:
|
||||
raise ValueError('Image file is missing')
|
||||
path = await asyncio.to_thread(Storage.get_file, file.path)
|
||||
with open(path, 'rb') as image:
|
||||
image_data = await asyncio.to_thread(image.read, BACKGROUND_IMAGE_MAX_BYTES + 1)
|
||||
content_type = await asyncio.to_thread(validate_background_image, image_data)
|
||||
data['background_image_data'] = f'data:{content_type};base64,' + base64.b64encode(image_data).decode(
|
||||
'ascii'
|
||||
)
|
||||
data['meta']['background_image_url'] = None
|
||||
except Exception as error:
|
||||
raise HTTPException(
|
||||
status_code=400, detail=f'Could not export background for model {model.id}.'
|
||||
) from error
|
||||
exported.append(data)
|
||||
return exported
|
||||
|
||||
|
||||
############################
|
||||
|
|
@ -486,7 +554,7 @@ async def import_models(
|
|||
updated_model.access_grants,
|
||||
'sharing.public_models',
|
||||
)
|
||||
await Models.update_model_by_id(model_id, updated_model, db=db)
|
||||
imported_model = updated_model
|
||||
else:
|
||||
# Insert new model
|
||||
model_data['meta'] = model_data.get('meta', {})
|
||||
|
|
@ -530,7 +598,58 @@ async def import_models(
|
|||
new_model.access_grants,
|
||||
'sharing.public_models',
|
||||
)
|
||||
await Models.insert_new_model(user_id=user.id, form_data=new_model, db=db)
|
||||
imported_model = new_model
|
||||
|
||||
uploaded = None
|
||||
try:
|
||||
encoded = model_data.pop('background_image_data', None)
|
||||
if encoded is not None:
|
||||
if (
|
||||
not isinstance(encoded, str)
|
||||
or len(encoded) > 4 * ((BACKGROUND_IMAGE_MAX_BYTES + 2) // 3) + 64
|
||||
):
|
||||
raise ValueError('Background image must be at most 5 MiB.')
|
||||
header, payload = encoded.split(',', 1)
|
||||
image_data = base64.b64decode(payload, validate=True)
|
||||
content_type = await asyncio.to_thread(validate_background_image, image_data)
|
||||
if header != f'data:{content_type};base64':
|
||||
raise ValueError('Invalid background image data URI.')
|
||||
from fastapi import UploadFile
|
||||
from open_webui.routers.files import upload_file_handler
|
||||
|
||||
uploaded = await upload_file_handler(
|
||||
request,
|
||||
file=UploadFile(
|
||||
file=io.BytesIO(image_data),
|
||||
filename='background.' + content_type.split('/')[1],
|
||||
),
|
||||
metadata=None,
|
||||
process=False,
|
||||
user=user,
|
||||
db=db,
|
||||
)
|
||||
imported_model.meta.background_image_url = f'/api/v1/files/{uploaded.id}/content'
|
||||
await _verify_background_image(
|
||||
imported_model.meta.background_image_url,
|
||||
user,
|
||||
db,
|
||||
existing_model.meta.background_image_url if existing_model else None,
|
||||
)
|
||||
saved = (
|
||||
await Models.update_model_by_id(model_id, imported_model, db=db)
|
||||
if existing_model
|
||||
else await Models.insert_new_model(user_id=user.id, form_data=imported_model, db=db)
|
||||
)
|
||||
if not saved:
|
||||
raise HTTPException(status_code=500, detail=f'Could not import model {model_id}.')
|
||||
except Exception:
|
||||
if uploaded:
|
||||
try:
|
||||
await Files.delete_file_by_id(uploaded.id, db=db)
|
||||
await asyncio.to_thread(Storage.delete_file, uploaded.path)
|
||||
except Exception:
|
||||
log.exception('Could not clean up failed model background upload')
|
||||
raise
|
||||
|
||||
imported_ids.append(model_id)
|
||||
await publish_event(
|
||||
|
|
@ -543,6 +662,10 @@ async def import_models(
|
|||
return True
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail='Invalid JSON format')
|
||||
except HTTPException:
|
||||
raise
|
||||
except ValueError as error:
|
||||
raise HTTPException(status_code=400, detail=str(error)) from error
|
||||
except Exception as e:
|
||||
log.exception(e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
|
@ -564,6 +687,14 @@ async def sync_models(
|
|||
user=Depends(get_admin_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
existing = {model.id: model for model in await Models.get_models_by_ids([m.id for m in form_data.models], db=db)}
|
||||
for model in form_data.models:
|
||||
previous = existing.get(model.id)
|
||||
if previous and 'background_image_url' not in model.meta.model_fields_set:
|
||||
model.meta.background_image_url = previous.meta.background_image_url
|
||||
await _verify_background_image(
|
||||
model.meta.background_image_url, user, db, previous.meta.background_image_url if previous else None
|
||||
)
|
||||
models = await Models.sync_models(user.id, form_data.models, db=db)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -649,18 +780,37 @@ async def get_model_profile_image(
|
|||
profile_image_url = None
|
||||
updated_at = None
|
||||
|
||||
bypass_access_control = BYPASS_MODEL_ACCESS_CONTROL or (user.role == 'admin' and BYPASS_ADMIN_ACCESS_CONTROL)
|
||||
|
||||
# First, check the database for regular models
|
||||
model_meta = await Models.get_model_meta_by_id(id, db=db)
|
||||
if model_meta:
|
||||
meta, updated_at = model_meta
|
||||
profile_image_url = (meta or {}).get('profile_image_url')
|
||||
meta, model_user_id, model_updated_at = model_meta
|
||||
# Denied callers get the default image rather than an error, so model ids stay unprobeable.
|
||||
if (
|
||||
bypass_access_control
|
||||
or user.id == model_user_id
|
||||
or await AccessGrants.has_access(
|
||||
user_id=user.id,
|
||||
resource_type='model',
|
||||
resource_id=id,
|
||||
permission='read',
|
||||
db=db,
|
||||
)
|
||||
):
|
||||
profile_image_url = (meta or {}).get('profile_image_url')
|
||||
updated_at = model_updated_at
|
||||
|
||||
# Fallback: check arena models stored in config (not in the DB)
|
||||
if not profile_image_url:
|
||||
arena_models = await Config.get('evaluation.arena.models', []) or []
|
||||
for arena_model in arena_models:
|
||||
if arena_model.get('id') == id:
|
||||
profile_image_url = arena_model.get('meta', {}).get('profile_image_url')
|
||||
arena_meta = arena_model.get('meta', {})
|
||||
if bypass_access_control or await has_access(
|
||||
user.id, permission='read', access_grants=arena_meta.get('access_grants', []), db=db
|
||||
):
|
||||
profile_image_url = arena_meta.get('profile_image_url')
|
||||
break
|
||||
|
||||
if profile_image_url:
|
||||
|
|
@ -829,6 +979,10 @@ async def update_model_by_id(
|
|||
if 'profile_image_url' not in form_data.meta.model_fields_set:
|
||||
form_data.meta.profile_image_url = model.meta.profile_image_url
|
||||
|
||||
if 'background_image_url' not in form_data.meta.model_fields_set:
|
||||
form_data.meta.background_image_url = model.meta.background_image_url
|
||||
await _verify_background_image(form_data.meta.background_image_url, user, db, model.meta.background_image_url)
|
||||
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
|
|
@ -855,7 +1009,7 @@ async def update_model_by_id(
|
|||
|
||||
|
||||
class ModelAccessGrantsForm(BaseModel):
|
||||
id: str
|
||||
id: str = Field(pattern=r'^\S+$')
|
||||
name: str | None = None
|
||||
access_grants: list[dict]
|
||||
|
||||
|
|
|
|||
|
|
@ -554,14 +554,15 @@ async def update_note_by_id(
|
|||
):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.DEFAULT())
|
||||
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
user.role,
|
||||
form_data.access_grants,
|
||||
'sharing.public_notes',
|
||||
db=db,
|
||||
)
|
||||
if form_data.access_grants is not None:
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
user.role,
|
||||
form_data.access_grants,
|
||||
'sharing.public_notes',
|
||||
db=db,
|
||||
)
|
||||
|
||||
try:
|
||||
note = await Notes.update_note_by_id(id, form_data, db=db)
|
||||
|
|
|
|||
|
|
@ -55,14 +55,15 @@ log = logging.getLogger(__name__)
|
|||
# response body. Forwarding them verbatim causes desktop / programmatic
|
||||
# clients to attempt decompression of an already-decoded payload, resulting
|
||||
# in ZlibError. See https://github.com/aio-libs/aiohttp/issues/4462.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'Content-Encoding', 'Content-Length', 'Transfer-Encoding'})
|
||||
# Also drop server and date: uvicorn adds its own and forwarding both duplicates them.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'content-encoding', 'content-length', 'transfer-encoding', 'server', 'date'})
|
||||
_MODEL_LIST_TIMEOUT = aiohttp.ClientTimeout(total=AIOHTTP_CLIENT_TIMEOUT_MODEL_LIST)
|
||||
BASE_MODELS_CACHE_KEY = f'{REDIS_KEY_PREFIX}:models:base'
|
||||
|
||||
|
||||
def _clean_proxy_headers(raw_headers) -> dict:
|
||||
"""Return a copy of *raw_headers* with stale encoding headers removed."""
|
||||
return {k: v for k, v in raw_headers.items() if k not in _STRIP_PROXY_HEADERS}
|
||||
"""Return a copy of *raw_headers* without the encoding, server and date headers."""
|
||||
return {k: v for k, v in raw_headers.items() if k.lower() not in _STRIP_PROXY_HEADERS}
|
||||
|
||||
|
||||
async def send_get_request(
|
||||
|
|
@ -119,7 +120,7 @@ async def send_request(
|
|||
}
|
||||
|
||||
if ENABLE_FORWARD_USER_INFO_HEADERS and user:
|
||||
headers = include_user_info_headers(headers, user)
|
||||
headers = include_user_info_headers(headers, user, request=request)
|
||||
if metadata and metadata.get('chat_id'):
|
||||
headers[FORWARD_SESSION_INFO_HEADER_CHAT_ID] = metadata.get('chat_id')
|
||||
|
||||
|
|
@ -478,13 +479,16 @@ async def get_filtered_models(models, user, db=None):
|
|||
|
||||
|
||||
@router.get('/api/tags')
|
||||
@router.get('/api/tags/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/api/tags/{url_idx}')
|
||||
async def get_ollama_tags(
|
||||
request: Request,
|
||||
url_idx: int | None = None,
|
||||
user=Depends(get_verified_user),
|
||||
):
|
||||
"""List Ollama model tags, optionally from a specific backend."""
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if not await Config.get('ollama.enable'):
|
||||
raise HTTPException(status_code=503, detail=ERROR_MESSAGES.OLLAMA_API_DISABLED)
|
||||
|
||||
|
|
@ -541,13 +545,16 @@ async def get_ollama_loaded_models(
|
|||
|
||||
|
||||
@router.get('/api/version')
|
||||
@router.get('/api/version/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/api/version/{url_idx}')
|
||||
async def get_ollama_versions(
|
||||
request: Request,
|
||||
user=Depends(get_verified_user),
|
||||
url_idx: int | None = None,
|
||||
):
|
||||
"""Return the lowest Ollama version across all configured backends."""
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if not await Config.get('ollama.enable'):
|
||||
return {'version': False}
|
||||
|
||||
|
|
@ -1479,7 +1486,7 @@ async def generate_responses(
|
|||
|
||||
|
||||
@router.get('/v1/models')
|
||||
@router.get('/v1/models/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/v1/models/{url_idx}')
|
||||
async def get_openai_models(
|
||||
request: Request,
|
||||
url_idx: int | None = None,
|
||||
|
|
@ -1487,6 +1494,9 @@ async def get_openai_models(
|
|||
db: AsyncSession = Depends(get_async_session),
|
||||
) -> dict:
|
||||
"""List models in the OpenAI-compatible format."""
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if url_idx is None:
|
||||
model_list = await get_all_models(request, user=user)
|
||||
raw_models = model_list['models']
|
||||
|
|
|
|||
|
|
@ -74,15 +74,16 @@ log = logging.getLogger(__name__)
|
|||
# response body. Forwarding them verbatim causes desktop / programmatic
|
||||
# clients to attempt decompression of an already-decoded payload, resulting
|
||||
# in ZlibError. See https://github.com/aio-libs/aiohttp/issues/4462.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'Content-Encoding', 'Content-Length', 'Transfer-Encoding'})
|
||||
# Also drop server and date: uvicorn adds its own and forwarding both duplicates them.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'content-encoding', 'content-length', 'transfer-encoding', 'server', 'date'})
|
||||
_MODEL_LIST_TIMEOUT = aiohttp.ClientTimeout(total=AIOHTTP_CLIENT_TIMEOUT_MODEL_LIST)
|
||||
_UNSUPPORTED_OPENAI_MODEL_KEYWORDS = ('babbage', 'dall-e', 'davinci', 'embedding', 'tts', 'whisper')
|
||||
BASE_MODELS_CACHE_KEY = f'{REDIS_KEY_PREFIX}:models:base'
|
||||
|
||||
|
||||
def _clean_proxy_headers(raw_headers) -> dict:
|
||||
"""Return a copy of *raw_headers* with stale encoding headers removed."""
|
||||
return {k: v for k, v in raw_headers.items() if k not in _STRIP_PROXY_HEADERS}
|
||||
"""Return a copy of *raw_headers* without the encoding, server and date headers."""
|
||||
return {k: v for k, v in raw_headers.items() if k.lower() not in _STRIP_PROXY_HEADERS}
|
||||
|
||||
|
||||
async def send_get_request(
|
||||
|
|
@ -103,7 +104,7 @@ async def send_get_request(
|
|||
cookies = None
|
||||
|
||||
if ENABLE_FORWARD_USER_INFO_HEADERS and user:
|
||||
headers = include_user_info_headers(headers, user)
|
||||
headers = include_user_info_headers(headers, user, request=request)
|
||||
|
||||
async with session.get(
|
||||
url,
|
||||
|
|
@ -159,7 +160,7 @@ async def get_headers_and_cookies(
|
|||
metadata: dict | None = None,
|
||||
user: UserModel = None,
|
||||
):
|
||||
cookies = {}
|
||||
cookies = getattr(request, 'cookies', {}) if config.get('forward_cookies', False) else {}
|
||||
headers = {
|
||||
'Content-Type': 'application/json',
|
||||
**(
|
||||
|
|
@ -176,7 +177,7 @@ async def get_headers_and_cookies(
|
|||
}
|
||||
|
||||
if ENABLE_FORWARD_USER_INFO_HEADERS and user:
|
||||
headers = include_user_info_headers(headers, user)
|
||||
headers = include_user_info_headers(headers, user, request=request)
|
||||
if metadata and metadata.get('chat_id'):
|
||||
headers[FORWARD_SESSION_INFO_HEADER_CHAT_ID] = metadata.get('chat_id')
|
||||
|
||||
|
|
@ -189,11 +190,8 @@ async def get_headers_and_cookies(
|
|||
elif auth_type == 'none':
|
||||
token = None
|
||||
elif auth_type == 'session':
|
||||
cookies = request.cookies
|
||||
token = request.state.token.credentials
|
||||
elif auth_type == 'system_oauth':
|
||||
cookies = request.cookies
|
||||
|
||||
oauth_token = None
|
||||
try:
|
||||
if request.cookies.get('oauth_session_id', None):
|
||||
|
|
@ -864,8 +862,11 @@ async def get_all_models(request: Request, user: UserModel) -> dict[str, list]:
|
|||
|
||||
|
||||
@router.get('/models')
|
||||
@router.get('/models/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/models/{url_idx}')
|
||||
async def get_models(request: Request, url_idx: int | None = None, user=Depends(get_verified_user)):
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if not await Config.get('openai.enable'):
|
||||
raise HTTPException(status_code=503, detail='OpenAI API is disabled')
|
||||
|
||||
|
|
@ -1416,14 +1417,19 @@ def convert_to_responses_payload(payload: dict) -> dict:
|
|||
converted_tool['description'] = func['description']
|
||||
if 'parameters' in func:
|
||||
converted_tool['parameters'] = func['parameters']
|
||||
if 'strict' in func:
|
||||
converted_tool['strict'] = func['strict']
|
||||
# Responses defaults strict to true, Chat Completions to false
|
||||
converted_tool['strict'] = func.get('strict', False)
|
||||
converted_tools.append(converted_tool)
|
||||
else:
|
||||
# Already in correct format or unknown format, pass through
|
||||
converted_tools.append(tool)
|
||||
responses_payload['tools'] = converted_tools
|
||||
|
||||
# Responses API expects a forced function choice as {"type": "function", "name": ...}
|
||||
tool_choice = responses_payload.get('tool_choice')
|
||||
if isinstance(tool_choice, dict) and isinstance(tool_choice.get('function'), dict):
|
||||
responses_payload['tool_choice'] = {'type': 'function', 'name': tool_choice['function'].get('name', '')}
|
||||
|
||||
return responses_payload
|
||||
|
||||
|
||||
|
|
@ -1431,17 +1437,32 @@ def convert_responses_result(response: dict) -> dict:
|
|||
"""
|
||||
Convert non-streaming Responses API result to Chat Completions format.
|
||||
|
||||
Extracts text from message output items so all downstream consumers
|
||||
Extracts text and function calls from output items so all downstream consumers
|
||||
(frontend tasks, get_content_from_response) work without modification.
|
||||
"""
|
||||
output_items = response.get('output', [])
|
||||
|
||||
content = ''
|
||||
tool_calls = []
|
||||
for item in output_items:
|
||||
if item.get('type') == 'message':
|
||||
for part in item.get('content', []):
|
||||
if part.get('type') == 'output_text':
|
||||
content += part.get('text', '')
|
||||
elif item.get('type') == 'function_call':
|
||||
arguments = item.get('arguments', '{}')
|
||||
if not isinstance(arguments, str):
|
||||
arguments = JSONCodec.dumps(arguments)
|
||||
tool_calls.append(
|
||||
{
|
||||
'id': item.get('call_id', ''),
|
||||
'type': 'function',
|
||||
'function': {
|
||||
'name': item.get('name', ''),
|
||||
'arguments': arguments,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
'id': response.get('id', ''),
|
||||
|
|
@ -1453,8 +1474,9 @@ def convert_responses_result(response: dict) -> dict:
|
|||
'message': {
|
||||
'role': 'assistant',
|
||||
'content': content,
|
||||
**({'tool_calls': tool_calls} if tool_calls else {}),
|
||||
},
|
||||
'finish_reason': 'stop',
|
||||
'finish_reason': 'tool_calls' if tool_calls else 'stop',
|
||||
}
|
||||
],
|
||||
'usage': response.get('usage', {}),
|
||||
|
|
@ -1566,6 +1588,20 @@ async def generate_chat_completion(
|
|||
|
||||
is_responses = api_config.get('api_type') == 'responses'
|
||||
|
||||
# Explicit continuation keeps llama.cpp from echoing the prefill in streamed replies.
|
||||
if (
|
||||
api_config.get('provider') == 'llama.cpp'
|
||||
# These flags apply to Chat Completions, not the Responses API.
|
||||
and not is_responses
|
||||
# The frontend sends this ID when the user clicks Continue.
|
||||
and (metadata or {}).get('assistant_message_id')
|
||||
# Tool follow-ups retain the metadata but must start a new assistant turn.
|
||||
and payload.get('messages')
|
||||
and payload['messages'][-1].get('role') == 'assistant'
|
||||
):
|
||||
payload['continue_final_message'] = True
|
||||
payload['add_generation_prompt'] = False
|
||||
|
||||
if api_config.get('azure') or api_config.get('provider') == 'azure':
|
||||
# Only set api-key header if not using Azure Entra ID authentication
|
||||
auth_type = api_config.get('auth_type', 'bearer')
|
||||
|
|
|
|||
|
|
@ -58,6 +58,7 @@ from open_webui.env import (
|
|||
SENTENCE_TRANSFORMERS_CROSS_ENCODER_MODEL_KWARGS,
|
||||
SENTENCE_TRANSFORMERS_CROSS_ENCODER_SIGMOID_ACTIVATION_FUNCTION,
|
||||
SENTENCE_TRANSFORMERS_MODEL_KWARGS,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.internal.db import get_async_db, get_async_session
|
||||
|
|
@ -82,7 +83,7 @@ from open_webui.retrieval.utils import (
|
|||
query_doc_with_hybrid_search,
|
||||
)
|
||||
from open_webui.retrieval.vector.async_client import ASYNC_VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import get_vector_db_client
|
||||
from open_webui.retrieval.vector.utils import filter_metadata
|
||||
from open_webui.retrieval.web.azure import search_azure
|
||||
from open_webui.retrieval.web.bing import search_bing
|
||||
|
|
@ -114,6 +115,7 @@ from open_webui.retrieval.web.serphouse import search_serphouse
|
|||
from open_webui.retrieval.web.serply import search_serply
|
||||
from open_webui.retrieval.web.serpstack import search_serpstack
|
||||
from open_webui.retrieval.web.sougou import search_sougou
|
||||
from open_webui.retrieval.web.staan import search_staan
|
||||
from open_webui.retrieval.web.tavily import search_tavily
|
||||
from open_webui.retrieval.web.utils import get_web_loader
|
||||
from open_webui.retrieval.web.yacy import search_yacy
|
||||
|
|
@ -128,7 +130,7 @@ from open_webui.utils.misc import (
|
|||
calculate_sha256_string,
|
||||
sanitize_text_for_db,
|
||||
)
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -150,7 +152,7 @@ def get_ef(
|
|||
auto_update: bool = RAG_EMBEDDING_MODEL_AUTO_UPDATE,
|
||||
):
|
||||
ef = None
|
||||
if embedding_model and engine == '':
|
||||
if embedding_model and engine == '' and not USE_SLIM:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
try:
|
||||
|
|
@ -178,6 +180,17 @@ def get_rf(
|
|||
rf = None
|
||||
# Convert timeout string to int or None (system default)
|
||||
timeout_value = int(external_reranker_timeout) if external_reranker_timeout else None
|
||||
if reranking_model and engine == 'external':
|
||||
from open_webui.retrieval.models.external import ExternalReranker
|
||||
|
||||
return ExternalReranker(
|
||||
url=external_reranker_url,
|
||||
api_key=external_reranker_api_key,
|
||||
model=reranking_model,
|
||||
timeout=timeout_value,
|
||||
)
|
||||
if USE_SLIM:
|
||||
return None
|
||||
if reranking_model:
|
||||
if any(model in reranking_model for model in ['jinaai/jina-colbert-v2']):
|
||||
try:
|
||||
|
|
@ -192,55 +205,39 @@ def get_rf(
|
|||
log.error(f'ColBERT: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'Error loading reranking model'))
|
||||
else:
|
||||
if engine == 'external':
|
||||
try:
|
||||
from open_webui.retrieval.models.external import ExternalReranker
|
||||
import sentence_transformers
|
||||
import torch
|
||||
|
||||
rf = ExternalReranker(
|
||||
url=external_reranker_url,
|
||||
api_key=external_reranker_api_key,
|
||||
model=reranking_model,
|
||||
timeout=timeout_value,
|
||||
)
|
||||
except Exception as e:
|
||||
log.error(f'ExternalReranking: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'Error loading reranking model'))
|
||||
else:
|
||||
import sentence_transformers
|
||||
import torch
|
||||
try:
|
||||
rf = sentence_transformers.CrossEncoder(
|
||||
get_model_path(reranking_model, auto_update),
|
||||
device=DEVICE_TYPE,
|
||||
trust_remote_code=RAG_RERANKING_MODEL_TRUST_REMOTE_CODE,
|
||||
backend=SENTENCE_TRANSFORMERS_CROSS_ENCODER_BACKEND,
|
||||
model_kwargs=SENTENCE_TRANSFORMERS_CROSS_ENCODER_MODEL_KWARGS,
|
||||
activation_fn=(
|
||||
torch.nn.Sigmoid() if SENTENCE_TRANSFORMERS_CROSS_ENCODER_SIGMOID_ACTIVATION_FUNCTION else None
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
log.error(f'CrossEncoder: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'CrossEncoder error'))
|
||||
|
||||
try:
|
||||
rf = sentence_transformers.CrossEncoder(
|
||||
get_model_path(reranking_model, auto_update),
|
||||
device=DEVICE_TYPE,
|
||||
trust_remote_code=RAG_RERANKING_MODEL_TRUST_REMOTE_CODE,
|
||||
backend=SENTENCE_TRANSFORMERS_CROSS_ENCODER_BACKEND,
|
||||
model_kwargs=SENTENCE_TRANSFORMERS_CROSS_ENCODER_MODEL_KWARGS,
|
||||
activation_fn=(
|
||||
torch.nn.Sigmoid()
|
||||
if SENTENCE_TRANSFORMERS_CROSS_ENCODER_SIGMOID_ACTIVATION_FUNCTION
|
||||
else None
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
log.error(f'CrossEncoder: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'CrossEncoder error'))
|
||||
|
||||
# Safely adjust pad_token_id if missing as some models do not have this in config
|
||||
try:
|
||||
model_cfg = getattr(rf, 'model', None)
|
||||
if model_cfg and hasattr(model_cfg, 'config'):
|
||||
cfg = model_cfg.config
|
||||
if getattr(cfg, 'pad_token_id', None) is None:
|
||||
# Fallback to eos_token_id when available
|
||||
eos = getattr(cfg, 'eos_token_id', None)
|
||||
if eos is not None:
|
||||
cfg.pad_token_id = eos
|
||||
log.debug('Missing pad_token_id detected; set to eos_token_id=%s', eos)
|
||||
else:
|
||||
log.warning('Neither pad_token_id nor eos_token_id present in model config')
|
||||
except Exception as e2:
|
||||
log.warning(f'Failed to adjust pad_token_id on CrossEncoder: {e2}')
|
||||
# Safely adjust pad_token_id if missing as some models do not have this in config
|
||||
try:
|
||||
model_cfg = getattr(rf, 'model', None)
|
||||
if model_cfg and hasattr(model_cfg, 'config'):
|
||||
cfg = model_cfg.config
|
||||
if getattr(cfg, 'pad_token_id', None) is None:
|
||||
# Fallback to eos_token_id when available
|
||||
eos = getattr(cfg, 'eos_token_id', None)
|
||||
if eos is not None:
|
||||
cfg.pad_token_id = eos
|
||||
log.debug('Missing pad_token_id detected; set to eos_token_id=%s', eos)
|
||||
else:
|
||||
log.warning('Neither pad_token_id nor eos_token_id present in model config')
|
||||
except Exception as e2:
|
||||
log.warning(f'Failed to adjust pad_token_id on CrossEncoder: {e2}')
|
||||
|
||||
return rf
|
||||
|
||||
|
|
@ -301,6 +298,7 @@ RETRIEVAL_CONFIG_KEYS = {
|
|||
'ENABLE_WEB_SEARCH_CONFIRMATION': 'web.search.confirmation.enable',
|
||||
'WEB_SEARCH_CONFIRMATION_CONTENT': 'web.search.confirmation.content',
|
||||
'EXA_API_KEY': 'web.search.exa_api_key',
|
||||
'EXA_MAX_CONTENT_LENGTH': 'web.search.exa_max_content_length',
|
||||
'EXTERNAL_DOCUMENT_LOADER_API_KEY': 'rag.external_document_loader_api_key',
|
||||
'EXTERNAL_DOCUMENT_LOADER_HEADERS': 'rag.external_document_loader_headers',
|
||||
'EXTERNAL_DOCUMENT_LOADER_URL': 'rag.external_document_loader_url',
|
||||
|
|
@ -383,6 +381,9 @@ RETRIEVAL_CONFIG_KEYS = {
|
|||
'SERPSTACK_HTTPS': 'web.search.serpstack_https',
|
||||
'SOUGOU_API_SID': 'web.search.sougou_api_sid',
|
||||
'SOUGOU_API_SK': 'web.search.sougou_api_sk',
|
||||
'STAAN_API_KEY': 'web.search.staan_api_key',
|
||||
'STAAN_MARKET': 'web.search.staan_market',
|
||||
'STAAN_MAX_SNIPPETS': 'web.search.staan_max_snippets',
|
||||
'TAVILY_API_KEY': 'web.search.tavily_api_key',
|
||||
'TAVILY_EXTRACT_DEPTH': 'web.search.tavily_extract_depth',
|
||||
'TEXT_SPLITTER': 'rag.text_splitter',
|
||||
|
|
@ -535,6 +536,8 @@ async def unload_embedding_model(request: Request):
|
|||
|
||||
@router.post('/embedding/update')
|
||||
async def update_embedding_config(request: Request, form_data: EmbeddingModelUpdateForm, user=Depends(get_admin_user)):
|
||||
if USE_SLIM and form_data.RAG_EMBEDDING_ENGINE == '':
|
||||
raise HTTPException(400, 'Slim requires an external embedding engine (openai, ollama, azure_openai).')
|
||||
config = await get_retrieval_config()
|
||||
log.info('Updating embedding model: %s to %s', config.RAG_EMBEDDING_MODEL, form_data.RAG_EMBEDDING_MODEL)
|
||||
await unload_embedding_model(request)
|
||||
|
|
@ -738,6 +741,9 @@ async def get_rag_config(request: Request, user=Depends(get_admin_user)):
|
|||
'SERPLY_API_KEY': config.SERPLY_API_KEY,
|
||||
'DDGS_BACKEND': config.DDGS_BACKEND,
|
||||
'TAVILY_API_KEY': config.TAVILY_API_KEY,
|
||||
'STAAN_API_KEY': config.STAAN_API_KEY,
|
||||
'STAAN_MARKET': config.STAAN_MARKET,
|
||||
'STAAN_MAX_SNIPPETS': config.STAAN_MAX_SNIPPETS,
|
||||
'SEARCHAPI_API_KEY': config.SEARCHAPI_API_KEY,
|
||||
'SEARCHAPI_ENGINE': config.SEARCHAPI_ENGINE,
|
||||
'SERPAPI_API_KEY': config.SERPAPI_API_KEY,
|
||||
|
|
@ -747,6 +753,7 @@ async def get_rag_config(request: Request, user=Depends(get_admin_user)):
|
|||
'BING_SEARCH_V7_ENDPOINT': config.BING_SEARCH_V7_ENDPOINT,
|
||||
'BING_SEARCH_V7_SUBSCRIPTION_KEY': config.BING_SEARCH_V7_SUBSCRIPTION_KEY,
|
||||
'EXA_API_KEY': config.EXA_API_KEY,
|
||||
'EXA_MAX_CONTENT_LENGTH': config.EXA_MAX_CONTENT_LENGTH,
|
||||
'PERPLEXITY_API_KEY': config.PERPLEXITY_API_KEY,
|
||||
'PERPLEXITY_MODEL': config.PERPLEXITY_MODEL,
|
||||
'PERPLEXITY_SEARCH_CONTEXT_USAGE': config.PERPLEXITY_SEARCH_CONTEXT_USAGE,
|
||||
|
|
@ -817,6 +824,9 @@ class WebConfig(BaseModel):
|
|||
SERPLY_API_KEY: str | None = None
|
||||
DDGS_BACKEND: str | None = None
|
||||
TAVILY_API_KEY: str | None = None
|
||||
STAAN_API_KEY: str | None = None
|
||||
STAAN_MARKET: str | None = None
|
||||
STAAN_MAX_SNIPPETS: int | None = None
|
||||
SEARCHAPI_API_KEY: str | None = None
|
||||
SEARCHAPI_ENGINE: str | None = None
|
||||
SERPAPI_API_KEY: str | None = None
|
||||
|
|
@ -826,6 +836,7 @@ class WebConfig(BaseModel):
|
|||
BING_SEARCH_V7_ENDPOINT: str | None = None
|
||||
BING_SEARCH_V7_SUBSCRIPTION_KEY: str | None = None
|
||||
EXA_API_KEY: str | None = None
|
||||
EXA_MAX_CONTENT_LENGTH: int | None = Field(default=None, gt=0, strict=True)
|
||||
PERPLEXITY_API_KEY: str | None = None
|
||||
PERPLEXITY_MODEL: str | None = None
|
||||
PERPLEXITY_SEARCH_CONTEXT_USAGE: str | None = None
|
||||
|
|
@ -952,6 +963,44 @@ class ConfigForm(BaseModel):
|
|||
async def update_rag_config(request: Request, form_data: ConfigForm, user=Depends(get_admin_user)):
|
||||
# RAG settings
|
||||
config = await get_retrieval_config()
|
||||
if USE_SLIM:
|
||||
if (
|
||||
form_data.web
|
||||
and form_data.web.WEB_SEARCH_ENGINE == 'duckduckgo'
|
||||
and config.WEB_SEARCH_ENGINE != 'duckduckgo'
|
||||
):
|
||||
raise HTTPException(
|
||||
400,
|
||||
'DDGS is unavailable in slim. Configure another web search provider in Admin Settings > Web Search.',
|
||||
)
|
||||
if (
|
||||
form_data.web
|
||||
and form_data.web.WEB_LOADER_ENGINE == 'playwright'
|
||||
and config.WEB_LOADER_ENGINE != 'playwright'
|
||||
):
|
||||
raise HTTPException(
|
||||
400, 'Playwright is unavailable in slim. Use basic HTTP fetching or an external web loader.'
|
||||
)
|
||||
if form_data.TEXT_SPLITTER == 'token_transformers' and config.TEXT_SPLITTER != 'token_transformers':
|
||||
raise HTTPException(
|
||||
400, 'Transformers tokenization is unavailable in slim. Use character or token splitting.'
|
||||
)
|
||||
reranker_engine = (
|
||||
form_data.RAG_RERANKING_ENGINE
|
||||
if form_data.RAG_RERANKING_ENGINE is not None
|
||||
else config.RAG_RERANKING_ENGINE
|
||||
)
|
||||
reranker_model = (
|
||||
form_data.RAG_RERANKING_MODEL if form_data.RAG_RERANKING_MODEL is not None else config.RAG_RERANKING_MODEL
|
||||
)
|
||||
if (
|
||||
reranker_engine != 'external'
|
||||
and reranker_model
|
||||
and (reranker_engine != config.RAG_RERANKING_ENGINE or reranker_model != config.RAG_RERANKING_MODEL)
|
||||
):
|
||||
raise HTTPException(
|
||||
400, 'Slim requires an external reranker, or an empty reranking model for cosine scoring.'
|
||||
)
|
||||
config.RAG_TEMPLATE = form_data.RAG_TEMPLATE if form_data.RAG_TEMPLATE is not None else config.RAG_TEMPLATE
|
||||
config.TOP_K = form_data.TOP_K if form_data.TOP_K is not None else config.TOP_K
|
||||
config.BYPASS_EMBEDDING_AND_RETRIEVAL = (
|
||||
|
|
@ -1176,7 +1225,8 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
else config.RAG_RERANKING_BATCH_SIZE
|
||||
)
|
||||
|
||||
log.info('Updating reranking model: %s to %s', config.RAG_RERANKING_MODEL, form_data.RAG_RERANKING_MODEL)
|
||||
if form_data.RAG_RERANKING_MODEL is not None:
|
||||
log.info('Updating reranking model: %s to %s', config.RAG_RERANKING_MODEL, form_data.RAG_RERANKING_MODEL)
|
||||
try:
|
||||
config.RAG_RERANKING_MODEL = (
|
||||
form_data.RAG_RERANKING_MODEL if form_data.RAG_RERANKING_MODEL is not None else config.RAG_RERANKING_MODEL
|
||||
|
|
@ -1297,6 +1347,9 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
config.SERPLY_API_KEY = form_data.web.SERPLY_API_KEY
|
||||
config.DDGS_BACKEND = form_data.web.DDGS_BACKEND
|
||||
config.TAVILY_API_KEY = form_data.web.TAVILY_API_KEY
|
||||
config.STAAN_API_KEY = form_data.web.STAAN_API_KEY
|
||||
config.STAAN_MARKET = form_data.web.STAAN_MARKET
|
||||
config.STAAN_MAX_SNIPPETS = form_data.web.STAAN_MAX_SNIPPETS
|
||||
config.SEARCHAPI_API_KEY = form_data.web.SEARCHAPI_API_KEY
|
||||
config.SEARCHAPI_ENGINE = form_data.web.SEARCHAPI_ENGINE
|
||||
config.SERPAPI_API_KEY = form_data.web.SERPAPI_API_KEY
|
||||
|
|
@ -1306,6 +1359,7 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
config.BING_SEARCH_V7_ENDPOINT = form_data.web.BING_SEARCH_V7_ENDPOINT
|
||||
config.BING_SEARCH_V7_SUBSCRIPTION_KEY = form_data.web.BING_SEARCH_V7_SUBSCRIPTION_KEY
|
||||
config.EXA_API_KEY = form_data.web.EXA_API_KEY
|
||||
config.EXA_MAX_CONTENT_LENGTH = form_data.web.EXA_MAX_CONTENT_LENGTH
|
||||
config.PERPLEXITY_API_KEY = form_data.web.PERPLEXITY_API_KEY
|
||||
config.PERPLEXITY_MODEL = form_data.web.PERPLEXITY_MODEL
|
||||
config.PERPLEXITY_SEARCH_CONTEXT_USAGE = form_data.web.PERPLEXITY_SEARCH_CONTEXT_USAGE
|
||||
|
|
@ -1449,6 +1503,9 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
'SERPHOUSE_DOMAIN': config.SERPHOUSE_DOMAIN,
|
||||
'SERPLY_API_KEY': config.SERPLY_API_KEY,
|
||||
'TAVILY_API_KEY': config.TAVILY_API_KEY,
|
||||
'STAAN_API_KEY': config.STAAN_API_KEY,
|
||||
'STAAN_MARKET': config.STAAN_MARKET,
|
||||
'STAAN_MAX_SNIPPETS': config.STAAN_MAX_SNIPPETS,
|
||||
'SEARCHAPI_API_KEY': config.SEARCHAPI_API_KEY,
|
||||
'SEARCHAPI_ENGINE': config.SEARCHAPI_ENGINE,
|
||||
'SERPAPI_API_KEY': config.SERPAPI_API_KEY,
|
||||
|
|
@ -1458,6 +1515,7 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
'BING_SEARCH_V7_ENDPOINT': config.BING_SEARCH_V7_ENDPOINT,
|
||||
'BING_SEARCH_V7_SUBSCRIPTION_KEY': config.BING_SEARCH_V7_SUBSCRIPTION_KEY,
|
||||
'EXA_API_KEY': config.EXA_API_KEY,
|
||||
'EXA_MAX_CONTENT_LENGTH': config.EXA_MAX_CONTENT_LENGTH,
|
||||
'PERPLEXITY_API_KEY': config.PERPLEXITY_API_KEY,
|
||||
'PERPLEXITY_MODEL': config.PERPLEXITY_MODEL,
|
||||
'PERPLEXITY_SEARCH_CONTEXT_USAGE': config.PERPLEXITY_SEARCH_CONTEXT_USAGE,
|
||||
|
|
@ -1589,6 +1647,8 @@ def merge_docs_to_target_size(
|
|||
|
||||
|
||||
def get_transformers_tokenizer(request: Request, config: RetrievalConfig):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Transformers tokenization is unavailable in slim. Use character or token splitting.')
|
||||
if config.RAG_TOKENIZER_MODEL:
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
|
|
@ -1634,6 +1694,14 @@ def get_splitter_length_function(
|
|||
return len
|
||||
|
||||
|
||||
def filter_file_metadata(metadata: dict | None) -> dict:
|
||||
metadata = dict(metadata or {})
|
||||
data = metadata.pop('data', None)
|
||||
if isinstance(data, dict):
|
||||
metadata = {**filter_metadata(data), **metadata}
|
||||
return filter_metadata(metadata)
|
||||
|
||||
|
||||
def save_docs_to_vector_db(
|
||||
request: Request,
|
||||
docs,
|
||||
|
|
@ -1665,7 +1733,7 @@ def save_docs_to_vector_db(
|
|||
|
||||
# Check if entries with the same hash (metadata.hash) already exist
|
||||
if metadata and 'hash' in metadata:
|
||||
result = VECTOR_DB_CLIENT.query(
|
||||
result = get_vector_db_client().query(
|
||||
collection_name=collection_name,
|
||||
filter={'hash': metadata['hash']},
|
||||
)
|
||||
|
|
@ -1765,11 +1833,11 @@ def save_docs_to_vector_db(
|
|||
]
|
||||
|
||||
try:
|
||||
if VECTOR_DB_CLIENT.has_collection(collection_name=collection_name):
|
||||
if get_vector_db_client().has_collection(collection_name=collection_name):
|
||||
log.info('collection %s already exists', collection_name)
|
||||
|
||||
if overwrite:
|
||||
VECTOR_DB_CLIENT.delete_collection(collection_name=collection_name)
|
||||
get_vector_db_client().delete_collection(collection_name=collection_name)
|
||||
log.info('deleting existing collection %s', collection_name)
|
||||
elif add is False:
|
||||
log.info('collection %s already exists, overwrite is False and add is False', collection_name)
|
||||
|
|
@ -1832,7 +1900,7 @@ def save_docs_to_vector_db(
|
|||
]
|
||||
|
||||
log.info('adding to collection %s', collection_name)
|
||||
VECTOR_DB_CLIENT.insert(
|
||||
get_vector_db_client().insert(
|
||||
collection_name=collection_name,
|
||||
items=items,
|
||||
)
|
||||
|
|
@ -1898,7 +1966,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=form_data.content.replace('<br/>', '\n'),
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
@ -1934,7 +2002,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=stored_content,
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
@ -1967,6 +2035,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=doc.page_content,
|
||||
metadata={
|
||||
**filter_file_metadata(file.meta),
|
||||
**filter_metadata(doc.metadata),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
|
|
@ -1981,7 +2050,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=file.data.get('content', ''),
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
@ -2636,6 +2705,19 @@ async def search_web(request: Request, engine: str, query: str, user=None) -> li
|
|||
)
|
||||
else:
|
||||
raise Exception('No TAVILY_API_KEY found in environment variables')
|
||||
elif engine == 'staan':
|
||||
if config.STAAN_API_KEY:
|
||||
return await asyncio.to_thread(
|
||||
search_staan,
|
||||
config.STAAN_API_KEY,
|
||||
query,
|
||||
config.WEB_SEARCH_RESULT_COUNT,
|
||||
config.WEB_SEARCH_DOMAIN_FILTER_LIST,
|
||||
market=config.STAAN_MARKET,
|
||||
max_snippets=config.STAAN_MAX_SNIPPETS,
|
||||
)
|
||||
else:
|
||||
raise Exception('No STAAN_API_KEY found in environment variables')
|
||||
elif engine == 'exa':
|
||||
if config.EXA_API_KEY:
|
||||
return await asyncio.to_thread(
|
||||
|
|
@ -2644,6 +2726,7 @@ async def search_web(request: Request, engine: str, query: str, user=None) -> li
|
|||
query,
|
||||
config.WEB_SEARCH_RESULT_COUNT,
|
||||
config.WEB_SEARCH_DOMAIN_FILTER_LIST,
|
||||
max_content_length=config.EXA_MAX_CONTENT_LENGTH,
|
||||
)
|
||||
else:
|
||||
raise Exception('No EXA_API_KEY found in environment variables')
|
||||
|
|
@ -3026,7 +3109,7 @@ async def query_doc_handler(
|
|||
query_embedding = await request.app.state.EMBEDDING_FUNCTION(
|
||||
form_data.query, prefix=RAG_EMBEDDING_QUERY_PREFIX, user=user
|
||||
)
|
||||
# query_doc wraps a blocking VECTOR_DB_CLIENT.search call;
|
||||
# query_doc wraps a blocking get_vector_db_client().search call;
|
||||
# offload so the request's event loop stays responsive.
|
||||
return await asyncio.to_thread(
|
||||
query_doc,
|
||||
|
|
@ -3314,7 +3397,7 @@ async def process_files_batch(
|
|||
Document(
|
||||
page_content=text_content.replace('<br/>', '\n'),
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
|
|||
|
|
@ -753,31 +753,60 @@ async def patch_user(
|
|||
)
|
||||
|
||||
update_data = {}
|
||||
fields = {
|
||||
'userName': 'email',
|
||||
'displayName': 'name',
|
||||
'emails[primary eq true].value': 'email',
|
||||
'name.formatted': 'name',
|
||||
}
|
||||
|
||||
for operation in patch_data.Operations:
|
||||
op = operation.op.lower()
|
||||
path = operation.path
|
||||
value = operation.value
|
||||
|
||||
if op == 'replace':
|
||||
if op not in ('add', 'replace', 'remove'):
|
||||
return scim_error(400, f'Unsupported PATCH operation: {operation.op}')
|
||||
if op == 'remove':
|
||||
if not path:
|
||||
return scim_error(400, 'Remove requires a path', 'noTarget')
|
||||
if path != 'externalId':
|
||||
return scim_error(400, f'Removing {path} is not supported', 'mutability')
|
||||
values = {path: None}
|
||||
elif path is None:
|
||||
if not isinstance(operation.value, dict) or not operation.value:
|
||||
return scim_error(400, 'A pathless operation requires an attribute object', 'invalidValue')
|
||||
values = operation.value
|
||||
else:
|
||||
values = {path: operation.value}
|
||||
|
||||
for path, value in values.items():
|
||||
if path == 'active':
|
||||
if not isinstance(value, bool):
|
||||
return scim_error(400, 'active must be a boolean', 'invalidValue')
|
||||
# Same guard as update_user: never demote an existing admin via SCIM.
|
||||
if user.role != 'admin':
|
||||
update_data['role'] = 'user' if value else 'pending'
|
||||
elif path == 'userName':
|
||||
update_data['email'] = value
|
||||
elif path == 'displayName':
|
||||
update_data['name'] = value
|
||||
elif path == 'emails[primary eq true].value':
|
||||
update_data['email'] = value
|
||||
elif path == 'name.formatted':
|
||||
update_data['name'] = value
|
||||
elif path in fields:
|
||||
if not isinstance(value, str):
|
||||
return scim_error(400, f'{path} must be a string', 'invalidValue')
|
||||
update_data[fields[path]] = value
|
||||
elif path == 'externalId':
|
||||
if value is not None and not isinstance(value, str):
|
||||
return scim_error(400, 'externalId must be a string or null', 'invalidValue')
|
||||
provider = get_scim_provider()
|
||||
await Users.update_user_scim_by_id(user_id, provider, value, db=db)
|
||||
scim = dict(update_data.get('scim', user.scim) or {})
|
||||
scim[provider] = {'external_id': value}
|
||||
update_data['scim'] = scim
|
||||
else:
|
||||
return scim_error(400, f'Unsupported PATCH path: {path}', 'invalidPath')
|
||||
|
||||
# Validate all operations before persisting once, and leave identical writes unchanged.
|
||||
update_data = {key: value for key, value in update_data.items() if value != getattr(user, key)}
|
||||
user_updated_fields = ['externalId' if field == 'scim' else field for field in update_data if field != 'role']
|
||||
|
||||
# Update user
|
||||
if update_data:
|
||||
update_data['updated_at'] = int(time.time())
|
||||
updated_user = await Users.update_user_by_id(user_id, update_data, db=db)
|
||||
if not updated_user:
|
||||
raise HTTPException(
|
||||
|
|
@ -788,7 +817,6 @@ async def patch_user(
|
|||
updated_user = user
|
||||
|
||||
role_changed = updated_user.role != user.role
|
||||
user_updated_fields = [field for field in update_data.keys() if field != 'role']
|
||||
|
||||
if user_updated_fields:
|
||||
await publish_event(
|
||||
|
|
|
|||
|
|
@ -33,13 +33,18 @@ from open_webui.utils.terminals import (
|
|||
)
|
||||
from starlette.background import BackgroundTask
|
||||
from starlette.requests import ClientDisconnect
|
||||
from yarl import URL
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
STREAMING_CONTENT_TYPES = ('application/octet-stream', 'image/', 'application/pdf')
|
||||
STRIPPED_RESPONSE_HEADERS = frozenset(('transfer-encoding', 'connection', 'content-encoding', 'content-length'))
|
||||
ADMIN_API_PATHS = ('api/v1/policies', 'api/v1/status', 'api/v1/terminals')
|
||||
# Drop the upstream's server and date: uvicorn adds its own and forwarding both duplicates them.
|
||||
STRIPPED_RESPONSE_HEADERS = frozenset(
|
||||
('transfer-encoding', 'connection', 'content-encoding', 'content-length', 'server', 'date')
|
||||
)
|
||||
|
||||
|
||||
def _sanitize_proxy_path(path: str) -> str | None:
|
||||
|
|
@ -62,7 +67,8 @@ def _sanitize_proxy_path(path: str) -> str | None:
|
|||
return None
|
||||
# posixpath splits on '/' only, so 'a/..\..\b' survives normpath as one component.
|
||||
# Upstreams that treat '\' as a separator would resolve it, so reject outright.
|
||||
if '\\' in decoded:
|
||||
# URL parsers also remove tabs/newlines, which can turn '.\t.' into '..'.
|
||||
if any(char in decoded for char in '\\\t\r\n'):
|
||||
return None
|
||||
had_trailing_slash = decoded.endswith('/')
|
||||
normalized = posixpath.normpath(decoded)
|
||||
|
|
@ -130,6 +136,15 @@ async def proxy_terminal(
|
|||
|
||||
target_url = f'{base_url}/{safe_path}'
|
||||
|
||||
# Check the path aiohttp will send, relative to the configured server root.
|
||||
base_path = URL(str(connection.get('url') or '')).path.rstrip('/')
|
||||
target_path = URL(target_url).path
|
||||
if any(
|
||||
target_path == f'{base_path}/{prefix}' or target_path.startswith(f'{base_path}/{prefix}/')
|
||||
for prefix in ADMIN_API_PATHS
|
||||
):
|
||||
return JSONResponse({'error': 'Path not allowed'}, status_code=403)
|
||||
|
||||
if request.query_params:
|
||||
target_url += f'?{request.query_params}'
|
||||
|
||||
|
|
@ -145,16 +160,14 @@ async def proxy_terminal(
|
|||
return JSONResponse({'error': 'A saved chat is required for this terminal'}, status_code=409)
|
||||
if context_id:
|
||||
headers[TERMINAL_CONTEXT_HEADER] = context_id
|
||||
cookies = {}
|
||||
cookies = getattr(request, 'cookies', {}) if connection.get('forward_cookies', False) else {}
|
||||
auth_type = connection.get('auth_type', 'bearer')
|
||||
|
||||
if auth_type == 'bearer':
|
||||
headers.update(bearer_auth_header(connection.get('key', '')))
|
||||
elif auth_type == 'session':
|
||||
cookies = request.cookies
|
||||
headers.update(bearer_auth_header(request.state.token.credentials))
|
||||
elif auth_type == 'system_oauth':
|
||||
cookies = request.cookies
|
||||
# Resolve the token server-side from the caller's OAuth session; never trust a client header.
|
||||
oauth_token = None
|
||||
try:
|
||||
|
|
@ -188,6 +201,7 @@ async def proxy_terminal(
|
|||
cookies=cookies,
|
||||
data=body or None,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
allow_redirects=False,
|
||||
)
|
||||
|
||||
upstream_content_type = upstream_response.headers.get('content-type', '')
|
||||
|
|
@ -326,7 +340,7 @@ async def ws_terminal(
|
|||
# For orchestrator-backed servers, pass user_id
|
||||
upstream_params['user_id'] = user.id
|
||||
context_id = terminal_context_id(connection, {'chat_id': chat_id}, 'chat')
|
||||
upstream_headers = {}
|
||||
upstream_headers = {'X-User-Id': user.id, 'X-Session-Id': chat_id}
|
||||
if terminal_context_config(connection, 'chat').get('context_id') == 'chat_id' and not context_id:
|
||||
await ws.close(code=4003, reason='A saved chat is required for this terminal')
|
||||
return
|
||||
|
|
@ -362,6 +376,8 @@ async def ws_terminal(
|
|||
await upstream.send_str(_json.dumps({'type': 'auth', 'token': key}))
|
||||
elif auth_type == 'session' and is_terminal_orchestrator(connection):
|
||||
await upstream.send_str(_json.dumps({'type': 'auth', 'token': token}))
|
||||
else:
|
||||
await upstream.send_str(_json.dumps({'type': 'auth', 'token': ''}))
|
||||
|
||||
await publish_event(
|
||||
app,
|
||||
|
|
|
|||
|
|
@ -27,11 +27,12 @@ from open_webui.models.tools import (
|
|||
)
|
||||
from open_webui.utils.access_control import (
|
||||
filter_allowed_access_grants,
|
||||
has_access,
|
||||
has_connection_access,
|
||||
has_permission,
|
||||
)
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.plugin import (
|
||||
get_tool_contents_cache,
|
||||
get_tools_cache,
|
||||
get_tool_module_from_cache,
|
||||
load_tool_module_by_id,
|
||||
|
|
@ -101,7 +102,7 @@ async def get_tools(
|
|||
)
|
||||
|
||||
# OpenAPI Tool Servers
|
||||
server_access_grants = {}
|
||||
server_connections = {}
|
||||
for server in await get_tool_servers(request):
|
||||
server_idx = server.get('idx', 0)
|
||||
connections = await Config.get('tool_server.connections', [])
|
||||
|
|
@ -112,10 +113,8 @@ async def get_tools(
|
|||
)
|
||||
continue
|
||||
connection = connections[server_idx]
|
||||
server_config = connection.get('config', {})
|
||||
|
||||
server_id = f'server:{server.get("id")}'
|
||||
server_access_grants[server_id] = server_config.get('access_grants', [])
|
||||
server_connections[server_id] = connection
|
||||
|
||||
tools.append(
|
||||
ToolUserResponse(
|
||||
|
|
@ -148,10 +147,8 @@ async def get_tools(
|
|||
user.id, f'mcp:{server_id}'
|
||||
)
|
||||
|
||||
server_config = server.get('config') or {}
|
||||
|
||||
tool_id = f'server:mcp:{info.get("id")}'
|
||||
server_access_grants[tool_id] = server_config.get('access_grants', [])
|
||||
server_connections[tool_id] = server
|
||||
|
||||
tools.append(
|
||||
ToolUserResponse(
|
||||
|
|
@ -180,12 +177,10 @@ async def get_tools(
|
|||
tool
|
||||
for tool in tools
|
||||
if not str(tool.id).startswith('server:')
|
||||
or await has_access(
|
||||
user.id,
|
||||
'read',
|
||||
server_access_grants.get(str(tool.id), []),
|
||||
or await has_connection_access(
|
||||
user,
|
||||
server_connections[str(tool.id)],
|
||||
user_group_ids,
|
||||
db=db,
|
||||
)
|
||||
]
|
||||
|
||||
|
|
@ -339,6 +334,7 @@ async def export_tools(
|
|||
return await Tools.get_tools(
|
||||
db=db,
|
||||
user_id=None if bypass_access_control else user.id,
|
||||
permission='write',
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -680,6 +676,8 @@ async def delete_tools_by_id(
|
|||
if result:
|
||||
TOOLS = get_tools_cache(request)
|
||||
TOOLS.pop(id, None)
|
||||
TOOL_CONTENTS = get_tool_contents_cache(request)
|
||||
TOOL_CONTENTS.pop(id, None)
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.TOOL_DELETED,
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ from open_webui.models.chats import Chats
|
|||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.oauth_sessions import OAuthSessions
|
||||
from open_webui.models.users import (
|
||||
InterfaceSettings,
|
||||
UserGroupIdsListResponse,
|
||||
UserGroupIdsModel,
|
||||
UserInfoListResponse,
|
||||
|
|
@ -68,23 +69,6 @@ def merge_user_ui_settings(defaults: dict, settings: dict) -> dict:
|
|||
return merged
|
||||
|
||||
|
||||
def strip_default_interface_settings(defaults: dict, settings: dict) -> dict:
|
||||
stripped = {}
|
||||
for key, value in settings.items():
|
||||
if value is None:
|
||||
continue
|
||||
|
||||
default_value = defaults.get(key)
|
||||
if isinstance(default_value, dict) and isinstance(value, dict):
|
||||
nested = strip_default_interface_settings(default_value, value)
|
||||
if nested:
|
||||
stripped[key] = nested
|
||||
elif value != default_value:
|
||||
stripped[key] = value
|
||||
|
||||
return stripped
|
||||
|
||||
|
||||
############################
|
||||
# GetUsers
|
||||
# A house is only as strong as its care for the least of
|
||||
|
|
@ -502,16 +486,37 @@ async def update_user_settings_by_session_user(
|
|||
user=Depends(get_verified_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
if user.role != 'admin' and not await has_permission(
|
||||
user.id, 'settings.interface', await Config.get('user.permissions')
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=ERROR_MESSAGES.ACCESS_PROHIBITED,
|
||||
)
|
||||
|
||||
updated_user_settings = form_data.model_dump(exclude_unset=True)
|
||||
ui_settings = updated_user_settings.get('ui')
|
||||
|
||||
if isinstance(ui_settings, dict):
|
||||
if user.role != 'admin' and not await has_permission(
|
||||
user.id, 'settings.interface', await Config.get('user.permissions'), db=db
|
||||
):
|
||||
# Omitted fields are unchanged, so unauthorized Interface fields can be discarded.
|
||||
for key in InterfaceSettings.model_fields:
|
||||
ui_settings.pop(key, None)
|
||||
|
||||
if (
|
||||
user.role != 'admin'
|
||||
and 'system' in ui_settings
|
||||
and (
|
||||
not await has_permission(user.id, 'chat.controls', await Config.get('user.permissions'), db=db)
|
||||
or not await has_permission(user.id, 'chat.system_prompt', await Config.get('user.permissions'), db=db)
|
||||
)
|
||||
):
|
||||
ui_settings.pop('system', None)
|
||||
|
||||
if (
|
||||
user.role != 'admin'
|
||||
and 'params' in ui_settings
|
||||
and (
|
||||
not await has_permission(user.id, 'chat.controls', await Config.get('user.permissions'), db=db)
|
||||
or not await has_permission(user.id, 'chat.params', await Config.get('user.permissions'), db=db)
|
||||
)
|
||||
):
|
||||
ui_settings.pop('params', None)
|
||||
|
||||
if (
|
||||
user.role != 'admin'
|
||||
and ui_settings is not None
|
||||
|
|
@ -522,8 +527,7 @@ async def update_user_settings_by_session_user(
|
|||
await Config.get('user.permissions'),
|
||||
)
|
||||
):
|
||||
# If the user is not an admin and does not have permission to use tool servers, remove the key
|
||||
updated_user_settings['ui'].pop('toolServers', None)
|
||||
ui_settings.pop('toolServers', None)
|
||||
|
||||
ui_notifications = ui_settings.get('notifications') if isinstance(ui_settings, dict) else None
|
||||
if (
|
||||
|
|
@ -542,11 +546,6 @@ async def update_user_settings_by_session_user(
|
|||
if isinstance(ui_notifications, dict):
|
||||
ui_notifications.pop('webhook_url', None)
|
||||
|
||||
default_interface_settings = await Config.get('ui.default_interface_settings')
|
||||
ui_settings = updated_user_settings.get('ui')
|
||||
if isinstance(default_interface_settings, dict) and isinstance(ui_settings, dict):
|
||||
updated_user_settings['ui'] = strip_default_interface_settings(default_interface_settings, ui_settings)
|
||||
|
||||
user = await Users.update_user_settings_by_id(user.id, updated_user_settings, db=db)
|
||||
if user:
|
||||
await publish_event(
|
||||
|
|
|
|||
|
|
@ -3,15 +3,13 @@ from __future__ import annotations
|
|||
import logging
|
||||
|
||||
import black
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from open_webui.config import DATA_DIR, ENABLE_ADMIN_EXPORT
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.models.chats import ChatTitleMessagesForm
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.code_interpreter import execute_code_jupyter
|
||||
from open_webui.utils.misc import get_gravatar_url
|
||||
from open_webui.utils.pdf_generator import PDFGenerator
|
||||
from pydantic import BaseModel
|
||||
from starlette.responses import FileResponse
|
||||
|
||||
|
|
@ -73,26 +71,6 @@ async def execute_code(request: Request, form_data: CodeForm, user=Depends(get_v
|
|||
)
|
||||
|
||||
|
||||
class ChatForm(BaseModel):
|
||||
title: str
|
||||
messages: list[dict]
|
||||
|
||||
|
||||
@router.post('/pdf')
|
||||
async def download_chat_as_pdf(form_data: ChatTitleMessagesForm, user=Depends(get_verified_user)):
|
||||
try:
|
||||
pdf_bytes = PDFGenerator(form_data).generate_chat_pdf()
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type='application/pdf',
|
||||
headers={'Content-Disposition': 'attachment;filename=chat.pdf'},
|
||||
)
|
||||
except Exception as e:
|
||||
log.exception(f'Error generating PDF: {e}')
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.get('/db/download')
|
||||
async def download_db(user=Depends(get_admin_user)):
|
||||
"""Download the raw SQLite database file (admin-only, SQLite deployments only)."""
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import copy
|
||||
import logging
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
from contextlib import suppress
|
||||
from typing import Any
|
||||
|
||||
import pycrdt as Y
|
||||
|
|
@ -37,17 +39,23 @@ from open_webui.models.folders import Folders
|
|||
from open_webui.models.notes import Notes, NoteUpdateForm
|
||||
from open_webui.models.users import UserNameResponse, Users
|
||||
from open_webui.socket.utils import CachedRedisDict, RedisDict, RedisLock, YdocManager
|
||||
from open_webui.tasks import create_task, stop_item_tasks
|
||||
from open_webui.tasks import (
|
||||
REDIS_PUBSUB_MAX_RECONNECT_INTERVAL,
|
||||
REDIS_PUBSUB_RECONNECT_INTERVAL,
|
||||
create_task,
|
||||
stop_item_tasks,
|
||||
)
|
||||
from open_webui.utils.access_control import has_permission
|
||||
from open_webui.utils.auth import get_verified_user_by_token
|
||||
from open_webui.utils.chat_id import is_saved_chat_id
|
||||
from open_webui.utils.json_codec import SOCKETIO_JSON
|
||||
from open_webui.utils.json_codec import SOCKETIO_JSON, JSONCodec, dumps_bytes
|
||||
from open_webui.utils.misc import get_output_text
|
||||
from open_webui.utils.redis import (
|
||||
build_sentinel_url,
|
||||
get_redis_connection,
|
||||
get_sentinels_from_env,
|
||||
)
|
||||
from redis.exceptions import RedisError
|
||||
from socketio.packet import Packet
|
||||
|
||||
logging.basicConfig(stream=sys.stdout, level=GLOBAL_LOG_LEVEL)
|
||||
|
|
@ -188,6 +196,11 @@ YDOC_MANAGER = YdocManager(
|
|||
redis_key_prefix=f'{REDIS_KEY_PREFIX}:ydoc:documents',
|
||||
)
|
||||
|
||||
REDIS_EVENT_CHANNEL = f'{REDIS_KEY_PREFIX}:direct_completion'
|
||||
|
||||
EVENT_QUEUES: dict[str, asyncio.Queue] = {}
|
||||
EVENT_PUBLISH_LOCK = asyncio.Lock()
|
||||
|
||||
|
||||
def get_session_pool_batches():
|
||||
"""All session pool entries, in bounded batches for the Redis backing."""
|
||||
|
|
@ -201,49 +214,53 @@ async def periodic_session_pool_cleanup():
|
|||
retry_delay = random.uniform(WEBSOCKET_REDIS_LOCK_TIMEOUT / 2, WEBSOCKET_REDIS_LOCK_TIMEOUT)
|
||||
renew_interval = max(WEBSOCKET_REDIS_LOCK_TIMEOUT / 2, 0.5)
|
||||
while True:
|
||||
if not session_aquire_func():
|
||||
log.debug('Session cleanup lock held by another node. Retrying.')
|
||||
await asyncio.sleep(retry_delay)
|
||||
continue
|
||||
|
||||
try:
|
||||
while True:
|
||||
if not session_renew_func():
|
||||
log.warning('Unable to renew session cleanup lock. Retrying cleanup ownership.')
|
||||
break
|
||||
if not session_aquire_func():
|
||||
log.debug('Session cleanup lock held by another node. Retrying.')
|
||||
await asyncio.sleep(retry_delay)
|
||||
continue
|
||||
|
||||
now = int(time.time())
|
||||
for batch in get_session_pool_batches():
|
||||
expired = [
|
||||
sid
|
||||
for sid, entry in batch
|
||||
if entry and now - entry.get('last_seen_at', 0) > SESSION_POOL_TIMEOUT
|
||||
]
|
||||
if expired:
|
||||
log.warning('Reaping %d orphaned session(s) from the session pool', len(expired))
|
||||
if WEBSOCKET_MANAGER == 'redis':
|
||||
SESSION_POOL.delete_many(*expired)
|
||||
else:
|
||||
for sid in expired:
|
||||
SESSION_POOL.pop(sid, None)
|
||||
await asyncio.sleep(0) # don't hold the loop for the whole sweep
|
||||
|
||||
next_cleanup_at = time.monotonic() + SESSION_POOL_TIMEOUT
|
||||
lock_lost = False
|
||||
try:
|
||||
while True:
|
||||
sleep_for = min(renew_interval, next_cleanup_at - time.monotonic())
|
||||
if sleep_for <= 0:
|
||||
break
|
||||
await asyncio.sleep(sleep_for)
|
||||
if not session_renew_func():
|
||||
log.warning('Unable to renew session cleanup lock. Retrying cleanup ownership.')
|
||||
lock_lost = True
|
||||
break
|
||||
|
||||
if lock_lost:
|
||||
break
|
||||
finally:
|
||||
session_release_func()
|
||||
now = int(time.time())
|
||||
for batch in get_session_pool_batches():
|
||||
expired = [
|
||||
sid
|
||||
for sid, entry in batch
|
||||
if entry and now - entry.get('last_seen_at', 0) > SESSION_POOL_TIMEOUT
|
||||
]
|
||||
if expired:
|
||||
log.warning('Reaping %d orphaned session(s) from the session pool', len(expired))
|
||||
if WEBSOCKET_MANAGER == 'redis':
|
||||
SESSION_POOL.delete_many(*expired)
|
||||
else:
|
||||
for sid in expired:
|
||||
SESSION_POOL.pop(sid, None)
|
||||
await asyncio.sleep(0) # don't hold the loop for the whole sweep
|
||||
|
||||
next_cleanup_at = time.monotonic() + SESSION_POOL_TIMEOUT
|
||||
lock_lost = False
|
||||
while True:
|
||||
sleep_for = min(renew_interval, next_cleanup_at - time.monotonic())
|
||||
if sleep_for <= 0:
|
||||
break
|
||||
await asyncio.sleep(sleep_for)
|
||||
if not session_renew_func():
|
||||
log.warning('Unable to renew session cleanup lock. Retrying cleanup ownership.')
|
||||
lock_lost = True
|
||||
break
|
||||
|
||||
if lock_lost:
|
||||
break
|
||||
finally:
|
||||
session_release_func()
|
||||
except Exception:
|
||||
log.exception('Session pool cleanup failed. Retrying.')
|
||||
await asyncio.sleep(retry_delay)
|
||||
|
||||
|
||||
async def periodic_usage_pool_cleanup():
|
||||
|
|
@ -822,27 +839,28 @@ async def yjs_document_update(sid, data):
|
|||
log.warning(f'User {user.get("id")} does not have write access to note {note_id}. Rejecting update.')
|
||||
return
|
||||
|
||||
user_id = data.get('user_id', sid)
|
||||
update = data.get('update') # List of bytes from frontend
|
||||
|
||||
update = data['update'] # List of bytes from frontend
|
||||
if update:
|
||||
user_id = data.get('user_id', sid)
|
||||
|
||||
await YDOC_MANAGER.append_to_updates(
|
||||
document_id=document_id,
|
||||
update=update, # Convert list of bytes to bytes
|
||||
)
|
||||
await YDOC_MANAGER.append_to_updates(
|
||||
document_id=document_id,
|
||||
update=update, # Convert list of bytes to bytes
|
||||
)
|
||||
|
||||
# Broadcast update to all other users in the document
|
||||
await sio.emit(
|
||||
'ydoc:document:update',
|
||||
{
|
||||
'document_id': document_id,
|
||||
'user_id': user_id,
|
||||
'update': update,
|
||||
'socket_id': sid, # Add socket_id to match frontend filtering
|
||||
},
|
||||
room=f'doc_{document_id}',
|
||||
skip_sid=sid,
|
||||
)
|
||||
# Broadcast update to all other users in the document
|
||||
await sio.emit(
|
||||
'ydoc:document:update',
|
||||
{
|
||||
'document_id': document_id,
|
||||
'user_id': user_id,
|
||||
'update': update,
|
||||
'socket_id': sid, # Add socket_id to match frontend filtering
|
||||
},
|
||||
room=f'doc_{document_id}',
|
||||
skip_sid=sid,
|
||||
)
|
||||
|
||||
async def debounced_save():
|
||||
await asyncio.sleep(0.5)
|
||||
|
|
@ -942,6 +960,62 @@ async def disconnect(sid, reason=None):
|
|||
# print(f"Unknown session ID {sid} disconnected")
|
||||
|
||||
|
||||
async def redis_event_listener() -> None:
|
||||
"""Route events received over Redis to their local queues."""
|
||||
reconnect_interval = REDIS_PUBSUB_RECONNECT_INTERVAL
|
||||
|
||||
while True:
|
||||
pubsub = None
|
||||
try:
|
||||
# RedisCluster can't route a pubsub subscribe until initialize() fills its slot cache.
|
||||
await REDIS.initialize()
|
||||
|
||||
pubsub = REDIS.pubsub()
|
||||
await pubsub.subscribe(REDIS_EVENT_CHANNEL)
|
||||
reconnect_interval = REDIS_PUBSUB_RECONNECT_INTERVAL
|
||||
|
||||
async for message in pubsub.listen():
|
||||
if message['type'] != 'message':
|
||||
continue
|
||||
event = JSONCodec.loads(message['data'])
|
||||
queue = EVENT_QUEUES.get(event['channel'])
|
||||
if queue is not None:
|
||||
await queue.put(event['data'])
|
||||
log.warning('Redis event listener stopped. Retrying.')
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
log.exception('Redis event listener failed. Retrying.')
|
||||
finally:
|
||||
if pubsub:
|
||||
with suppress(Exception):
|
||||
await pubsub.aclose()
|
||||
|
||||
await asyncio.sleep(reconnect_interval)
|
||||
reconnect_interval = min(reconnect_interval * 2, REDIS_PUBSUB_MAX_RECONNECT_INTERVAL)
|
||||
|
||||
|
||||
@sio.on('*')
|
||||
async def socket_event_handler(event: Any, sid: str, *args: Any) -> None:
|
||||
"""Route user-owned stream events to a local queue or another worker."""
|
||||
if not isinstance(event, str) or event.count(':') != 2 or not args:
|
||||
return
|
||||
|
||||
user = await get_socket_session_user(sid)
|
||||
if not user or user.get('id') != event.split(':', 1)[0]:
|
||||
return
|
||||
|
||||
queue = EVENT_QUEUES.get(event)
|
||||
if queue is not None:
|
||||
await queue.put(args[0])
|
||||
elif WEBSOCKET_MANAGER == 'redis':
|
||||
try:
|
||||
async with EVENT_PUBLISH_LOCK:
|
||||
await REDIS.publish(REDIS_EVENT_CHANNEL, dumps_bytes({'channel': event, 'data': args[0]}))
|
||||
except RedisError as e:
|
||||
log.debug('Failed to relay socket event %s: %s', event, e)
|
||||
|
||||
|
||||
async def _make_channel_emitter(request_info):
|
||||
"""Event emitter that routes pipeline output to a channel message.
|
||||
|
||||
|
|
@ -1005,6 +1079,9 @@ async def _make_channel_emitter(request_info):
|
|||
if not content and not output and not done:
|
||||
return
|
||||
|
||||
if isinstance(output, list):
|
||||
state['output'] = copy.deepcopy(output)
|
||||
|
||||
now = time.time()
|
||||
if done or (now - state['last_emit_at']) >= THROTTLE_INTERVAL:
|
||||
state['last_emit_at'] = now
|
||||
|
|
|
|||
|
|
@ -3,12 +3,16 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
import pycrdt as Y
|
||||
from open_webui.env import REDIS_KEY_PREFIX
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.redis import get_redis_connection
|
||||
from redis.exceptions import RedisClusterException, RedisError
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
YDOC_KEY_PREFIX = f'{REDIS_KEY_PREFIX}:ydoc:documents'
|
||||
SCAN_BATCH_SIZE = 200
|
||||
|
|
@ -58,7 +62,10 @@ class RedisLock:
|
|||
return bool(self.redis.eval(self._RENEW_SCRIPT, 1, self.lock_name, self.lock_id, self.timeout_secs))
|
||||
|
||||
def release_lock(self):
|
||||
self.redis.eval(self._RELEASE_SCRIPT, 1, self.lock_name, self.lock_id)
|
||||
try:
|
||||
self.redis.eval(self._RELEASE_SCRIPT, 1, self.lock_name, self.lock_id)
|
||||
except (RedisClusterException, RedisError) as e:
|
||||
log.warning('Failed to release lock %s; it expires on its own: %s', self.lock_name, e)
|
||||
|
||||
|
||||
class RedisDict:
|
||||
|
|
|
|||
|
|
@ -1,315 +0,0 @@
|
|||
/* HTML and Body */
|
||||
@font-face {
|
||||
font-family: 'NotoSans';
|
||||
src: url('fonts/NotoSans-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansJP';
|
||||
src: url('fonts/NotoSansJP-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansKR';
|
||||
src: url('fonts/NotoSansKR-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansSC';
|
||||
src: url('fonts/NotoSansSC-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansSC-Regular';
|
||||
src: url('fonts/NotoSansSC-Regular.ttf');
|
||||
}
|
||||
|
||||
html {
|
||||
font-family:
|
||||
-apple-system, BlinkMacSystemFont, 'Segoe UI', 'NotoSans', 'NotoSansJP', 'NotoSansKR',
|
||||
'NotoSansSC', 'Twemoji', 'STSong-Light', 'MSung-Light', 'HeiseiMin-W3', 'HYSMyeongJo-Medium',
|
||||
Roboto, 'Helvetica Neue', Arial, sans-serif;
|
||||
font-size: 14px; /* Default font size */
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
*,
|
||||
*::before,
|
||||
*::after {
|
||||
box-sizing: inherit;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background-color: #fff;
|
||||
width: auto;
|
||||
}
|
||||
|
||||
/* Typography */
|
||||
h1,
|
||||
h2,
|
||||
h3,
|
||||
h4,
|
||||
h5,
|
||||
h6 {
|
||||
font-weight: 500;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 2.5rem;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 2rem;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 1.75rem;
|
||||
}
|
||||
|
||||
h4 {
|
||||
font-size: 1.5rem;
|
||||
}
|
||||
|
||||
h5 {
|
||||
font-size: 1.25rem;
|
||||
}
|
||||
|
||||
h6 {
|
||||
font-size: 1rem;
|
||||
}
|
||||
|
||||
p {
|
||||
margin-top: 0;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
/* Grid System */
|
||||
.container {
|
||||
width: 100%;
|
||||
padding-right: 15px;
|
||||
padding-left: 15px;
|
||||
margin-right: auto;
|
||||
margin-left: auto;
|
||||
}
|
||||
|
||||
/* Utilities */
|
||||
.text-center {
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
/* Additional Text Utilities */
|
||||
.text-muted {
|
||||
color: #6c757d; /* Muted text color */
|
||||
}
|
||||
|
||||
/* Small Text */
|
||||
small {
|
||||
font-size: 80%; /* Smaller font size relative to the base */
|
||||
color: #6c757d; /* Lighter text color for secondary information */
|
||||
margin-bottom: 0;
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
/* Strong Element Styles */
|
||||
strong {
|
||||
font-weight: bolder; /* Ensures the text is bold */
|
||||
color: inherit; /* Inherits the color from its parent element */
|
||||
}
|
||||
|
||||
/* link */
|
||||
a {
|
||||
color: #007bff;
|
||||
text-decoration: none;
|
||||
background-color: transparent;
|
||||
}
|
||||
|
||||
a:hover {
|
||||
color: #0056b3;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* General styles for lists */
|
||||
ol,
|
||||
ul,
|
||||
li {
|
||||
padding-left: 40px; /* Increase padding to move bullet points to the right */
|
||||
margin-left: 20px; /* Indent lists from the left */
|
||||
}
|
||||
|
||||
/* Ordered list styles */
|
||||
ol {
|
||||
list-style-type: decimal; /* Use numbers for ordered lists */
|
||||
margin-bottom: 10px; /* Space after each list */
|
||||
}
|
||||
|
||||
ol li {
|
||||
margin-bottom: 0.5rem; /* Space between ordered list items */
|
||||
}
|
||||
|
||||
/* Unordered list styles */
|
||||
ul {
|
||||
list-style-type: disc; /* Use bullets for unordered lists */
|
||||
margin-bottom: 10px; /* Space after each list */
|
||||
}
|
||||
|
||||
ul li {
|
||||
margin-bottom: 0.5rem; /* Space between unordered list items */
|
||||
}
|
||||
|
||||
/* List item styles */
|
||||
li {
|
||||
margin-bottom: 5px; /* Space between list items */
|
||||
line-height: 1.5; /* Line height for better readability */
|
||||
}
|
||||
|
||||
/* Nested lists */
|
||||
ol ol,
|
||||
ol ul,
|
||||
ul ol,
|
||||
ul ul {
|
||||
padding-left: 20px;
|
||||
margin-left: 30px; /* Further indent nested lists */
|
||||
margin-bottom: 0; /* Remove extra margin at the bottom of nested lists */
|
||||
}
|
||||
|
||||
/* Code blocks */
|
||||
pre {
|
||||
background-color: #f4f4f4;
|
||||
padding: 10px;
|
||||
overflow-x: auto;
|
||||
max-width: 100%; /* Ensure it doesn't overflow the page */
|
||||
width: 80%; /* Set a specific width for a container-like appearance */
|
||||
margin: 0 1em; /* Center the pre block */
|
||||
box-sizing: border-box; /* Include padding in the width */
|
||||
border: 1px solid #ccc; /* Optional: Add a border for better definition */
|
||||
border-radius: 4px; /* Optional: Add rounded corners */
|
||||
}
|
||||
|
||||
code {
|
||||
font-family: 'Courier New', Courier, monospace;
|
||||
background-color: #f4f4f4;
|
||||
padding: 2px 4px;
|
||||
border-radius: 4px;
|
||||
box-sizing: border-box; /* Include padding in the width */
|
||||
}
|
||||
|
||||
.message {
|
||||
margin-top: 8px;
|
||||
margin-bottom: 8px;
|
||||
max-width: 100%;
|
||||
overflow-wrap: break-word;
|
||||
}
|
||||
|
||||
/* Table Styles */
|
||||
table {
|
||||
width: 100%;
|
||||
margin-bottom: 1rem;
|
||||
color: #212529;
|
||||
border-collapse: collapse; /* Removes the space between borders */
|
||||
}
|
||||
|
||||
th,
|
||||
td {
|
||||
margin: 0;
|
||||
padding: 0.75rem;
|
||||
vertical-align: top;
|
||||
border-top: 1px solid #dee2e6;
|
||||
}
|
||||
|
||||
thead th {
|
||||
vertical-align: bottom;
|
||||
border-bottom: 2px solid #dee2e6;
|
||||
}
|
||||
|
||||
tbody + tbody {
|
||||
border-top: 2px solid #dee2e6;
|
||||
}
|
||||
|
||||
/* markdown-section styles */
|
||||
.markdown-section blockquote,
|
||||
.markdown-section h1,
|
||||
.markdown-section h2,
|
||||
.markdown-section h3,
|
||||
.markdown-section h4,
|
||||
.markdown-section h5,
|
||||
.markdown-section h6,
|
||||
.markdown-section p,
|
||||
.markdown-section pre,
|
||||
.markdown-section table,
|
||||
.markdown-section ul {
|
||||
/* Give most block elements margin top and bottom */
|
||||
margin-top: 1rem;
|
||||
}
|
||||
|
||||
/* Remove top margin if it's the first child */
|
||||
.markdown-section blockquote:first-child,
|
||||
.markdown-section h1:first-child,
|
||||
.markdown-section h2:first-child,
|
||||
.markdown-section h3:first-child,
|
||||
.markdown-section h4:first-child,
|
||||
.markdown-section h5:first-child,
|
||||
.markdown-section h6:first-child,
|
||||
.markdown-section p:first-child,
|
||||
.markdown-section pre:first-child,
|
||||
.markdown-section table:first-child,
|
||||
.markdown-section ul:first-child {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
/* Remove top margin of <ul> following a <p> */
|
||||
.markdown-section p + ul {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
/* Remove bottom margin of <p> if it is followed by a <ul> */
|
||||
/* Note: :has is not supported in CSS, so you would need JavaScript for this behavior */
|
||||
.markdown-section p {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
/* List item styles */
|
||||
.markdown-section li {
|
||||
padding: 2px;
|
||||
}
|
||||
|
||||
.markdown-section li p {
|
||||
margin-bottom: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
/* Avoid margins for nested lists */
|
||||
.markdown-section li > ul {
|
||||
margin-top: 0;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
/* Table styles */
|
||||
.markdown-section table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
|
||||
.markdown-section th,
|
||||
.markdown-section td {
|
||||
border: 1px solid #ddd;
|
||||
padding: 0.5rem;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.markdown-section th {
|
||||
background-color: #f2f2f2;
|
||||
}
|
||||
|
||||
.markdown-section pre {
|
||||
padding: 10px;
|
||||
margin: 10px;
|
||||
}
|
||||
|
||||
.markdown-section pre code {
|
||||
position: relative;
|
||||
color: rgb(172, 0, 95);
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -5,14 +5,6 @@ import shutil
|
|||
from abc import ABC, abstractmethod
|
||||
from typing import BinaryIO, Dict, Tuple
|
||||
|
||||
import boto3
|
||||
from azure.core.exceptions import ResourceNotFoundError
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from azure.storage.blob import BlobServiceClient
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
from google.cloud import storage
|
||||
from google.cloud.exceptions import GoogleCloudError, NotFound
|
||||
from open_webui.config import (
|
||||
AZURE_STORAGE_CONTAINER_NAME,
|
||||
AZURE_STORAGE_ENDPOINT,
|
||||
|
|
@ -34,6 +26,18 @@ from open_webui.config import (
|
|||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
|
||||
from open_webui.env import USE_SLIM
|
||||
|
||||
if not USE_SLIM:
|
||||
import boto3
|
||||
from azure.core.exceptions import ResourceNotFoundError
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from azure.storage.blob import BlobServiceClient
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
from google.cloud import storage
|
||||
from google.cloud.exceptions import GoogleCloudError, NotFound
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
|
@ -332,6 +336,10 @@ class AzureStorageProvider(StorageProvider):
|
|||
|
||||
|
||||
def get_storage_provider(storage_provider: str):
|
||||
if USE_SLIM and storage_provider != 'local':
|
||||
raise RuntimeError(
|
||||
'Slim requires local file storage. Set STORAGE_PROVIDER=local, or use the standard image to access cloud storage.'
|
||||
)
|
||||
if storage_provider == 'local':
|
||||
Storage = LocalStorageProvider()
|
||||
elif storage_provider == 's3':
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ from uuid import uuid4
|
|||
|
||||
from redis.asyncio import Redis
|
||||
|
||||
from open_webui.env import REDIS_KEY_PREFIX, REDIS_RESPONSE_STREAM_TTL
|
||||
from open_webui.env import REDIS_KEY_PREFIX, REDIS_RESPONSE_STREAM_TTL, REDIS_TASK_TTL
|
||||
from open_webui.utils.json_codec import JSONCodec, dumps_bytes
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -66,13 +66,28 @@ async def redis_task_command_listener(app):
|
|||
reconnect_interval = min(reconnect_interval * 2, REDIS_PUBSUB_MAX_RECONNECT_INTERVAL)
|
||||
|
||||
|
||||
async def redis_task_heartbeat(app):
|
||||
redis: Redis = app.state.redis
|
||||
while True:
|
||||
await asyncio.sleep(REDIS_TASK_TTL / 4)
|
||||
try:
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
for task_id in list(tasks):
|
||||
# EXPIRE cannot recreate a task already removed by cleanup.
|
||||
pipe.expire(f'{REDIS_TASKS_KEY}:{task_id}', REDIS_TASK_TTL)
|
||||
await pipe.execute()
|
||||
except Exception:
|
||||
log.exception('Redis task heartbeat failed')
|
||||
|
||||
|
||||
### ------------------------------
|
||||
### REDIS-ENABLED HANDLERS
|
||||
### ------------------------------
|
||||
|
||||
|
||||
async def redis_save_task(redis: Redis, task_id: str, item_id: str | None):
|
||||
pipe = redis.pipeline()
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
pipe.set(f'{REDIS_TASKS_KEY}:{task_id}', '1', ex=REDIS_TASK_TTL or None)
|
||||
pipe.hset(REDIS_TASKS_KEY, task_id, item_id or '')
|
||||
if item_id:
|
||||
pipe.sadd(f'{REDIS_ITEM_TASKS_KEY}:{item_id}', task_id)
|
||||
|
|
@ -80,25 +95,36 @@ async def redis_save_task(redis: Redis, task_id: str, item_id: str | None):
|
|||
|
||||
|
||||
async def redis_cleanup_task(redis: Redis, task_id: str, item_id: str | None):
|
||||
pipe = redis.pipeline()
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
pipe.delete(f'{REDIS_TASKS_KEY}:{task_id}')
|
||||
pipe.hdel(REDIS_TASKS_KEY, task_id)
|
||||
pipe.hdel(REDIS_RESPONSE_STREAMS_KEY, task_id)
|
||||
if item_id:
|
||||
pipe.srem(f'{REDIS_ITEM_TASKS_KEY}:{item_id}', task_id)
|
||||
await pipe.execute()
|
||||
# Remove the set key entirely if no tasks remain for this item
|
||||
if await redis.scard(f'{REDIS_ITEM_TASKS_KEY}:{item_id}') == 0:
|
||||
await redis.delete(f'{REDIS_ITEM_TASKS_KEY}:{item_id}')
|
||||
else:
|
||||
await pipe.execute()
|
||||
await pipe.execute()
|
||||
|
||||
|
||||
async def redis_list_tasks(redis: Redis) -> list[str]:
|
||||
return list(await redis.hkeys(REDIS_TASKS_KEY))
|
||||
async def redis_list_tasks(redis: Redis, item_id: str | None = None) -> list[str]:
|
||||
task_ids = list(
|
||||
await redis.smembers(f'{REDIS_ITEM_TASKS_KEY}:{item_id}')
|
||||
if item_id is not None
|
||||
else await redis.hkeys(REDIS_TASKS_KEY)
|
||||
)
|
||||
if not task_ids or REDIS_TASK_TTL == 0:
|
||||
return task_ids
|
||||
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
for task_id in task_ids:
|
||||
pipe.exists(f'{REDIS_TASKS_KEY}:{task_id}')
|
||||
|
||||
async def redis_list_item_tasks(redis: Redis, item_id: str) -> list[str]:
|
||||
return list(await redis.smembers(f'{REDIS_ITEM_TASKS_KEY}:{item_id}'))
|
||||
active = []
|
||||
for task_id, exists in zip(task_ids, await pipe.execute()):
|
||||
if exists:
|
||||
active.append(task_id)
|
||||
else:
|
||||
task_item_id = item_id if item_id is not None else await redis.hget(REDIS_TASKS_KEY, task_id)
|
||||
await redis_cleanup_task(redis, task_id, task_item_id or None)
|
||||
return active
|
||||
|
||||
|
||||
async def redis_send_command(redis: Redis, command: dict):
|
||||
|
|
@ -140,10 +166,11 @@ async def create_task(redis, coroutine, id=None, task_id=None):
|
|||
tasks[task_id] = task
|
||||
|
||||
# If an ID is provided, associate the task with that ID
|
||||
if item_tasks.get(id):
|
||||
item_tasks[id].append(task_id)
|
||||
else:
|
||||
item_tasks[id] = [task_id]
|
||||
if id:
|
||||
if item_tasks.get(id):
|
||||
item_tasks[id].append(task_id)
|
||||
else:
|
||||
item_tasks[id] = [task_id]
|
||||
|
||||
if redis:
|
||||
await redis_save_task(redis, task_id, id)
|
||||
|
|
@ -165,8 +192,8 @@ async def list_task_ids_by_item_id(redis, id):
|
|||
List all tasks associated with a specific ID.
|
||||
"""
|
||||
if redis:
|
||||
return await redis_list_item_tasks(redis, id)
|
||||
return item_tasks.get(id, [])
|
||||
return await redis_list_tasks(redis, id)
|
||||
return list(item_tasks.get(id, []))
|
||||
|
||||
|
||||
async def save_response_stream(
|
||||
|
|
@ -274,10 +301,10 @@ async def stop_item_tasks(redis: Redis, item_id: str):
|
|||
if not task_ids:
|
||||
return {'status': True, 'message': f'No tasks found for item {item_id}.'}
|
||||
|
||||
for task_id in task_ids:
|
||||
result = await stop_task(redis, task_id)
|
||||
if not result['status']:
|
||||
return result # Return the first failure
|
||||
# Cleanup mutates the local task list while cancellation is awaited.
|
||||
for task_id in list(task_ids):
|
||||
# A task that already finished needs no stopping; continue with the rest.
|
||||
await stop_task(redis, task_id)
|
||||
|
||||
return {'status': True, 'message': f'All tasks for item {item_id} stopped.'}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import asyncio
|
|||
import logging
|
||||
import time
|
||||
from typing import Literal, Optional
|
||||
from urllib.parse import unquote
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
|
|
@ -28,7 +29,7 @@ from open_webui.models.memories import Memories
|
|||
from open_webui.models.messages import Message, Messages
|
||||
from open_webui.models.notes import Notes
|
||||
from open_webui.models.users import UserModel
|
||||
from open_webui.retrieval.utils import get_content_from_url
|
||||
from open_webui.retrieval.utils import filter_source_metadata, get_content_from_url
|
||||
from open_webui.retrieval.vector.async_client import ASYNC_VECTOR_DB_CLIENT
|
||||
from open_webui.routers.images import (
|
||||
CreateImageForm,
|
||||
|
|
@ -525,6 +526,7 @@ async def ask_user(
|
|||
Use this when the next step depends on user intent, preference, or a tradeoff that cannot be inferred safely.
|
||||
|
||||
:param questions: 1-3 question objects, each with id, header, question, and 2-3 options. Each option needs label and description.
|
||||
List the option you recommend first; the UI labels the first option Recommended.
|
||||
:param allow_other: Whether users may enter a free-form answer instead of choosing one of the options
|
||||
:param timeout_ms: How long the browser should keep the prompt open before cancelling it
|
||||
:return: JSON with status and answers keyed by question id
|
||||
|
|
@ -1341,6 +1343,14 @@ async def replace_note_content(
|
|||
"""
|
||||
Update an existing note by replacing the whole markdown content or applying range operations.
|
||||
|
||||
Prefer "replace_range" when only part of the note changes.
|
||||
A "replace" operation must be the only operation in the request.
|
||||
start and end are 0-indexed character offsets into the markdown content from view_note.
|
||||
end is exclusive.
|
||||
Offsets never shift as operations are applied.
|
||||
Ranges must not overlap.
|
||||
expected is optional. When set, the request is rejected if the range's current text does not match it.
|
||||
|
||||
:param note_id: The ID of the note to update
|
||||
:param content: The new markdown content for a whole-note update
|
||||
:param operations: Optional note operations:
|
||||
|
|
@ -2452,6 +2462,7 @@ async def grep_chat_files(
|
|||
"""
|
||||
Search exact text across files attached to the current chat.
|
||||
Pass file_id from the attached_files block to search one file.
|
||||
Auto-detected regex uses RE2 syntax; no lookarounds/backreferences, and shorthand classes are ASCII-only.
|
||||
|
||||
:param pattern: The text pattern to search for
|
||||
:param file_id: Optional attached file ID to search within a single file
|
||||
|
|
@ -2489,7 +2500,7 @@ async def grep_chat_files(
|
|||
if not files_to_search:
|
||||
return JSONCodec.dumps({'error': 'No accessible files found'})
|
||||
|
||||
return _grep_file_models(files_to_search, pattern, case_insensitive, count_only)
|
||||
return await asyncio.to_thread(_grep_file_models, files_to_search, pattern, case_insensitive, count_only)
|
||||
except Exception as e:
|
||||
log.exception(f'grep_chat_files error: {e}')
|
||||
return JSONCodec.dumps({'error': str(e)})
|
||||
|
|
@ -2600,6 +2611,7 @@ async def query_chat_files(
|
|||
for idx, doc in enumerate(documents):
|
||||
metadata = metadatas[idx] if idx < len(metadatas) and isinstance(metadatas[idx], dict) else {}
|
||||
chunk = {
|
||||
**filter_source_metadata(metadata),
|
||||
'content': doc,
|
||||
'source': metadata.get('source', metadata.get('name', source_info.get('name', 'Unknown'))),
|
||||
'file_id': metadata.get('file_id', source_info.get('id', '')),
|
||||
|
|
@ -2627,6 +2639,7 @@ async def grep_knowledge_files(
|
|||
Search for exact text across knowledge files. Returns matching lines with line numbers.
|
||||
Unlike query_knowledge_files (semantic/vector search), this performs exact string matching.
|
||||
Automatically detects regex patterns (e.g. "error|warn", "version \\d+").
|
||||
Regex uses RE2 syntax; no lookarounds/backreferences, and shorthand character classes are ASCII-only.
|
||||
Helpful for literal strings, identifiers, error messages, or regex-style searches.
|
||||
|
||||
:param pattern: The text pattern to search for (regex auto-detected)
|
||||
|
|
@ -2727,7 +2740,7 @@ async def grep_knowledge_files(
|
|||
if not files_to_search:
|
||||
return JSONCodec.dumps({'error': 'No accessible files found'})
|
||||
|
||||
return _grep_file_models(files_to_search, pattern, case_insensitive, count_only)
|
||||
return await asyncio.to_thread(_grep_file_models, files_to_search, pattern, case_insensitive, count_only)
|
||||
|
||||
except Exception as e:
|
||||
log.exception(f'grep_knowledge_files error: {e}')
|
||||
|
|
@ -3312,6 +3325,7 @@ async def query_knowledge_files(
|
|||
|
||||
for idx, doc in enumerate(documents):
|
||||
chunk_info = {
|
||||
**filter_source_metadata(metadatas[idx]),
|
||||
'content': doc,
|
||||
'source': metadatas[idx].get('source', metadatas[idx].get('name', 'Unknown')),
|
||||
'file_id': metadatas[idx].get('file_id', ''),
|
||||
|
|
@ -3335,6 +3349,7 @@ async def query_knowledge_files(
|
|||
for idx, doc in enumerate(documents):
|
||||
metadata = metadatas[idx] if idx < len(metadatas) else {}
|
||||
chunk_info = {
|
||||
**filter_source_metadata(metadata),
|
||||
'content': doc,
|
||||
'source': metadata.get('source', metadata.get('name', knowledge.name)),
|
||||
'file_id': metadata.get('file_id', f'external-{knowledge.id}'),
|
||||
|
|
@ -3468,6 +3483,7 @@ async def view_skill(
|
|||
id: str,
|
||||
__request__: Request = None,
|
||||
__user__: dict = None,
|
||||
__metadata__: dict = None,
|
||||
) -> str:
|
||||
"""
|
||||
Load the full instructions of a skill by its id from the available skills manifest.
|
||||
|
|
@ -3483,6 +3499,16 @@ async def view_skill(
|
|||
return JSONCodec.dumps({'error': 'User context not available'})
|
||||
|
||||
try:
|
||||
terminal_skill_prefix = 'terminal:'
|
||||
if isinstance(id, str) and id.startswith(terminal_skill_prefix):
|
||||
from open_webui.utils.terminals import get_terminal_skill
|
||||
|
||||
skill_name = unquote(id.removeprefix(terminal_skill_prefix))
|
||||
skill = await get_terminal_skill(__request__, __user__, __metadata__ or {}, skill_name)
|
||||
if not skill:
|
||||
return JSONCodec.dumps({'error': f"Skill '{id}' not found"})
|
||||
return JSONCodec.dumps(skill, ensure_ascii=False)
|
||||
|
||||
from open_webui.models.access_grants import AccessGrants
|
||||
from open_webui.models.skills import Skills
|
||||
|
||||
|
|
@ -3742,7 +3768,7 @@ async def create_automation(
|
|||
|
||||
# Validate the RRULE
|
||||
try:
|
||||
validate_rrule(rrule, tz=user.timezone)
|
||||
await validate_rrule(rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
return JSONCodec.dumps({'error': f'Invalid schedule: {e}'})
|
||||
|
||||
|
|
@ -3768,7 +3794,7 @@ async def create_automation(
|
|||
is_active=True,
|
||||
)
|
||||
|
||||
automation = await Automations.insert(user_id, form, next_run_ns(rrule, tz=tz))
|
||||
automation = await Automations.insert(user_id, form, await next_run_ns(rrule, tz=tz))
|
||||
|
||||
return JSONCodec.dumps(
|
||||
{
|
||||
|
|
@ -3779,7 +3805,7 @@ async def create_automation(
|
|||
'model_id': model_id,
|
||||
'target': automation.data.get('target'),
|
||||
'is_active': automation.is_active,
|
||||
'next_runs': next_n_runs_ns(rrule, tz=tz),
|
||||
'next_runs': await next_n_runs_ns(rrule, tz=tz),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
|
@ -3850,7 +3876,7 @@ async def update_automation(
|
|||
# Validate RRULE if changed
|
||||
if rrule is not None:
|
||||
try:
|
||||
validate_rrule(new_rrule, tz=user.timezone)
|
||||
await validate_rrule(new_rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
return JSONCodec.dumps({'error': f'Invalid schedule: {e}'})
|
||||
|
||||
|
|
@ -3872,7 +3898,7 @@ async def update_automation(
|
|||
is_active=automation.is_active,
|
||||
)
|
||||
|
||||
updated = await Automations.update_by_id(automation_id, form, next_run_ns(new_rrule, tz=tz))
|
||||
updated = await Automations.update_by_id(automation_id, form, await next_run_ns(new_rrule, tz=tz))
|
||||
|
||||
return JSONCodec.dumps(
|
||||
{
|
||||
|
|
@ -3883,7 +3909,7 @@ async def update_automation(
|
|||
'model_id': new_model_id,
|
||||
'target': updated.data.get('target'),
|
||||
'is_active': updated.is_active,
|
||||
'next_runs': next_n_runs_ns(new_rrule, tz=tz),
|
||||
'next_runs': await next_n_runs_ns(new_rrule, tz=tz),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
|
@ -3951,7 +3977,7 @@ async def list_automations(
|
|||
'rrule': rrule,
|
||||
'is_active': item.is_active,
|
||||
'last_run_at': item.last_run_at,
|
||||
'next_runs': next_n_runs_ns(rrule, tz=user.timezone if user else None),
|
||||
'next_runs': await next_n_runs_ns(rrule, tz=user.timezone if user else None),
|
||||
}
|
||||
)
|
||||
|
||||
|
|
@ -3998,7 +4024,7 @@ async def toggle_automation(
|
|||
rrule = automation.data.get('rrule', '')
|
||||
toggled = await Automations.toggle(
|
||||
automation_id,
|
||||
next_run_ns(rrule, tz=user.timezone if user else None),
|
||||
await next_run_ns(rrule, tz=user.timezone if user else None),
|
||||
)
|
||||
|
||||
return JSONCodec.dumps(
|
||||
|
|
|
|||
|
|
@ -7,15 +7,17 @@ for AI models to interact with knowledge bases using commands they already know.
|
|||
Re-exported through builtin.py for consistent imports.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextvars
|
||||
import logging
|
||||
import re
|
||||
import shlex
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from contextlib import contextmanager
|
||||
from typing import Optional
|
||||
|
||||
import regex
|
||||
import re2
|
||||
from fastapi import Request
|
||||
|
||||
from open_webui.env import (
|
||||
|
|
@ -29,12 +31,9 @@ log = logging.getLogger(__name__)
|
|||
DEFAULT_HEAD_LINES = 10
|
||||
DEFAULT_TAIL_LINES = 10
|
||||
|
||||
# Matching time allowed per tool call. Backtracking cost is exponential in the length of the
|
||||
# matched text, so capping the pattern or the line does not bound it.
|
||||
# Total matching time allowed per tool call, checked between RE2's linear-time searches.
|
||||
MATCH_BUDGET_SECONDS = 2.0
|
||||
MAX_REGEX_QUANTIFIER_COUNT = 2_000
|
||||
MAX_REGEX_QUANTIFIER_EXPANSION = 100_000
|
||||
_COUNTED_QUANTIFIER_RE = re.compile(r'(?<!\\)\{(\d+)(?:,\d*)?\}')
|
||||
MAX_SEARCH_PATTERN_LENGTH = 4_096
|
||||
|
||||
|
||||
class MatchBudgetExceeded(Exception):
|
||||
|
|
@ -69,66 +68,55 @@ def match_budget():
|
|||
|
||||
|
||||
def is_regex_pattern(pattern: str) -> bool:
|
||||
"""Detect if a pattern looks like regex (|, .*, .+, \d, \w, \s, [...])."""
|
||||
r"""Detect if a pattern looks like regex (|, .*, .+, \d, \w, \s, [...])."""
|
||||
return (
|
||||
'|' in pattern
|
||||
or '.*' in pattern
|
||||
or '.+' in pattern
|
||||
or '.?' in pattern
|
||||
or '\d' in pattern
|
||||
or '\w' in pattern
|
||||
or '\s' in pattern
|
||||
or r'\d' in pattern
|
||||
or r'\w' in pattern
|
||||
or r'\s' in pattern
|
||||
or bool(re.search(r'\[.+\]', pattern))
|
||||
)
|
||||
|
||||
|
||||
def normalize_regex(pattern: str) -> str:
|
||||
"""Normalize POSIX BRE patterns to Python regex (\| → |)."""
|
||||
return pattern.replace('\\|', '|').replace('\|', '|')
|
||||
|
||||
|
||||
def validate_regex_quantifiers(pattern: str) -> str | None:
|
||||
"""Reject counted quantifiers that make regex compilation expand too much."""
|
||||
quantifier_expansion = 1
|
||||
for quantifier in _COUNTED_QUANTIFIER_RE.finditer(pattern):
|
||||
count_text = quantifier.group(1)
|
||||
count = int(count_text) if len(count_text) <= 6 else MAX_REGEX_QUANTIFIER_COUNT + 1
|
||||
if count > MAX_REGEX_QUANTIFIER_COUNT:
|
||||
return f'Regex quantifier counts over {MAX_REGEX_QUANTIFIER_COUNT:g} are not supported'
|
||||
|
||||
# ponytail: conservative expansion catches nested quantifier bombs without mirroring regex syntax.
|
||||
quantifier_expansion *= max(count, 1)
|
||||
if quantifier_expansion > MAX_REGEX_QUANTIFIER_EXPANSION:
|
||||
return 'Regex quantifiers expand too much, lower the counts'
|
||||
|
||||
return None
|
||||
r"""Normalize POSIX BRE patterns to Python regex (\| → |)."""
|
||||
# Two passes: an escaped backslash in front of a pipe leaves a second escape behind.
|
||||
return pattern.replace(r'\|', '|').replace(r'\|', '|')
|
||||
|
||||
|
||||
def build_matcher(pattern: str, case_insensitive: bool = False, use_regex: bool = False) -> tuple:
|
||||
"""Build a matcher function. Returns (match_fn, error_str_or_None)."""
|
||||
if len(pattern) > MAX_SEARCH_PATTERN_LENGTH:
|
||||
return None, f'Search patterns over {MAX_SEARCH_PATTERN_LENGTH} characters are not supported'
|
||||
|
||||
if not use_regex and is_regex_pattern(pattern):
|
||||
use_regex = True
|
||||
|
||||
if use_regex:
|
||||
normalized = normalize_regex(pattern)
|
||||
quantifier_error = validate_regex_quantifiers(normalized)
|
||||
if quantifier_error:
|
||||
return None, quantifier_error
|
||||
try:
|
||||
re_flags = regex.IGNORECASE if case_insensitive else 0
|
||||
compiled = regex.compile(normalized, re_flags)
|
||||
except regex.error as e:
|
||||
return None, f'Invalid regex: {e}'
|
||||
options = re2.Options()
|
||||
options.case_sensitive = not case_insensitive
|
||||
options.max_mem = 1 << 20 # Bound compiled programs and the engine's matching cache to 1 MiB.
|
||||
options.log_errors = False
|
||||
compiled = re2.compile(normalized, options=options)
|
||||
except re2.error as e:
|
||||
return None, f'Invalid or unsupported regex (RE2 syntax): {e}'
|
||||
|
||||
budget = _active_budget.get() or MatchBudget()
|
||||
|
||||
def matches(line: str) -> bool:
|
||||
started = time.monotonic()
|
||||
try:
|
||||
# A negative timeout disables it, so an exhausted budget must not reach search().
|
||||
if budget.remaining <= 0:
|
||||
raise TimeoutError
|
||||
return bool(compiled.search(line, timeout=budget.remaining))
|
||||
matched = bool(compiled.search(line))
|
||||
if time.monotonic() - started >= budget.remaining:
|
||||
raise TimeoutError
|
||||
return matched
|
||||
except TimeoutError:
|
||||
raise MatchBudgetExceeded(f'Search exceeded {MATCH_BUDGET_SECONDS:g}s, narrow the pattern') from None
|
||||
finally:
|
||||
|
|
@ -586,7 +574,7 @@ async def _kb_ls(args: list[str], flags: set[str], user: dict, model_knowledge:
|
|||
if flat_mode:
|
||||
# Flat mode: build full tree (legitimate use)
|
||||
tree = await _build_directory_tree(kb_id)
|
||||
for f in tree['files']:
|
||||
for f in _sort_files(tree['files'], flags):
|
||||
lines.append(f' {f["id"]} {f["path"]} {_fmt_size(f)} {_fmt_date(f)}')
|
||||
lines.append('')
|
||||
continue
|
||||
|
|
@ -609,7 +597,7 @@ async def _kb_ls(args: list[str], flags: set[str], user: dict, model_knowledge:
|
|||
# Show files at this level (filter from accessible files)
|
||||
accessible = await _get_accessible_files(user, model_knowledge, knowledge_id=kb_id)
|
||||
dir_files = [f for f in accessible if f['directory_id'] == target_dir_id]
|
||||
for f in dir_files:
|
||||
for f in _sort_files(dir_files, flags):
|
||||
lines.append(f' {f["id"]} {f["filename"]} {_fmt_size(f)} {_fmt_date(f)}')
|
||||
|
||||
if not subdirs and not dir_files:
|
||||
|
|
@ -618,13 +606,24 @@ async def _kb_ls(args: list[str], flags: set[str], user: dict, model_knowledge:
|
|||
|
||||
if direct_files and not target_kb_id and not dir_path:
|
||||
lines.append('Attached Files:')
|
||||
for f in direct_files:
|
||||
for f in _sort_files(direct_files, flags):
|
||||
lines.append(f' {f["id"]} {f["filename"]} {_fmt_size(f)} {_fmt_date(f)}')
|
||||
lines.append('')
|
||||
|
||||
return '\n'.join(lines).rstrip()
|
||||
|
||||
|
||||
def _sort_files(files: list[dict], flags: set[str]) -> list[dict]:
|
||||
"""ls-style ordering: by name or path, -t newest first, -S largest first, -r reverses."""
|
||||
if 't' in flags:
|
||||
files = sorted(files, key=lambda f: f.get('updated_at') or 0, reverse=True)
|
||||
elif 'S' in flags:
|
||||
files = sorted(files, key=lambda f: f.get('size') or 0, reverse=True)
|
||||
else:
|
||||
files = sorted(files, key=lambda f: f.get('path') or f['filename'])
|
||||
return files[::-1] if 'r' in flags else files
|
||||
|
||||
|
||||
def _fmt_size(f: dict) -> str:
|
||||
return f'{f["size"]:,} bytes' if f.get('size') else ''
|
||||
|
||||
|
|
@ -715,6 +714,10 @@ async def _kb_tail(
|
|||
return result
|
||||
|
||||
|
||||
def _match_lines(content: str, matches: Callable[[str], bool]) -> list[tuple[int, str]]:
|
||||
return [(i, line) for i, line in enumerate(content.split('\n'), 1) if matches(line)]
|
||||
|
||||
|
||||
async def _kb_grep(
|
||||
args: list[str], flags: set[str], user: dict, model_knowledge: list[dict] | None, piped_input: str | None = None
|
||||
) -> str:
|
||||
|
|
@ -740,17 +743,14 @@ async def _kb_grep(
|
|||
count_only = 'c' in flags
|
||||
use_regex = 'E' in flags
|
||||
|
||||
_matches, err = build_matcher(pattern, case_insensitive, use_regex)
|
||||
_matches, err = await asyncio.to_thread(build_matcher, pattern, case_insensitive, use_regex)
|
||||
if err:
|
||||
return err
|
||||
|
||||
# Grep on piped input
|
||||
if piped_input is not None:
|
||||
lines = piped_input.split('\n')
|
||||
matched = []
|
||||
for i, line in enumerate(lines, 1):
|
||||
if _matches(line):
|
||||
matched.append(f'{i}: {line}')
|
||||
found = await asyncio.to_thread(_match_lines, piped_input, _matches)
|
||||
matched = [f'{i}: {line}' for i, line in found]
|
||||
if count_only:
|
||||
return str(len(matched))
|
||||
if filenames_only:
|
||||
|
|
@ -766,11 +766,8 @@ async def _kb_grep(
|
|||
elif 'error' in resolved:
|
||||
return resolved['error']
|
||||
else:
|
||||
lines = resolved['content'].split('\n')
|
||||
matched = []
|
||||
for i, line in enumerate(lines, 1):
|
||||
if _matches(line):
|
||||
matched.append(f'{i}: {line}')
|
||||
found = await asyncio.to_thread(_match_lines, resolved['content'], _matches)
|
||||
matched = [f'{i}: {line}' for i, line in found]
|
||||
|
||||
if count_only:
|
||||
return f'{resolved["id"]} {resolved["filename"]}: {len(matched)}'
|
||||
|
|
@ -821,11 +818,7 @@ async def _kb_grep(
|
|||
if not content:
|
||||
continue
|
||||
|
||||
lines = content.split('\n')
|
||||
file_matches = []
|
||||
for i, line in enumerate(lines, 1):
|
||||
if _matches(line):
|
||||
file_matches.append((i, line))
|
||||
file_matches = await asyncio.to_thread(_match_lines, content, _matches)
|
||||
|
||||
if file_matches:
|
||||
files_with_matches.append(file_info)
|
||||
|
|
@ -892,7 +885,7 @@ async def _kb_find(args: list[str], flags: set[str], user: dict, model_knowledge
|
|||
return f'No files matching "{pattern}"{scope_str}'
|
||||
|
||||
lines = []
|
||||
for f in matched:
|
||||
for f in _sort_files(matched, flags):
|
||||
kb_info = f' ({f["knowledge_name"]})' if f.get('knowledge_name') else ''
|
||||
lines.append(f'{f["id"]} {f["filename"]}{kb_info}')
|
||||
return '\n'.join(lines)
|
||||
|
|
@ -1064,7 +1057,7 @@ async def _kb_tree(args: list[str], flags: set[str], user: dict, model_knowledge
|
|||
def _render_tree(parent_id, prefix=' '):
|
||||
items = []
|
||||
subdirs = _get_subdirs(tree, parent_id)
|
||||
files = _get_files_in_dir(tree, parent_id)
|
||||
files = _sort_files(_get_files_in_dir(tree, parent_id), flags)
|
||||
entries = [('dir', d) for d in subdirs] + [('file', f) for f in files]
|
||||
|
||||
for idx, (etype, entry) in enumerate(entries):
|
||||
|
|
@ -1089,7 +1082,7 @@ async def _kb_tree(args: list[str], flags: set[str], user: dict, model_knowledge
|
|||
|
||||
if direct_files and not dir_scope:
|
||||
output.append('Attached Files:')
|
||||
for idx, f in enumerate(direct_files):
|
||||
for idx, f in enumerate(_sort_files(direct_files, flags)):
|
||||
connector = '└── ' if idx == len(direct_files) - 1 else '├── '
|
||||
output.append(f' {connector}{f["filename"]}')
|
||||
output.append(f'\n 0 directories, {len(direct_files)} files')
|
||||
|
|
@ -1157,6 +1150,9 @@ async def kb_exec(
|
|||
ls — list root files and directories
|
||||
ls docs/ — list contents of a directory
|
||||
ls -a — flat list of all files with full paths
|
||||
ls -t — newest modified first
|
||||
ls -S — largest first
|
||||
ls -r — reverse file order
|
||||
tree — recursive directory tree view
|
||||
tree docs/ — subtree from a directory
|
||||
cat -n <file> — read file with line numbers
|
||||
|
|
@ -1171,11 +1167,13 @@ async def kb_exec(
|
|||
grep "text" *.py — filter by extension
|
||||
find "*.md" — find files by glob
|
||||
find docs/ "*.md" — find within a directory
|
||||
find -t "*.md", tree -t — same sort flags as ls
|
||||
wc <file> — line/word/char counts
|
||||
stat <file> — file metadata
|
||||
|
||||
Pipes: grep "auth" | head -5
|
||||
Files: reference by path (docs/api/auth.md), filename, or file ID
|
||||
Regex: RE2 syntax; no lookarounds/backreferences. Shorthand character classes are ASCII-only.
|
||||
|
||||
:param command: A filesystem command string
|
||||
:return: Command output as text
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import logging
|
||||
from typing import Any
|
||||
|
||||
from open_webui.config import DEFAULT_USER_PERMISSIONS
|
||||
|
|
@ -14,6 +15,8 @@ from open_webui.models.users import UserModel
|
|||
from open_webui.utils.json_codec import JSONCodec
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def fill_missing_permissions(permissions: dict[str, Any], default_permissions: dict[str, Any]) -> dict[str, Any]:
|
||||
"""
|
||||
|
|
@ -325,6 +328,13 @@ async def has_base_model_access(
|
|||
seen.add(base_model_id)
|
||||
base_model_info = await Models.get_model_by_id(base_model_id, db=db)
|
||||
if base_model_info is None:
|
||||
if user_role != 'admin':
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r base_model_id=%r reason=base_model_unregistered',
|
||||
user_id,
|
||||
model_info.id,
|
||||
base_model_id,
|
||||
)
|
||||
return user_role == 'admin'
|
||||
if not (
|
||||
user_id == base_model_info.user_id
|
||||
|
|
@ -337,6 +347,12 @@ async def has_base_model_access(
|
|||
db=db,
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r base_model_id=%r reason=base_model_read_denied',
|
||||
user_id,
|
||||
model_info.id,
|
||||
base_model_id,
|
||||
)
|
||||
return False
|
||||
base_model_id = getattr(base_model_info, 'base_model_id', None)
|
||||
return True
|
||||
|
|
@ -381,6 +397,11 @@ async def check_model_access(
|
|||
user_group_ids=user_group_ids,
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=model_read_denied',
|
||||
user.id,
|
||||
model_info.id,
|
||||
)
|
||||
raise HTTPException(status_code=403, detail='Model not found')
|
||||
|
||||
# Enforce access on chained base models
|
||||
|
|
@ -388,4 +409,5 @@ async def check_model_access(
|
|||
raise HTTPException(status_code=403, detail='Model not found')
|
||||
else:
|
||||
if user.role != 'admin':
|
||||
log.warning('Model access denied: user_id=%r reason=model_unregistered', user.id)
|
||||
raise HTTPException(status_code=403, detail='Model not found')
|
||||
|
|
|
|||
|
|
@ -63,28 +63,6 @@ async def has_access_to_file(
|
|||
) and (access_type == 'read' or knowledge_base.user_id == file.user_id):
|
||||
return True
|
||||
|
||||
knowledge_base_id = file.meta.get('collection_name') if file.meta else None
|
||||
if knowledge_base_id:
|
||||
# Fetch the one referenced knowledge base instead of listing every
|
||||
# knowledge base the user can access just to scan for this id.
|
||||
knowledge_base = await Knowledges.get_knowledge_by_id(knowledge_base_id, db=db)
|
||||
if (
|
||||
knowledge_base
|
||||
and (access_type == 'read' or knowledge_base.user_id == file.user_id)
|
||||
and (
|
||||
knowledge_base.user_id == user.id
|
||||
or await AccessGrants.has_access(
|
||||
user_id=user.id,
|
||||
resource_type='knowledge',
|
||||
resource_id=knowledge_base.id,
|
||||
permission=access_type,
|
||||
user_group_ids=user_group_ids,
|
||||
db=db,
|
||||
)
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
# Check if the file is associated with any channels the user has access to
|
||||
channels = await Channels.get_channels_by_file_id_and_user_id(file_id, user.id, db=db)
|
||||
if access_type == 'read' and channels:
|
||||
|
|
@ -106,7 +84,9 @@ async def has_access_to_file(
|
|||
|
||||
# Check if the file is directly attached to a shared workspace model (per the ownership
|
||||
# note above, model write is conferred only for files the model owner owns).
|
||||
model_owners = await Models.get_model_owners_attaching_file(file.id, db=db)
|
||||
model_owners = await Models.get_model_owner_ids_by_file_id(
|
||||
file.id, db=db, include_background=access_type == 'read'
|
||||
)
|
||||
if access_type != 'read':
|
||||
model_owners = {model_id: owner_id for model_id, owner_id in model_owners.items() if owner_id == file.user_id}
|
||||
if user.id in model_owners.values():
|
||||
|
|
|
|||
|
|
@ -202,7 +202,9 @@ def convert_anthropic_to_openai_payload(
|
|||
)
|
||||
)
|
||||
elif block_type in ('thinking', 'redacted_thinking'):
|
||||
openai_content.append(_copy_cache_control(block, dict(block)))
|
||||
# Unsigned thinking cannot be replayed upstream
|
||||
if block_type == 'redacted_thinking' or block.get('signature'):
|
||||
openai_content.append(_copy_cache_control(block, dict(block)))
|
||||
elif block_type == 'image':
|
||||
source = block.get('source', {})
|
||||
if source.get('type') == 'base64':
|
||||
|
|
@ -369,7 +371,7 @@ def convert_anthropic_to_openai_payload(
|
|||
msg_dict['content'] = ''
|
||||
msg_dict['tool_calls'] = tool_calls
|
||||
messages.append(msg_dict)
|
||||
elif openai_content:
|
||||
elif openai_content or role == 'assistant':
|
||||
messages.append({'role': role, 'content': _finalize_openai_content(openai_content)})
|
||||
else:
|
||||
messages.append({'role': role, 'content': str(content) if content else ''})
|
||||
|
|
|
|||
|
|
@ -8,6 +8,8 @@ import logging
|
|||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timedelta
|
||||
from threading import Lock
|
||||
from time import monotonic
|
||||
from typing import Optional, Union
|
||||
|
||||
import bcrypt
|
||||
|
|
@ -42,6 +44,7 @@ from open_webui.utils.access_control import has_permission
|
|||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.misc import parse_duration
|
||||
from pytz import UTC
|
||||
from redis.exceptions import RedisError
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -248,14 +251,41 @@ def decode_token(token: str) -> dict | None:
|
|||
return None
|
||||
|
||||
|
||||
class RateLimitFilter(logging.Filter):
|
||||
"""Limit a logger to one record per interval per process."""
|
||||
|
||||
def __init__(self, interval=60):
|
||||
super().__init__()
|
||||
self.interval = interval
|
||||
self.next_allowed = float('-inf')
|
||||
self.lock = Lock()
|
||||
|
||||
def filter(self, record):
|
||||
with self.lock:
|
||||
now = monotonic()
|
||||
if now < self.next_allowed:
|
||||
return False
|
||||
self.next_allowed = now + self.interval
|
||||
return True
|
||||
|
||||
|
||||
revocation_log = logging.getLogger(f'{__name__}.revocation')
|
||||
revocation_log.addFilter(RateLimitFilter())
|
||||
|
||||
|
||||
async def is_valid_token(decoded, redis=None) -> bool:
|
||||
"""
|
||||
Check whether a JWT has been revoked. Two mechanisms:
|
||||
1. Per-token (jti) — used by user-initiated sign-out (known jti).
|
||||
2. Per-user (revoked_at) — used by password changes and OIDC back-channel
|
||||
logout when individual jti values are unknown; rejects tokens with iat <= revoked_at.
|
||||
|
||||
Fail open on Redis errors to preserve availability; revoked tokens may be accepted.
|
||||
"""
|
||||
if redis:
|
||||
if not redis:
|
||||
return True
|
||||
|
||||
try:
|
||||
# Per-token revocation
|
||||
jti = decoded.get('jti')
|
||||
if jti:
|
||||
|
|
@ -276,6 +306,8 @@ async def is_valid_token(decoded, redis=None) -> bool:
|
|||
return False
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
except RedisError as e:
|
||||
revocation_log.warning('Revocation check failed; accepting token: %s', e)
|
||||
|
||||
return True
|
||||
|
||||
|
|
@ -296,6 +328,10 @@ async def invalidate_token(request, token):
|
|||
ttl = exp - int(datetime.now(UTC).timestamp()) # Calculate time-to-live for the token
|
||||
|
||||
if ttl > 0:
|
||||
# Revoked tokens must not be able to disconnect newer sessions.
|
||||
if not await is_valid_token(decoded, request.app.state.redis):
|
||||
return
|
||||
|
||||
# Store the revoked token in Redis with an expiration time
|
||||
await request.app.state.redis.set(
|
||||
f'{REDIS_KEY_PREFIX}:auth:token:{jti}:revoked',
|
||||
|
|
@ -303,6 +339,12 @@ async def invalidate_token(request, token):
|
|||
ex=ttl,
|
||||
)
|
||||
|
||||
user_id = decoded.get('id')
|
||||
if user_id:
|
||||
from open_webui.socket.main import disconnect_user_sessions
|
||||
|
||||
await disconnect_user_sessions(user_id)
|
||||
|
||||
|
||||
async def revoke_user_tokens(request, user_id: str):
|
||||
"""Reject every token already issued to a user. Requires Redis."""
|
||||
|
|
@ -324,6 +366,10 @@ async def revoke_user_tokens(request, user_id: str):
|
|||
ex=int(expires_delta.total_seconds()) if expires_delta else None,
|
||||
)
|
||||
|
||||
from open_webui.socket.main import disconnect_user_sessions
|
||||
|
||||
await disconnect_user_sessions(user_id)
|
||||
|
||||
|
||||
def extract_token_from_auth_header(auth_header: str):
|
||||
return auth_header[len('Bearer ') :]
|
||||
|
|
@ -386,6 +432,7 @@ async def get_current_user(
|
|||
|
||||
# Scope-backed, so outer middleware (audit) can reuse the resolved user
|
||||
request.state.user = user
|
||||
request.state.auth_type = 'api_key'
|
||||
return user
|
||||
|
||||
# auth by jwt token
|
||||
|
|
@ -437,6 +484,7 @@ async def get_current_user(
|
|||
|
||||
# Scope-backed, so outer middleware (audit) can reuse the resolved user
|
||||
request.state.user = user
|
||||
request.state.auth_type = 'jwt'
|
||||
return user
|
||||
else:
|
||||
raise HTTPException(
|
||||
|
|
|
|||
|
|
@ -20,12 +20,10 @@ import logging
|
|||
import os
|
||||
import random
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from datetime import timedelta
|
||||
from typing import Optional
|
||||
from uuid import uuid4
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from dateutil.rrule import HOURLY, MINUTELY, SECONDLY, rruleset, rrulestr
|
||||
from fastapi import Request
|
||||
from fastapi.security import HTTPAuthorizationCredentials
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
|
|
@ -39,6 +37,13 @@ from open_webui.models.messages import MessageForm
|
|||
from open_webui.models.users import Users
|
||||
from open_webui.utils.auth import create_token
|
||||
from open_webui.utils.misc import parse_duration
|
||||
from open_webui.utils.recurrence import (
|
||||
_resolve_tz,
|
||||
next_n_runs_ns,
|
||||
next_run_ns,
|
||||
rrule_interval_seconds,
|
||||
validate_rrule,
|
||||
)
|
||||
from open_webui.utils.task import prompt_template
|
||||
from open_webui.utils.terminals import get_terminal_server_url
|
||||
from starlette.datastructures import Headers
|
||||
|
|
@ -50,153 +55,6 @@ TIMER_POLL_INTERVAL = int(os.getenv('TIMER_POLL_INTERVAL', '1'))
|
|||
CALENDAR_ALERT_LOOKAHEAD_MINUTES = int(os.getenv('CALENDAR_ALERT_LOOKAHEAD_MINUTES', '10'))
|
||||
|
||||
|
||||
####################
|
||||
# RRULE Helpers
|
||||
####################
|
||||
|
||||
|
||||
def _resolve_tz(tz: str = None) -> Optional[ZoneInfo]:
|
||||
"""Safely resolve a timezone string to ZoneInfo.
|
||||
|
||||
Returns None (→ server-local fallback) when *tz* is empty, None,
|
||||
or an unrecognised IANA key. Logs a warning on bad keys so
|
||||
misconfiguration is visible in the server logs.
|
||||
"""
|
||||
if not tz:
|
||||
return None
|
||||
try:
|
||||
return ZoneInfo(tz)
|
||||
except (KeyError, Exception):
|
||||
log.warning('Unknown timezone %r — falling back to server time', tz)
|
||||
return None
|
||||
|
||||
|
||||
def _parse_rule(s: str, now: Optional[datetime] = None):
|
||||
"""Parse RRULE with clock-aligned DTSTART for sub-daily frequencies.
|
||||
|
||||
SECONDLY/MINUTELY/HOURLY rules use a fixed epoch DTSTART (2000-01-01 00:00)
|
||||
so intervals snap to clock boundaries (e.g. every 5min = :00, :05, :10).
|
||||
"""
|
||||
upper = s.upper()
|
||||
if 'EXRULE' in upper:
|
||||
raise ValueError('EXRULE is not supported in recurrence rules')
|
||||
|
||||
parsed = rrulestr(s, ignoretz=True)
|
||||
rules = parsed._rrule if isinstance(parsed, rruleset) else [parsed]
|
||||
if len(rules) > 1:
|
||||
raise ValueError('only one RRULE is supported per recurrence rule')
|
||||
|
||||
rule = rules[0]
|
||||
start = rule._dtstart.replace(tzinfo=None)
|
||||
anchor = now or datetime.now()
|
||||
lines = s.splitlines()
|
||||
stripped = '\n'.join(line for line in lines if not line.upper().startswith('DTSTART')) or s
|
||||
has_dtstart = any(line.upper().startswith('DTSTART') for line in lines)
|
||||
step = {
|
||||
SECONDLY: timedelta(seconds=rule._interval),
|
||||
MINUTELY: timedelta(minutes=rule._interval),
|
||||
HOURLY: timedelta(hours=rule._interval),
|
||||
}.get(rule._freq)
|
||||
|
||||
if step is None:
|
||||
if not rule._dtstart.tzinfo:
|
||||
return parsed
|
||||
return rrulestr(stripped, dtstart=start, ignoretz=True)
|
||||
|
||||
if rule._interval < 1:
|
||||
raise ValueError('RRULE INTERVAL must be a positive integer')
|
||||
dtstart = None
|
||||
if has_dtstart:
|
||||
emitted = ((anchor - start) // step) if anchor > start else 0
|
||||
emitted *= len(rule._byminute or (0,)) * len(rule._bysecond or (0,))
|
||||
if emitted <= 100_000:
|
||||
if rule._dtstart.tzinfo:
|
||||
dtstart = start
|
||||
else:
|
||||
return parsed
|
||||
if not has_dtstart or dtstart is None:
|
||||
epoch = datetime(2000, 1, 1)
|
||||
dtstart = epoch + ((anchor - epoch) // step) * step
|
||||
|
||||
return rrulestr(stripped, dtstart=dtstart, ignoretz=True)
|
||||
|
||||
|
||||
def validate_rrule(s: str, tz: str = None) -> None:
|
||||
"""Raise ValueError if the RRULE is malformed or exhausted.
|
||||
|
||||
When *tz* is provided the "now" reference uses the user's local
|
||||
clock so that near-future schedules are not incorrectly rejected
|
||||
on servers whose system clock is ahead (e.g. UTC vs US timezones).
|
||||
"""
|
||||
upper = s.upper()
|
||||
if 'COUNT=' in upper and 'DTSTART' not in upper:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_COUNT_REQUIRES_DTSTART)
|
||||
zi = _resolve_tz(tz)
|
||||
now = datetime.now(zi).replace(tzinfo=None) if zi else datetime.now()
|
||||
try:
|
||||
rule = _parse_rule(s, now)
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if rule.after(now) is None:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_NO_FUTURE_RUNS)
|
||||
|
||||
|
||||
def next_run_ns(s: str, tz: str = None) -> Optional[int]:
|
||||
"""Next occurrence as epoch nanoseconds, respecting user timezone."""
|
||||
zi = _resolve_tz(tz)
|
||||
now = datetime.now(zi) if zi else datetime.now()
|
||||
now_naive = now.replace(tzinfo=None)
|
||||
dt = _parse_rule(s, now_naive).after(now_naive)
|
||||
if dt is None:
|
||||
return None
|
||||
if zi:
|
||||
dt = dt.replace(tzinfo=zi)
|
||||
return int(dt.timestamp() * 1_000_000_000)
|
||||
|
||||
|
||||
def next_n_runs_ns(s: str, n: int = 5, tz: str = None) -> list[int]:
|
||||
"""Compute next N occurrences for UI preview.
|
||||
|
||||
Uses the user's timezone for the starting "now" so that the
|
||||
preview matches the user's local clock (same as next_run_ns).
|
||||
"""
|
||||
zi = _resolve_tz(tz)
|
||||
result = []
|
||||
now = datetime.now(zi).replace(tzinfo=None) if zi else datetime.now()
|
||||
rule = _parse_rule(s, now)
|
||||
dt = now
|
||||
for _ in range(n):
|
||||
dt = rule.after(dt)
|
||||
if not dt:
|
||||
break
|
||||
if zi:
|
||||
dt_tz = dt.replace(tzinfo=zi)
|
||||
result.append(int(dt_tz.timestamp() * 1_000_000_000))
|
||||
else:
|
||||
result.append(int(dt.timestamp() * 1_000_000_000))
|
||||
return result
|
||||
|
||||
|
||||
def rrule_interval_seconds(s: str) -> Optional[int]:
|
||||
"""Approximate interval between recurrences in seconds.
|
||||
|
||||
Returns None for one-shot (COUNT=1) schedules or rules
|
||||
with fewer than two future occurrences.
|
||||
"""
|
||||
if 'COUNT=1' in s:
|
||||
return None
|
||||
s = '\n'.join(line for line in s.splitlines() if not line.upper().startswith('DTSTART')) or s
|
||||
now = datetime.now()
|
||||
rule = _parse_rule(s, now)
|
||||
first = rule.after(now)
|
||||
if first is None:
|
||||
return None
|
||||
second = rule.after(first)
|
||||
if second is None:
|
||||
return None
|
||||
return int((second - first).total_seconds())
|
||||
|
||||
|
||||
############################
|
||||
# Worker Loop
|
||||
############################
|
||||
|
|
@ -305,17 +163,20 @@ def _build_request(
|
|||
return request
|
||||
|
||||
|
||||
async def _resolve_model_defaults(app, model_id: str) -> tuple[list[str], dict, list[str], Optional[str]]:
|
||||
async def _resolve_model_defaults(app, model_id: str) -> dict:
|
||||
models = getattr(app.state, 'MODELS', {})
|
||||
model = models.get(model_id, {})
|
||||
meta = model.get('info', {}).get('meta', {})
|
||||
|
||||
tool_ids = list(meta.get('toolIds') or [])
|
||||
filter_ids = list(meta.get('defaultFilterIds') or [])
|
||||
terminal_id = meta.get('terminalId') or None
|
||||
defaults = {
|
||||
'tool_ids': list(meta.get('toolIds') or []),
|
||||
'filter_ids': list(meta.get('defaultFilterIds') or []),
|
||||
'terminal_id': meta.get('terminalId'),
|
||||
}
|
||||
defaults = {key: value for key, value in defaults.items() if value}
|
||||
default_feature_ids = meta.get('defaultFeatureIds', [])
|
||||
if not default_feature_ids:
|
||||
return tool_ids, {}, filter_ids, terminal_id
|
||||
return defaults
|
||||
|
||||
capabilities = meta.get('capabilities') or {}
|
||||
features = {}
|
||||
|
|
@ -333,7 +194,9 @@ async def _resolve_model_defaults(app, model_id: str) -> tuple[list[str], dict,
|
|||
if capabilities.get(feature_id) and feature_checks[feature_id]:
|
||||
features[feature_id] = True
|
||||
|
||||
return tool_ids, features, filter_ids, terminal_id
|
||||
if features:
|
||||
defaults['features'] = features
|
||||
return defaults
|
||||
|
||||
|
||||
async def _set_terminal_cwd(app, server_id: str, user, cwd: str, chat_id: str) -> None:
|
||||
|
|
@ -436,9 +299,8 @@ async def _execute_channel_automation(
|
|||
db,
|
||||
)
|
||||
|
||||
tool_ids, features, filter_ids, _ = await _resolve_model_defaults(app, model_id)
|
||||
|
||||
form_data = {
|
||||
**await _resolve_model_defaults(app, model_id),
|
||||
'model': model_id,
|
||||
'messages': [
|
||||
{
|
||||
|
|
@ -454,13 +316,6 @@ async def _execute_channel_automation(
|
|||
'automation_id': automation.id,
|
||||
'background_tasks': {},
|
||||
}
|
||||
if tool_ids:
|
||||
form_data['tool_ids'] = tool_ids
|
||||
if features:
|
||||
form_data['features'] = features
|
||||
if filter_ids:
|
||||
form_data['filter_ids'] = filter_ids
|
||||
|
||||
await app.state.CHAT_COMPLETION_HANDLER(request, form_data, user=user)
|
||||
|
||||
from open_webui.socket.main import sio
|
||||
|
|
@ -615,11 +470,9 @@ async def execute_automation(app, automation: AutomationModel) -> None:
|
|||
room=f'user:{automation.user_id}',
|
||||
)
|
||||
|
||||
# Resolve model defaults (frontend does this, backend doesn't)
|
||||
tool_ids, features, filter_ids, terminal_id = await _resolve_model_defaults(app, model_id)
|
||||
|
||||
# Build the same payload the frontend sends to /api/chat/completions
|
||||
form_data = {
|
||||
**await _resolve_model_defaults(app, model_id),
|
||||
'model': model_id,
|
||||
'messages': [{'role': 'user', 'content': prompt}],
|
||||
'stream': True,
|
||||
|
|
@ -636,15 +489,6 @@ async def execute_automation(app, automation: AutomationModel) -> None:
|
|||
'automation_id': automation.id,
|
||||
'background_tasks': {},
|
||||
}
|
||||
if tool_ids:
|
||||
form_data['tool_ids'] = tool_ids
|
||||
if features:
|
||||
form_data['features'] = features
|
||||
if filter_ids:
|
||||
form_data['filter_ids'] = filter_ids
|
||||
if terminal_id:
|
||||
form_data['terminal_id'] = terminal_id
|
||||
|
||||
# Call the full chat completion pipeline (same as POST /api/chat/completions).
|
||||
# The handler reference is stored on app.state to avoid circular imports.
|
||||
request = _build_request(app, token=token)
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ def expand_recurring_event(
|
|||
|
||||
original_start_ns = event_dict['start_at']
|
||||
original_start = to_local_datetime(original_start_ns)
|
||||
rule_str = '\n'.join(line for line in rrule_str.splitlines() if not line.upper().startswith('DTSTART')) or rrule_str
|
||||
rule_str = '\n'.join(part for part in rrule_str.split() if not part.upper().startswith('DTSTART')) or rrule_str
|
||||
|
||||
try:
|
||||
# Anchor to the event's real start so day-of-week / day-of-month are correct
|
||||
|
|
|
|||
|
|
@ -23,9 +23,9 @@ from open_webui.routers.pipelines import (
|
|||
process_pipeline_outlet_filter,
|
||||
)
|
||||
from open_webui.socket.main import (
|
||||
EVENT_QUEUES,
|
||||
get_event_call,
|
||||
get_event_emitter,
|
||||
sio,
|
||||
)
|
||||
from open_webui.utils.filter import (
|
||||
get_filter_functions,
|
||||
|
|
@ -71,39 +71,36 @@ async def generate_direct_chat_completion(
|
|||
logging.info('WebSocket channel: %s', channel)
|
||||
|
||||
if form_data.get('stream'):
|
||||
q = asyncio.Queue()
|
||||
|
||||
async def message_listener(sid, data):
|
||||
"""
|
||||
Handle received socket messages and push them into the queue.
|
||||
"""
|
||||
await q.put(data)
|
||||
|
||||
# Register the listener
|
||||
sio.on(channel, message_listener)
|
||||
queue = asyncio.Queue()
|
||||
EVENT_QUEUES[channel] = queue
|
||||
|
||||
# Start processing chat completion in background
|
||||
res = await event_caller(
|
||||
{
|
||||
'type': 'request:chat:completion',
|
||||
'data': {
|
||||
'form_data': form_data,
|
||||
'model': models[form_data['model']],
|
||||
'channel': channel,
|
||||
'session_id': session_id,
|
||||
},
|
||||
}
|
||||
)
|
||||
try:
|
||||
res = await event_caller(
|
||||
{
|
||||
'type': 'request:chat:completion',
|
||||
'data': {
|
||||
'form_data': form_data,
|
||||
'model': models[form_data['model']],
|
||||
'channel': channel,
|
||||
'session_id': session_id,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
log.info('res: %s', res)
|
||||
log.info('res: %s', res)
|
||||
|
||||
if res.get('status', False):
|
||||
status = res.get('status', False)
|
||||
except BaseException:
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
raise
|
||||
|
||||
if status:
|
||||
# Define a generator to stream responses
|
||||
async def event_generator():
|
||||
nonlocal q
|
||||
try:
|
||||
while True:
|
||||
data = await q.get() # Wait for new messages
|
||||
data = await queue.get() # Wait for new messages
|
||||
if isinstance(data, dict):
|
||||
if 'done' in data and data['done']:
|
||||
break # Stop streaming when 'done' is received
|
||||
|
|
@ -117,17 +114,17 @@ async def generate_direct_chat_completion(
|
|||
except Exception as e:
|
||||
log.debug('Error in event generator: %s', e)
|
||||
pass
|
||||
finally:
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
|
||||
# Define a background task to run the event generator
|
||||
async def background():
|
||||
try:
|
||||
del sio.handlers['/'][channel]
|
||||
except Exception as e:
|
||||
pass
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
|
||||
# Return the streaming response
|
||||
return StreamingResponse(event_generator(), media_type='text/event-stream', background=background)
|
||||
else:
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
raise Exception(str(res))
|
||||
else:
|
||||
res = await event_caller(
|
||||
|
|
@ -260,24 +257,25 @@ async def generate_chat_completion(
|
|||
bypass_filter=True,
|
||||
bypass_system_prompt=bypass_system_prompt,
|
||||
)
|
||||
# Upstream errors come back as a response object.
|
||||
if not isinstance(response, StreamingResponse):
|
||||
return response
|
||||
return StreamingResponse(
|
||||
stream_wrapper(response.body_iterator),
|
||||
media_type='text/event-stream',
|
||||
background=response.background,
|
||||
)
|
||||
else:
|
||||
return {
|
||||
**(
|
||||
await generate_chat_completion(
|
||||
request,
|
||||
form_data,
|
||||
user,
|
||||
bypass_filter=True,
|
||||
bypass_system_prompt=bypass_system_prompt,
|
||||
)
|
||||
),
|
||||
'selected_model_id': selected_model_id,
|
||||
}
|
||||
response = await generate_chat_completion(
|
||||
request,
|
||||
form_data,
|
||||
user,
|
||||
bypass_filter=True,
|
||||
bypass_system_prompt=bypass_system_prompt,
|
||||
)
|
||||
if not isinstance(response, dict):
|
||||
return response
|
||||
return {**response, 'selected_model_id': selected_model_id}
|
||||
|
||||
if model.get('pipe'):
|
||||
# Below does not require bypass_filter because this is the only route the uses this function and it is already bypassing the filter
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ from open_webui.storage.provider import Storage
|
|||
|
||||
BASE64_IMAGE_URL_PREFIX = re.compile(r'data:image/\w+;base64,', re.IGNORECASE)
|
||||
MARKDOWN_IMAGE_URL_PATTERN = re.compile(r'!\[(.*?)\]\((.+?)\)', re.IGNORECASE)
|
||||
FILE_CONTENT_URL_PATTERN = re.compile(r'^/api/v1/files/([^/?#]+)/content(?:[?#]|$)')
|
||||
|
||||
# Extension-based MIME fallback, only used when ENABLE_IMAGE_CONTENT_TYPE_EXTENSION_FALLBACK is True.
|
||||
_IMAGE_MIME_FALLBACK = {
|
||||
|
|
@ -74,9 +75,16 @@ async def get_image_base64_from_url(url: str, user=None) -> Optional[str]:
|
|||
# rebinding DNS answer that passed validate_url cannot reach an internal address.
|
||||
async with get_ssrf_safe_session() as session:
|
||||
async with session.get(
|
||||
url, ssl=AIOHTTP_CLIENT_SESSION_SSL, allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS
|
||||
url,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,
|
||||
headers={'Accept-Encoding': 'identity'},
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
# Accept-Encoding is only a request; the sender can still compress and pick our decompressed size.
|
||||
encodings = response.headers.getall('Content-Encoding', ())
|
||||
if any(encoding.lower() not in ('', 'identity') for encoding in encodings):
|
||||
return None
|
||||
image_data = bytearray()
|
||||
total = 0
|
||||
async for chunk in response.content.iter_chunked(64 * 1024):
|
||||
|
|
@ -90,7 +98,8 @@ async def get_image_base64_from_url(url: str, user=None) -> Optional[str]:
|
|||
else:
|
||||
# Non-URL string — treat as file_id. Delegate to the canonical
|
||||
# file-ID resolver which enforces ownership/access checks.
|
||||
return await get_image_base64_from_file_id(url, user=user)
|
||||
file_id_match = FILE_CONTENT_URL_PATTERN.match(url)
|
||||
return await get_image_base64_from_file_id(file_id_match.group(1) if file_id_match else url, user=user)
|
||||
|
||||
except Exception:
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
import logging
|
||||
import time
|
||||
from string import punctuation
|
||||
from typing import Any, Optional
|
||||
from urllib.parse import quote
|
||||
|
||||
import jwt
|
||||
from open_webui.env import (
|
||||
FORWARD_USER_INFO_HEADER_AUTH_TYPE,
|
||||
FORWARD_USER_INFO_HEADER_JWT,
|
||||
FORWARD_USER_INFO_HEADER_JWT_EXPIRES_SECONDS,
|
||||
FORWARD_USER_INFO_HEADER_JWT_SECRET,
|
||||
|
|
@ -47,15 +49,20 @@ def _mint_forward_user_jwt(user: Any) -> str:
|
|||
return jwt.encode(payload, FORWARD_USER_INFO_HEADER_JWT_SECRET, algorithm='HS256')
|
||||
|
||||
|
||||
def include_user_info_headers(headers: dict, user: Optional[Any] = None) -> dict:
|
||||
def include_user_info_headers(headers: dict, user: Optional[Any] = None, *, request=None) -> dict:
|
||||
"""
|
||||
Forward user identity to external backends: signed JWT in
|
||||
FORWARD_USER_INFO_HEADER_JWT if FORWARD_USER_INFO_HEADER_JWT_SECRET is set;
|
||||
otherwise the legacy X-OpenWebUI-User-* headers.
|
||||
Include the verified incoming auth type when a request provides it.
|
||||
"""
|
||||
if user is None:
|
||||
return headers
|
||||
|
||||
auth_type = getattr(getattr(request, 'state', None), 'auth_type', None)
|
||||
if auth_type in ('api_key', 'jwt'):
|
||||
headers = {**headers, FORWARD_USER_INFO_HEADER_AUTH_TYPE: auth_type}
|
||||
|
||||
if FORWARD_USER_INFO_HEADER_JWT_SECRET:
|
||||
try:
|
||||
token = _mint_forward_user_jwt(user)
|
||||
|
|
@ -141,6 +148,7 @@ def parse_custom_headers(
|
|||
'{{USER_GROUPS}}': ','.join(group.name.strip() for group in user_groups) if user_groups else '',
|
||||
'{{USER_GROUP_IDS}}': ','.join(group.id for group in user_groups) if user_groups else '',
|
||||
'{{USER_AGENT}}': user_agent,
|
||||
'{{AUTH_TYPE}}': getattr(getattr(request, 'state', None), 'auth_type', None) or '',
|
||||
}
|
||||
|
||||
parsed_headers = {}
|
||||
|
|
@ -149,6 +157,7 @@ def parse_custom_headers(
|
|||
value = str(value)
|
||||
for token, val in template_vars.items():
|
||||
value = value.replace(token, val)
|
||||
parsed_headers[key] = value
|
||||
# Encode Unicode and controls after substitution; preserve ASCII header syntax and existing escapes.
|
||||
parsed_headers[key] = quote(value, safe=punctuation + ' \t')
|
||||
|
||||
return parsed_headers
|
||||
|
|
|
|||
|
|
@ -89,8 +89,14 @@ class MCPClient:
|
|||
if not self.session:
|
||||
raise RuntimeError('MCP client is not connected.')
|
||||
|
||||
result = await self.session.list_tools()
|
||||
tools = result.tools
|
||||
tools = []
|
||||
cursor = None
|
||||
while True:
|
||||
result = await self.session.list_tools(cursor=cursor)
|
||||
tools.extend(result.tools)
|
||||
cursor = result.nextCursor
|
||||
if cursor is None:
|
||||
break
|
||||
|
||||
tool_specs = []
|
||||
for tool in tools:
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1223,6 +1223,12 @@ def strict_match_mime_type(supported: list[str] | str, header: str) -> str | Non
|
|||
return None
|
||||
|
||||
|
||||
def is_raster_image_content_type(content_type: str | None) -> bool:
|
||||
"""Return True if the content type is an image that decodes as a bitmap; SVG is XML."""
|
||||
base_content_type = (content_type or '').split(';')[0].strip().lower()
|
||||
return base_content_type.startswith('image/') and base_content_type != 'image/svg+xml'
|
||||
|
||||
|
||||
def extract_urls(text: str) -> list[str]:
|
||||
# Regex pattern to match URLs
|
||||
url_pattern = re.compile(r'(https?://[^\s]+)', re.IGNORECASE) # Matches http and https URLs
|
||||
|
|
|
|||
|
|
@ -438,7 +438,14 @@ async def get_all_models(request, refresh: bool = False, user: UserModel = None)
|
|||
|
||||
log.debug('get_all_models() returned %s models', len(models))
|
||||
|
||||
models_dict = {model['id']: model for model in models}
|
||||
models_dict = {}
|
||||
for model in models:
|
||||
model = model.copy()
|
||||
if model.get('ollama'):
|
||||
# Keep the moving expiry in the API response, outside the registry signature.
|
||||
model['ollama'] = model['ollama'].copy()
|
||||
model['ollama'].pop('expires_at', None)
|
||||
models_dict[model['id']] = model
|
||||
if isinstance(request.app.state.MODELS, RedisDict):
|
||||
try:
|
||||
request.app.state.MODELS.set(models_dict)
|
||||
|
|
@ -461,12 +468,22 @@ async def check_model_access(user, model, model_info=None, db=None):
|
|||
access_grants=access_grants,
|
||||
db=db,
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=arena_read_denied',
|
||||
user.id,
|
||||
model.get('id'),
|
||||
)
|
||||
raise Exception('Model not found')
|
||||
else:
|
||||
# Callers that already fetched the row (chat completion entry) pass it in
|
||||
if model_info is None or model_info.id != model.get('id'):
|
||||
model_info = await Models.get_model_by_id(model.get('id'), db=db)
|
||||
if not model_info:
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=model_unregistered',
|
||||
user.id,
|
||||
model.get('id'),
|
||||
)
|
||||
raise Exception('Model not found')
|
||||
|
||||
# One group-membership fetch shared by the direct check and every
|
||||
|
|
@ -486,6 +503,11 @@ async def check_model_access(user, model, model_info=None, db=None):
|
|||
db=db,
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=model_read_denied',
|
||||
user.id,
|
||||
model_info.id,
|
||||
)
|
||||
raise Exception('Model not found')
|
||||
|
||||
# Enforce access on chained base models
|
||||
|
|
|
|||
|
|
@ -46,8 +46,8 @@ def _normalize_target(target: dict[str, Any], existing: dict[str, Any] | None =
|
|||
|
||||
target_id = str(target.get('id') or existing.get('id') or '').strip()
|
||||
if not target_id:
|
||||
hostname = urlparse(url).hostname or 'webhook'
|
||||
target_id = re.sub(r'[^a-zA-Z0-9_-]+', '-', hostname).strip('-').lower() or 'target'
|
||||
target_id = urlparse(url).hostname or 'webhook'
|
||||
target_id = re.sub(r'[^a-zA-Z0-9_-]+', '-', target_id).strip('-').lower() or 'target'
|
||||
|
||||
events = target['events'] if 'events' in target else existing.get('events', [])
|
||||
if events is None:
|
||||
|
|
|
|||
|
|
@ -91,7 +91,7 @@ from open_webui.utils.auth import (
|
|||
)
|
||||
from open_webui.utils.groups import apply_default_group_assignment
|
||||
from open_webui.utils.misc import parse_duration
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from starlette.responses import RedirectResponse
|
||||
|
||||
# Some IdPs put private params in ID token JOSE headers (CAS: client_id, CyberArk: app_id).
|
||||
|
|
@ -188,6 +188,16 @@ def _default_value(value):
|
|||
return getattr(value, 'value', value)
|
||||
|
||||
|
||||
def _get_roles_claim(claims: dict, claim: str) -> list | str | int | None:
|
||||
"""Read nested or flat claims, preserving explicit empty values and zero."""
|
||||
value = claims
|
||||
for key in claim.split('.'):
|
||||
value = value.get(key) if isinstance(value, dict) else None
|
||||
if not isinstance(value, (list, str, int)):
|
||||
value = claims.get(claim)
|
||||
return value if isinstance(value, (list, str, int)) else None
|
||||
|
||||
|
||||
async def get_oauth_runtime_config() -> SimpleNamespace:
|
||||
keys = [key for key, _default in OAUTH_RUNTIME_CONFIG.values()]
|
||||
stored = await Config.get_many(*keys)
|
||||
|
|
@ -202,7 +212,7 @@ NON_EXPIRING_TOKEN_EXPIRES_AT = 253402300799 # 9999-12-31 23:59:59 UTC
|
|||
|
||||
|
||||
def _normalize_token_expiry(token: dict) -> dict:
|
||||
"""Ensure a token dict always has a numeric ``expires_at``.
|
||||
"""Ensure a token dict always has a numeric access-token ``expires_at``.
|
||||
|
||||
Resolution order:
|
||||
1. If *expires_at* is already present and non-None, trust it.
|
||||
|
|
@ -233,16 +243,6 @@ def _normalize_token_expiry(token: dict) -> dict:
|
|||
)
|
||||
expires_at = NON_EXPIRING_TOKEN_EXPIRES_AT
|
||||
|
||||
id_token = token.get('id_token')
|
||||
if id_token:
|
||||
# Cap at the id_token expiry so pipes and tools never receive an expired JWT
|
||||
try:
|
||||
exp = jwt.decode(id_token, options={'verify_signature': False}).get('exp')
|
||||
if exp is not None:
|
||||
expires_at = min(expires_at, int(exp))
|
||||
except Exception as e:
|
||||
log.debug('Could not read exp from id_token: %s', e)
|
||||
|
||||
token['expires_at'] = expires_at
|
||||
return token
|
||||
|
||||
|
|
@ -1263,7 +1263,7 @@ class OAuthClientManager:
|
|||
if token and not token.get('access_token'):
|
||||
error_desc = token.get('error_description', token.get('error', 'Unknown error'))
|
||||
error_message = f'Token exchange failed: {error_desc}'
|
||||
log.error(f'Invalid token response for client_id {client_id}: {token}')
|
||||
log.error('Invalid token response for client_id %s: %s', client_id, error_desc)
|
||||
token = None
|
||||
|
||||
if token:
|
||||
|
|
@ -1372,10 +1372,21 @@ class OAuthManager:
|
|||
)
|
||||
return None
|
||||
|
||||
# SSO integrations may consume the ID token as well as the access token.
|
||||
expires_at = session.expires_at
|
||||
id_token = session.token.get('id_token')
|
||||
if id_token and expires_at is not None:
|
||||
try:
|
||||
exp = jwt.decode(id_token, options={'verify_signature': False}).get('exp')
|
||||
if exp is not None:
|
||||
expires_at = min(expires_at, int(exp))
|
||||
except Exception as e:
|
||||
log.debug('Could not read exp from id_token: %s', e)
|
||||
|
||||
if (
|
||||
force_refresh
|
||||
or session.expires_at is None
|
||||
or datetime.now() + timedelta(minutes=5) >= datetime.fromtimestamp(session.expires_at)
|
||||
or expires_at is None
|
||||
or datetime.now() + timedelta(minutes=5) >= datetime.fromtimestamp(expires_at)
|
||||
):
|
||||
log.debug('Token refresh needed for user %s, provider %s', user_id, session.provider)
|
||||
refreshed_token = await self._refresh_token(session)
|
||||
|
|
@ -1504,7 +1515,7 @@ class OAuthManager:
|
|||
log.error(f'Exception during token refresh for provider {provider}: {e}')
|
||||
return None
|
||||
|
||||
async def get_user_role(self, user, user_data):
|
||||
async def get_user_role(self, user, user_data, *, access_token: str | None = None):
|
||||
auth_config = await get_oauth_runtime_config()
|
||||
user_count = await Users.get_num_users()
|
||||
if user and user_count == 1:
|
||||
|
|
@ -1528,18 +1539,15 @@ class OAuthManager:
|
|||
# Keep existing users at their current role unless the provider sent roles.
|
||||
role = user.role if user else auth_config.DEFAULT_USER_ROLE
|
||||
|
||||
# Next block extracts the roles from the user data, accepting nested claims of any depth
|
||||
if oauth_claim and oauth_allowed_roles and oauth_admin_roles:
|
||||
claim_data = user_data
|
||||
nested_claims = oauth_claim.split('.')
|
||||
for nested_claim in nested_claims:
|
||||
claim_data = claim_data.get(nested_claim, {})
|
||||
|
||||
# Try flat claim structure as alternative
|
||||
if not claim_data:
|
||||
claim_data = user_data.get(oauth_claim, {})
|
||||
|
||||
oauth_roles = []
|
||||
if oauth_claim:
|
||||
claim_data = _get_roles_claim(user_data, oauth_claim)
|
||||
if claim_data is None and access_token is not None:
|
||||
# The exchange endpoint has already validated this token with the provider's userinfo endpoint.
|
||||
try:
|
||||
token_claims = jwt.decode(access_token, options={'verify_signature': False})
|
||||
claim_data = _get_roles_claim(token_claims, oauth_claim)
|
||||
except jwt.PyJWTError as e:
|
||||
log.debug('Token exchange: cannot decode token claims: %s', e)
|
||||
|
||||
if isinstance(claim_data, list):
|
||||
oauth_roles = claim_data
|
||||
|
|
@ -1552,6 +1560,10 @@ class OAuthManager:
|
|||
elif isinstance(claim_data, int):
|
||||
oauth_roles = [str(claim_data)]
|
||||
|
||||
if access_token is not None and not oauth_roles and oauth_allowed_roles and '*' not in oauth_allowed_roles:
|
||||
log.warning('Token exchange denied: no readable roles claim in userinfo or the token')
|
||||
raise HTTPException(status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
log.debug('Oauth Roles claim: %s', oauth_claim)
|
||||
log.debug('User roles from oauth: %s', oauth_roles)
|
||||
log.debug('Accepted user roles: %s', oauth_allowed_roles)
|
||||
|
|
@ -1598,9 +1610,10 @@ class OAuthManager:
|
|||
user_data,
|
||||
provider,
|
||||
*,
|
||||
access_token: str | None = None,
|
||||
db=None,
|
||||
):
|
||||
determined_role = await self.get_user_role(user, user_data)
|
||||
determined_role = await self.get_user_role(user, user_data, access_token=access_token)
|
||||
if user.role == determined_role:
|
||||
return user
|
||||
|
||||
|
|
@ -1811,7 +1824,7 @@ class OAuthManager:
|
|||
picture = await resp.read()
|
||||
base64_encoded_picture = base64.b64encode(picture).decode('utf-8')
|
||||
try:
|
||||
return validate_profile_image_url(f'data:{upstream_mime};base64,{base64_encoded_picture}')
|
||||
return validate_image_url(f'data:{upstream_mime};base64,{base64_encoded_picture}')
|
||||
except ValueError:
|
||||
log.warning(
|
||||
f'Rejected OAuth profile picture from {picture_url}: '
|
||||
|
|
@ -1917,7 +1930,7 @@ class OAuthManager:
|
|||
if provider == 'feishu' and isinstance(user_data, dict) and 'data' in user_data:
|
||||
user_data = user_data['data']
|
||||
if not user_data:
|
||||
log.warning(f'OAuth callback failed, user data is missing: {token}')
|
||||
log.warning('OAuth callback failed for provider %s, user data is missing', provider)
|
||||
raise HTTPException(400, detail=ERROR_MESSAGES.INVALID_CRED)
|
||||
|
||||
# Extract the "sub" claim, using custom claim if configured
|
||||
|
|
|
|||
|
|
@ -1,140 +0,0 @@
|
|||
import site
|
||||
from datetime import datetime
|
||||
from html import escape
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from fpdf import FPDF
|
||||
from markdown import markdown
|
||||
from open_webui.env import FONTS_DIR, STATIC_DIR
|
||||
from open_webui.models.chats import ChatTitleMessagesForm
|
||||
|
||||
|
||||
class PDFGenerator:
|
||||
"""
|
||||
Description:
|
||||
The `PDFGenerator` class is designed to create PDF documents from chat messages.
|
||||
The process involves transforming markdown content into HTML and then into a PDF format
|
||||
|
||||
Attributes:
|
||||
- `form_data`: An instance of `ChatTitleMessagesForm` containing title and messages.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, form_data: ChatTitleMessagesForm):
|
||||
self.html_body = None
|
||||
self.messages_html = None
|
||||
self.form_data = form_data
|
||||
|
||||
self.css = Path(STATIC_DIR / 'assets' / 'pdf-style.css').read_text()
|
||||
|
||||
def format_timestamp(self, timestamp: float) -> str:
|
||||
"""Convert a UNIX timestamp to a formatted date string."""
|
||||
try:
|
||||
date_time = datetime.fromtimestamp(timestamp)
|
||||
return date_time.strftime('%Y-%m-%d, %H:%M:%S')
|
||||
except (ValueError, TypeError) as e:
|
||||
# Log the error if necessary
|
||||
return ''
|
||||
|
||||
def _build_html_message(self, message: Dict[str, Any]) -> str:
|
||||
"""Build HTML for a single message."""
|
||||
role = escape(message.get('role', 'user'))
|
||||
content = escape(message.get('content', ''))
|
||||
timestamp = message.get('timestamp')
|
||||
|
||||
model = escape(message.get('model') if role == 'assistant' else '')
|
||||
|
||||
date_str = escape(self.format_timestamp(timestamp) if timestamp else '')
|
||||
|
||||
# extends pymdownx extension to convert markdown to html.
|
||||
# - https://facelessuser.github.io/pymdown-extensions/usage_notes/
|
||||
# html_content = markdown(content, extensions=["pymdownx.extra"])
|
||||
|
||||
content = content.replace('\n', '<br/>')
|
||||
html_message = f"""
|
||||
<div>
|
||||
<div>
|
||||
<h4>
|
||||
<strong>{role.title()}</strong>
|
||||
<span style="font-size: 12px;">{model}</span>
|
||||
</h4>
|
||||
<div> {date_str} </div>
|
||||
</div>
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
<div>
|
||||
{content}
|
||||
</div>
|
||||
</div>
|
||||
<br/>
|
||||
"""
|
||||
return html_message
|
||||
|
||||
def _generate_html_body(self) -> str:
|
||||
"""Generate the full HTML body for the PDF."""
|
||||
escaped_title = escape(self.form_data.title)
|
||||
return f"""
|
||||
<html>
|
||||
<head>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
</head>
|
||||
<body>
|
||||
<div>
|
||||
<div>
|
||||
<h2>{escaped_title}</h2>
|
||||
{self.messages_html}
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
def generate_chat_pdf(self) -> bytes:
|
||||
"""
|
||||
Generate a PDF from chat messages.
|
||||
"""
|
||||
try:
|
||||
global FONTS_DIR
|
||||
|
||||
pdf = FPDF()
|
||||
pdf.add_page()
|
||||
|
||||
# When running using `pip install` the static directory is in the site packages.
|
||||
if not FONTS_DIR.exists():
|
||||
FONTS_DIR = Path(site.getsitepackages()[0]) / 'static/fonts'
|
||||
# When running using `pip install -e .` the static directory is in the site packages.
|
||||
# This path only works if `open-webui serve` is run from the root of this project.
|
||||
if not FONTS_DIR.exists():
|
||||
FONTS_DIR = Path('.') / 'backend' / 'static' / 'fonts'
|
||||
|
||||
pdf.add_font('NotoSans', '', f'{FONTS_DIR}/NotoSans-Regular.ttf')
|
||||
pdf.add_font('NotoSans', 'b', f'{FONTS_DIR}/NotoSans-Bold.ttf')
|
||||
pdf.add_font('NotoSans', 'i', f'{FONTS_DIR}/NotoSans-Italic.ttf')
|
||||
pdf.add_font('NotoSansKR', '', f'{FONTS_DIR}/NotoSansKR-Regular.ttf')
|
||||
pdf.add_font('NotoSansJP', '', f'{FONTS_DIR}/NotoSansJP-Regular.ttf')
|
||||
pdf.add_font('NotoSansSC', '', f'{FONTS_DIR}/NotoSansSC-Regular.ttf')
|
||||
pdf.add_font('Twemoji', '', f'{FONTS_DIR}/Twemoji.ttf')
|
||||
|
||||
pdf.set_font('NotoSans', size=12)
|
||||
pdf.set_fallback_fonts(['NotoSansKR', 'NotoSansJP', 'NotoSansSC', 'Twemoji'])
|
||||
|
||||
pdf.set_auto_page_break(auto=True, margin=15)
|
||||
|
||||
# Build HTML messages
|
||||
messages_html_list: List[str] = [self._build_html_message(msg) for msg in self.form_data.messages]
|
||||
self.messages_html = '<div>' + ''.join(messages_html_list) + '</div>'
|
||||
|
||||
# Generate full HTML body
|
||||
self.html_body = self._generate_html_body()
|
||||
|
||||
pdf.write_html(self.html_body)
|
||||
|
||||
# Save the pdf with name .pdf
|
||||
pdf_bytes = pdf.output()
|
||||
|
||||
return bytes(pdf_bytes)
|
||||
except Exception as e:
|
||||
raise e
|
||||
|
|
@ -1,7 +1,8 @@
|
|||
import time
|
||||
from typing import Dict, Optional
|
||||
from typing import Optional
|
||||
|
||||
from open_webui.env import REDIS_KEY_PREFIX
|
||||
from redis.asyncio import Redis
|
||||
|
||||
|
||||
class RateLimiter:
|
||||
|
|
@ -10,30 +11,26 @@ class RateLimiter:
|
|||
Falls back to in-memory storage if Redis is not available.
|
||||
"""
|
||||
|
||||
# In-memory fallback storage
|
||||
_memory_store: Dict[str, Dict[int, int]] = {}
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
redis_client,
|
||||
limit: int,
|
||||
window: int,
|
||||
bucket_size: int = 60,
|
||||
enabled: bool = True,
|
||||
):
|
||||
"""
|
||||
:param redis_client: Redis client instance or None
|
||||
:param limit: Max allowed events in the window
|
||||
:param window: Time window in seconds
|
||||
:param bucket_size: Bucket resolution
|
||||
:param enabled: Turn on/off rate limiting globally
|
||||
"""
|
||||
self.r = redis_client
|
||||
self.limit = limit
|
||||
self.window = window
|
||||
self.bucket_size = bucket_size
|
||||
self.num_buckets = window // bucket_size
|
||||
self.enabled = enabled
|
||||
# bucket index -> rate-limit key -> hits
|
||||
self._memory_store: dict[int, dict[str, int]] = {}
|
||||
|
||||
def _bucket_key(self, key: str, bucket_index: int) -> str:
|
||||
return f'{REDIS_KEY_PREFIX}:ratelimit:{key.lower()}:{bucket_index}'
|
||||
|
|
@ -41,10 +38,13 @@ class RateLimiter:
|
|||
def _current_bucket(self) -> int:
|
||||
return int(time.time()) // self.bucket_size
|
||||
|
||||
def _redis_available(self) -> bool:
|
||||
return self.r is not None
|
||||
def _prune_memory_store(self, now_bucket: int) -> None:
|
||||
min_bucket = now_bucket - self.num_buckets
|
||||
expired = [bucket_index for bucket_index in self._memory_store if bucket_index < min_bucket]
|
||||
for bucket_index in expired:
|
||||
del self._memory_store[bucket_index]
|
||||
|
||||
def is_limited(self, key: str) -> bool:
|
||||
async def is_limited(self, redis: Redis | None, key: str) -> bool:
|
||||
"""
|
||||
Main rate-limit check.
|
||||
Gracefully handles missing or failing Redis.
|
||||
|
|
@ -52,85 +52,63 @@ class RateLimiter:
|
|||
if not self.enabled:
|
||||
return False
|
||||
|
||||
if self._redis_available():
|
||||
if redis is not None:
|
||||
try:
|
||||
return self._is_limited_redis(key)
|
||||
return await self._is_limited_redis(redis, key)
|
||||
except Exception:
|
||||
return self._is_limited_memory(key)
|
||||
else:
|
||||
return self._is_limited_memory(key)
|
||||
|
||||
def get_count(self, key: str) -> int:
|
||||
async def get_count(self, redis: Redis | None, key: str) -> int:
|
||||
if not self.enabled:
|
||||
return 0
|
||||
|
||||
if self._redis_available():
|
||||
if redis is not None:
|
||||
try:
|
||||
return self._get_count_redis(key)
|
||||
return await self._get_count_redis(redis, key)
|
||||
except Exception:
|
||||
return self._get_count_memory(key)
|
||||
else:
|
||||
return self._get_count_memory(key)
|
||||
|
||||
def remaining(self, key: str) -> int:
|
||||
used = self.get_count(key)
|
||||
async def remaining(self, redis: Redis | None, key: str) -> int:
|
||||
used = await self.get_count(redis, key)
|
||||
return max(0, self.limit - used)
|
||||
|
||||
def _is_limited_redis(self, key: str) -> bool:
|
||||
async def _is_limited_redis(self, redis: Redis, key: str) -> bool:
|
||||
now_bucket = self._current_bucket()
|
||||
bucket_key = self._bucket_key(key, now_bucket)
|
||||
|
||||
attempts = self.r.incr(bucket_key)
|
||||
attempts = await redis.incr(bucket_key)
|
||||
if attempts == 1:
|
||||
self.r.expire(bucket_key, self.window + self.bucket_size)
|
||||
await redis.expire(bucket_key, self.window + self.bucket_size)
|
||||
|
||||
# Collect buckets
|
||||
buckets = [self._bucket_key(key, now_bucket - i) for i in range(self.num_buckets + 1)]
|
||||
|
||||
counts = self.r.mget(buckets)
|
||||
counts = await redis.mget(buckets)
|
||||
total = sum(int(c) for c in counts if c)
|
||||
|
||||
return total > self.limit
|
||||
|
||||
def _get_count_redis(self, key: str) -> int:
|
||||
async def _get_count_redis(self, redis: Redis, key: str) -> int:
|
||||
now_bucket = self._current_bucket()
|
||||
buckets = [self._bucket_key(key, now_bucket - i) for i in range(self.num_buckets + 1)]
|
||||
counts = self.r.mget(buckets)
|
||||
counts = await redis.mget(buckets)
|
||||
return sum(int(c) for c in counts if c)
|
||||
|
||||
def _is_limited_memory(self, key: str) -> bool:
|
||||
now_bucket = self._current_bucket()
|
||||
self._prune_memory_store(now_bucket)
|
||||
|
||||
# Init storage
|
||||
if key not in self._memory_store:
|
||||
self._memory_store[key] = {}
|
||||
current_bucket_counts = self._memory_store.setdefault(now_bucket, {})
|
||||
current_bucket_counts[key] = current_bucket_counts.get(key, 0) + 1
|
||||
|
||||
store = self._memory_store[key]
|
||||
|
||||
# Increment bucket
|
||||
store[now_bucket] = store.get(now_bucket, 0) + 1
|
||||
|
||||
# Drop expired buckets
|
||||
min_bucket = now_bucket - self.num_buckets
|
||||
expired = [b for b in store if b < min_bucket]
|
||||
for b in expired:
|
||||
del store[b]
|
||||
|
||||
# Count totals
|
||||
total = sum(store.values())
|
||||
total = sum(bucket_counts.get(key, 0) for bucket_counts in self._memory_store.values())
|
||||
return total > self.limit
|
||||
|
||||
def _get_count_memory(self, key: str) -> int:
|
||||
now_bucket = self._current_bucket()
|
||||
if key not in self._memory_store:
|
||||
return 0
|
||||
|
||||
store = self._memory_store[key]
|
||||
min_bucket = now_bucket - self.num_buckets
|
||||
|
||||
# Remove expired
|
||||
expired = [b for b in store if b < min_bucket]
|
||||
for b in expired:
|
||||
del store[b]
|
||||
|
||||
return sum(store.values())
|
||||
self._prune_memory_store(now_bucket)
|
||||
return sum(bucket_counts.get(key, 0) for bucket_counts in self._memory_store.values())
|
||||
|
|
|
|||
171
backend/open_webui/utils/recurrence.py
Normal file
171
backend/open_webui/utils/recurrence.py
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
"""Recurrence calculations isolated from application/DB imports for worker processes."""
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from anyio import fail_after, to_process
|
||||
from dateutil.rrule import HOURLY, MINUTELY, SECONDLY, rruleset, rrulestr
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
RRULE_TIMEOUT_SECONDS = 2
|
||||
|
||||
|
||||
class RecurrenceEvaluationTimeout(ValueError):
|
||||
"""The evaluation budget expired; the schedule may still have occurrences."""
|
||||
|
||||
|
||||
def _resolve_tz(tz: str = None) -> Optional[ZoneInfo]:
|
||||
"""Safely resolve a timezone string to ZoneInfo.
|
||||
|
||||
Returns None (→ server-local fallback) when *tz* is empty, None,
|
||||
or an unrecognised IANA key. Logs a warning on bad keys so
|
||||
misconfiguration is visible in the server logs.
|
||||
"""
|
||||
if not tz:
|
||||
return None
|
||||
try:
|
||||
return ZoneInfo(tz)
|
||||
except (KeyError, Exception):
|
||||
log.warning('Unknown timezone %r — falling back to server time', tz)
|
||||
return None
|
||||
|
||||
|
||||
def _parse_rule(s: str, now: Optional[datetime] = None):
|
||||
"""Parse RRULE with clock-aligned DTSTART for sub-daily frequencies.
|
||||
|
||||
SECONDLY/MINUTELY/HOURLY rules use a fixed epoch DTSTART (2000-01-01 00:00)
|
||||
so intervals snap to clock boundaries (e.g. every 5min = :00, :05, :10).
|
||||
"""
|
||||
upper = s.upper()
|
||||
if 'EXRULE' in upper:
|
||||
raise ValueError('EXRULE is not supported in recurrence rules')
|
||||
|
||||
parsed = rrulestr(s, ignoretz=True)
|
||||
rules = parsed._rrule if isinstance(parsed, rruleset) else [parsed]
|
||||
if len(rules) > 1:
|
||||
raise ValueError('only one RRULE is supported per recurrence rule')
|
||||
|
||||
rule = rules[0]
|
||||
start = rule._dtstart.replace(tzinfo=None)
|
||||
anchor = now or datetime.now()
|
||||
parts = s.split()
|
||||
stripped = '\n'.join(part for part in parts if not part.upper().startswith('DTSTART')) or s
|
||||
has_dtstart = any(part.upper().startswith('DTSTART') for part in parts)
|
||||
step = {
|
||||
SECONDLY: timedelta(seconds=rule._interval),
|
||||
MINUTELY: timedelta(minutes=rule._interval),
|
||||
HOURLY: timedelta(hours=rule._interval),
|
||||
}.get(rule._freq)
|
||||
|
||||
if step is None:
|
||||
if not rule._dtstart.tzinfo:
|
||||
return parsed
|
||||
return rrulestr(stripped, dtstart=start, ignoretz=True)
|
||||
|
||||
if rule._interval < 1:
|
||||
raise ValueError('RRULE INTERVAL must be a positive integer')
|
||||
dtstart = None
|
||||
if has_dtstart:
|
||||
emitted = ((anchor - start) // step) if anchor > start else 0
|
||||
emitted *= len(rule._byminute or (0,)) * len(rule._bysecond or (0,))
|
||||
if emitted <= 100_000:
|
||||
if rule._dtstart.tzinfo:
|
||||
dtstart = start
|
||||
else:
|
||||
return parsed
|
||||
if not has_dtstart or dtstart is None:
|
||||
epoch = datetime(2000, 1, 1)
|
||||
dtstart = epoch + ((anchor - epoch) // step) * step
|
||||
|
||||
return rrulestr(stripped, dtstart=dtstart, ignoretz=True)
|
||||
|
||||
|
||||
def _next_occurrences(s: str, now: datetime, n: int) -> list[datetime]:
|
||||
rule = _parse_rule(s, now)
|
||||
occurrences = []
|
||||
for _ in range(n):
|
||||
now = rule.after(now)
|
||||
if now is None:
|
||||
break
|
||||
occurrences.append(now)
|
||||
return occurrences
|
||||
|
||||
|
||||
async def _get_next_occurrences(s: str, now: datetime, n: int) -> list[datetime]:
|
||||
# A result-count or date limit cannot bound work before the first match.
|
||||
try:
|
||||
with fail_after(RRULE_TIMEOUT_SECONDS):
|
||||
return await to_process.run_sync(_next_occurrences, s, now, n, cancellable=True)
|
||||
except TimeoutError as e:
|
||||
raise RecurrenceEvaluationTimeout('Schedule took too long to evaluate; simplify its recurrence rule.') from e
|
||||
|
||||
|
||||
async def validate_rrule(s: str, tz: str = None) -> None:
|
||||
"""Raise ValueError if the RRULE is malformed or exhausted.
|
||||
|
||||
When *tz* is provided the "now" reference uses the user's local
|
||||
clock so that near-future schedules are not incorrectly rejected
|
||||
on servers whose system clock is ahead (e.g. UTC vs US timezones).
|
||||
"""
|
||||
upper = s.upper()
|
||||
if 'COUNT=' in upper and 'DTSTART' not in upper:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_COUNT_REQUIRES_DTSTART)
|
||||
zi = _resolve_tz(tz)
|
||||
now = datetime.now(zi).replace(tzinfo=None) if zi else datetime.now()
|
||||
try:
|
||||
occurrences = await _get_next_occurrences(s, now, 1)
|
||||
except RecurrenceEvaluationTimeout:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if not occurrences:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_NO_FUTURE_RUNS)
|
||||
|
||||
|
||||
async def next_run_ns(s: str, tz: str = None) -> Optional[int]:
|
||||
"""Next occurrence as epoch nanoseconds, respecting user timezone."""
|
||||
zi = _resolve_tz(tz)
|
||||
now = datetime.now(zi) if zi else datetime.now()
|
||||
now_naive = now.replace(tzinfo=None)
|
||||
occurrences = await _get_next_occurrences(s, now_naive, 1)
|
||||
if not occurrences:
|
||||
return None
|
||||
dt = occurrences[0]
|
||||
if zi:
|
||||
dt = dt.replace(tzinfo=zi)
|
||||
return int(dt.timestamp() * 1_000_000_000)
|
||||
|
||||
|
||||
async def next_n_runs_ns(s: str, n: int = 5, tz: str = None) -> list[int]:
|
||||
"""Compute next N occurrences for UI preview.
|
||||
|
||||
Uses the user's timezone for the starting "now" so that the
|
||||
preview matches the user's local clock (same as next_run_ns).
|
||||
"""
|
||||
zi = _resolve_tz(tz)
|
||||
result = []
|
||||
now = datetime.now(zi).replace(tzinfo=None) if zi else datetime.now()
|
||||
for dt in await _get_next_occurrences(s, now, n):
|
||||
if zi:
|
||||
dt_tz = dt.replace(tzinfo=zi)
|
||||
result.append(int(dt_tz.timestamp() * 1_000_000_000))
|
||||
else:
|
||||
result.append(int(dt.timestamp() * 1_000_000_000))
|
||||
return result
|
||||
|
||||
|
||||
async def rrule_interval_seconds(s: str) -> Optional[int]:
|
||||
"""Approximate interval between recurrences in seconds.
|
||||
|
||||
Returns None for one-shot (COUNT=1) schedules or rules
|
||||
with fewer than two future occurrences.
|
||||
"""
|
||||
s = '\n'.join(part for part in s.split() if not part.upper().startswith('DTSTART')) or s
|
||||
now = datetime.now()
|
||||
occurrences = await _get_next_occurrences(s, now, 2)
|
||||
if len(occurrences) < 2:
|
||||
return None
|
||||
return int((occurrences[1] - occurrences[0]).total_seconds())
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue