mirror of
https://github.com/open-webui/open-webui.git
synced 2026-10-08 03:08:02 +00:00
Merge pull request #29960 from open-webui/dev
Some checks failed
Python CI / Ruff Format (3.12) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Frontend Build / Format & Build (push) Has been cancelled
Frontend Build / Unit Tests (push) Has been cancelled
Python CI / Ruff Format (3.11) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Release to PyPI / release (push) Has been cancelled
Release / publish (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda, cuda) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / notify-helm-charts (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (, main) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda126, cuda126) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-ollama, ollama) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-slim, slim) (push) Has been cancelled
Some checks failed
Python CI / Ruff Format (3.12) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Frontend Build / Format & Build (push) Has been cancelled
Frontend Build / Unit Tests (push) Has been cancelled
Python CI / Ruff Format (3.11) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true
USE_CUDA_VER=cu126
free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Release to PyPI / release (push) Has been cancelled
Release / publish (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda, cuda) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / notify-helm-charts (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (, main) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda126, cuda126) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-ollama, ollama) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-slim, slim) (push) Has been cancelled
0.11.4
This commit is contained in:
Tim Baek
GitHub
commit
8bd8b4fac5
423 changed files with 116747 additions and 84008 deletions
|
|
@ -18,3 +18,6 @@ uploads
|
|||
**/*.db
|
||||
_test
|
||||
backend/data/*
|
||||
|
||||
.venv
|
||||
.git
|
||||
|
|
|
|||
|
|
@ -22,6 +22,9 @@ ENABLE_RAG_CSV_SUMMARY=false
|
|||
# Set to true to preserve backing file records, storage blobs, and per-file vectors when files are removed from knowledge bases.
|
||||
ENABLE_KNOWLEDGE_FILE_RETENTION=false
|
||||
|
||||
# Comma-separated chunk metadata keys to expose to the model alongside retrieved content.
|
||||
RAG_SOURCE_METADATA_KEYS=''
|
||||
|
||||
# Set to false to disable workspace Tools and Functions.
|
||||
ENABLE_PLUGINS=true
|
||||
|
||||
|
|
|
|||
6
.github/ISSUE_TEMPLATE/bug_report.yaml
vendored
6
.github/ISSUE_TEMPLATE/bug_report.yaml
vendored
|
|
@ -12,6 +12,8 @@ body:
|
|||
|
||||
Before submitting, search open and closed [Issues](https://github.com/open-webui/open-webui/issues) and [Discussions](https://github.com/open-webui/open-webui/discussions). The issue may already be reported or fixed on `dev`.
|
||||
|
||||
**Test on the latest release AND on `dev`, right before you submit this report, not last week.** A huge share of reports are for bugs already fixed on `dev`, sometimes weeks earlier, because the reporter only tested an old version and never rechecked. Reports that don't reproduce on latest and on current `dev` at submission time will be closed without further discussion, no exceptions.
|
||||
|
||||
Please do not open a code pull request for this report unless a maintainer asks for one, or the change is only i18n/localization. If you want to share code as reference, include it here as a local diff or patch. Actionable reproduction details are the most useful next step.
|
||||
|
||||
Security vulnerabilities must not be reported publicly. Use the [GitHub security page](https://github.com/open-webui/open-webui/security) instead.
|
||||
|
|
@ -23,12 +25,10 @@ body:
|
|||
options:
|
||||
- label: I searched open and closed issues and discussions for an existing report.
|
||||
required: true
|
||||
- label: I checked whether this is already fixed on the `dev` branch or latest source.
|
||||
- label: I reproduced this bug on the latest release AND on the current `dev` branch, right before submitting this report. I did not just check an old version or rely on a check from days ago.
|
||||
required: true
|
||||
- label: I understand that maintainers want a well-written issue before any code pull request.
|
||||
required: true
|
||||
- label: I am using the latest available version of Open WebUI for my install method.
|
||||
required: true
|
||||
- label: This is not a security vulnerability.
|
||||
required: true
|
||||
|
||||
|
|
|
|||
25
.github/pull_request_template.md
vendored
25
.github/pull_request_template.md
vendored
|
|
@ -1,31 +1,30 @@
|
|||
<!--
|
||||
Important checks for contributors:
|
||||
1. Target the `dev` branch. PRs targeting `main` will be closed.
|
||||
2. Code pull requests are not the default contribution path.
|
||||
3. Do not open a code PR as the first step. Start with a well-written Issue or Discussion unless a maintainer asked for the PR or the change is only i18n/localization.
|
||||
4. Do not delete the Contributor License Agreement section at the bottom. The CLA bot requires it.
|
||||
1. DO NOT OPEN A CODE PULL REQUEST unless a maintainer explicitly asked you to, or the change is strictly limited to i18n/localization.
|
||||
2. Target the `dev` branch. PRs targeting `main` will be closed.
|
||||
3. Do not delete the Contributor License Agreement section at the bottom. The CLA bot requires it.
|
||||
-->
|
||||
|
||||
# Pull Request
|
||||
|
||||
Thanks for wanting to improve Open WebUI. The most useful contribution is usually a clear, well-written Issue, not an unsolicited code pull request.
|
||||
**Do not open a code pull request unless a maintainer has explicitly requested it or the change is limited to i18n/localization.**
|
||||
|
||||
Open a code pull request only when a maintainer asks for one, or when the change is only i18n/localization. For real, reproducible bugs, start with a well-described [Issue](https://github.com/open-webui/open-webui/issues). For feature requests, UI/UX changes, behavior changes, architecture changes, suspected fixes, or unconfirmed approaches, start with an active [Discussion](https://github.com/open-webui/open-webui/discussions).
|
||||
The most useful way to help is to give us a clear understanding of the problem: report reproducible bugs in [Issues](https://github.com/open-webui/open-webui/issues) and share proposals in [Discussions](https://github.com/open-webui/open-webui/discussions). We use that context to evaluate solutions and refine the implementation internally, accounting for the broader codebase and ongoing work. External implementations usually require substantial reworking to fit the project's standards, and coordinating those revisions usually takes more effort than developing the solution internally. Please follow this process before investing time in a pull request. PRs opened outside these guidelines are generally closed without review.
|
||||
|
||||
Before continuing, make sure the linked Issue or Discussion explains the user-facing problem, the expected outcome, the affected workflow, and any examples, logs, screenshots, constraints, or reproduction details needed for maintainers to evaluate it.
|
||||
## Maintainer Request
|
||||
|
||||
If you have implementation notes, include them as reference in the Issue or Discussion. If you want to share code as reference, include it there as a local diff, patch, or branch note. Do not open a pull request for reference code.
|
||||
|
||||
Unsolicited PRs may be closed without review, especially when they introduce product, architecture, compatibility, dependency, or maintenance decisions that have not been discussed.
|
||||
Link the maintainer's request for this PR, or state that the change is limited to i18n/localization.
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] I have read and I understand the [contribution policy](https://docs.openwebui.com/contributing/#submit-code).
|
||||
- [ ] This PR targets the `dev` branch.
|
||||
- [ ] This PR links to a well-described, confirmed Issue or active Discussion: `Closes #___` / `Relates to #___`.
|
||||
- [ ] A maintainer explicitly asked me to open this PR, or this PR only updates i18n/localization.
|
||||
- [ ] The change is one logical unit with no unrelated commits.
|
||||
- [ ] I matched nearby code patterns and avoided unnecessary new settings, abstractions, or dependencies.
|
||||
- [ ] I manually tested the changed workflow and any nearby behavior that could be affected.
|
||||
- [ ] I have not added or rewritten automated tests, fixtures, snapshots, or testing infrastructure unless a maintainer explicitly requested them.
|
||||
- [ ] I updated relevant docs, including the [Open WebUI Docs Repository](https://github.com/open-webui/docs), if needed.
|
||||
- [ ] I added screenshots for UI changes, and a recording when motion or interaction matters.
|
||||
- [ ] I reviewed any AI-generated code before submitting it.
|
||||
|
|
@ -50,9 +49,11 @@ Use one of the following prefixes:
|
|||
|
||||
Describe the change, the problem it solves, and the impact on users.
|
||||
|
||||
## Testing
|
||||
## Verification
|
||||
|
||||
List the exact manual checks you ran. Include commands, setup details, screenshots, or recordings where helpful.
|
||||
Describe how you reproduced the problem and manually checked the behavior before and after the change. Include exact steps, setup details, and relevant logs, screenshots, or recordings. Report results from relevant existing checks and anything you could not verify.
|
||||
|
||||
Do not add or rewrite automated tests unless a maintainer explicitly requests them. Tests that repeat an implementation's assumptions can pass while preserving the same mistake; maintainers determine the regression coverage needed. Do not remove, disable, or weaken existing tests to make the change pass.
|
||||
|
||||
## Changelog Entry
|
||||
|
||||
|
|
|
|||
248
CHANGELOG.md
248
CHANGELOG.md
|
|
@ -5,6 +5,253 @@ All notable changes to this project will be documented in this file.
|
|||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.11.4] - 2026-09-21
|
||||
|
||||
### Added
|
||||
|
||||
- 📉 **Far smaller slim image.** A slim build now comes down at around 175 MB, near enough 89% smaller than the last release, the local models, the packages around them and the tools that installed them all gone from it; what that changes about the way an instance behaves is set out under Changed below and in the documentation. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1), [Commit](https://github.com/open-webui/open-webui/commit/d27aa72ab4a7b5632b4ad49e8467081ad3d7ebb4)
|
||||
- 📦 **Smaller standard image.** The image no longer carries a second copy of Python, two sets of fonts nothing ever loaded, packages nothing imports, or the tool that installed them, taking about 170 MB off a standard build. [#29731](https://github.com/open-webui/open-webui/pull/29731), [#29723](https://github.com/open-webui/open-webui/pull/29723), [#29725](https://github.com/open-webui/open-webui/pull/29725), [#29726](https://github.com/open-webui/open-webui/pull/29726), [#29728](https://github.com/open-webui/open-webui/pull/29728), [Commit](https://github.com/open-webui/open-webui/commit/91f8775b28b52c9ae7f2ab990cf2490bda8055d6), [Commit](https://github.com/open-webui/open-webui/commit/98fcb844e1b19f7dd6289af26273cdec5447dc52), [Commit](https://github.com/open-webui/open-webui/commit/508de20779168003e538bb936b49a31d4a8fb8bb), [Commit](https://github.com/open-webui/open-webui/commit/a1c02098aa2687c72482a59117efe643b785df51)
|
||||
- 🧑💻 **Skills from a terminal.** Skills a connected terminal server offers now sit beside workspace skills everywhere skills are picked — the "$" and "/" menus, the integrations menu and the skills panel, each marked Terminal — and are used the same way: picking one puts its instructions, its folder and the files it ships with in front of the model, and a model that was only told a skill exists can open it itself. They follow whichever terminal is selected and clear when that changes. [Commit](https://github.com/open-webui/open-webui/commit/e69236bccb1e12d14045b09b76ebac0490014851)
|
||||
- 📔 **Terminal instructions file.** A model working with a terminal is now handed the AGENTS.md sitting in that terminal's home directory, read afresh at the start of every turn, so the instructions you keep beside your work reach the model without being pasted in. [Commit](https://github.com/open-webui/open-webui/commit/946be432375057dc18dbcc7c3513feb322a83a4b), [Commit](https://github.com/open-webui/open-webui/commit/f6922a4c4293449805938c80fbe54174994a1fc6)
|
||||
- 📇 **Automatic skill discovery.** Every skill you can reach is now listed to the model by name and description, and the full text of one is loaded only when it decides to use it; before, a skill you had not selected in the message box was invisible to it, and this applies to models with built-in tools on. [Commit](https://github.com/open-webui/open-webui/commit/e69236bccb1e12d14045b09b76ebac0490014851)
|
||||
- 🌄 **Model background images.** A workspace model can now carry a background image, uploaded in its editor and drawn behind the chat whenever that model is selected, sitting below a folder's own background and above your personal one, and it travels with the model through export and import. [Commit](https://github.com/open-webui/open-webui/commit/66addbd6b47bfb25cf9aa37ec70d24db5b28b6b6)
|
||||
- 📓 **Skill creation from a chat.** Typing "/skills:create" in a chat that already has content, with a terminal selected, turns the workflow you just went through into a reusable skill written to ".agents/skills" under that terminal's root working directory rather than whatever folder the shell happens to sit in, and authored to Open WebUI's skill standards. [Commit](https://github.com/open-webui/open-webui/commit/113c56fc8c1986359556107a20e206159ca32f59), [Commit](https://github.com/open-webui/open-webui/commit/924a4a10fbd0be508a69faf66bf08ac9761bc24d), [Commit](https://github.com/open-webui/open-webui/commit/58078ab3045cc7aee409d9215f68adf0e02c9165), [Commit](https://github.com/open-webui/open-webui/commit/d25f6c7135e0aee93dc2c840ce320d97617a6e47), [Commit](https://github.com/open-webui/open-webui/commit/a096961a31499be23890b98a040ecf2917405568)
|
||||
- 🖥️ **Terminal tabs per command.** The terminal pane now carries a tab for each command a model is running alongside your own shell, so you can watch them as they go, move between them and take the shell yourself, each tab opening with the command that produced it. Opening the pane puts you in a tab, starting your shell where nothing else is running, and closing the last tab folds the pane away again. [Commit](https://github.com/open-webui/open-webui/commit/54a7a7a7ce22725074c29c7e827446f5dce1421c), [Commit](https://github.com/open-webui/open-webui/commit/f3eade42aead0a3b96ada3a300d6c294c89f74a8), [Commit](https://github.com/open-webui/open-webui/commit/de1203f9b5d3d8b2a84bf5c89d1c61542469367a), [Commit](https://github.com/open-webui/open-webui/commit/675b9839f17df94f866c871cbdb9881323a18a8a)
|
||||
- 📂 **Folder uploads to terminals.** The file browser beside a terminal takes a folder now, dropped onto it or picked from the Upload Folder entry in its menus, and rebuilds what is inside it as it goes, subfolders and all, where before a drop uploaded only the files sitting loose at the top. [Commit](https://github.com/open-webui/open-webui/commit/f80ef8bd001d7ef650fb278d6fbd05bea4afad0f), [Commit](https://github.com/open-webui/open-webui/commit/4cc0d48b4d83503199bcc5f2322881722971a499), [Commit](https://github.com/open-webui/open-webui/commit/674760bfc1122e0a19fe299e05a86d1cbb528e6f)
|
||||
- ⚖️ **Side-by-side file comparison.** Picking two files in the file browser lights up a Compare button that lays them side by side or one above the other, numbering the lines, marking what was added and removed down to the part of the line that changed, counting both, and letting you swap which is which or leave whitespace out of it, a choice it remembers; a terminal too old to offer the comparison says so rather than failing quietly. [Commit](https://github.com/open-webui/open-webui/commit/8556033c6b64fa53e44152ebab1f889a578529af), [Commit](https://github.com/open-webui/open-webui/commit/3808eace6c2beb1904f0f04fdd443ec544f94acb)
|
||||
- 🌐 **Suggested prompts in your language.** A fresh install now offers its suggested starter prompts in the language the interface is set to, rather than the same four in English for everyone, and the model defaults panel carries a way back to those defaults once its own suggestions have been edited. [Commit](https://github.com/open-webui/open-webui/commit/30881dbcc966206cc15fb0b2276b41d88f7c0f09), [Commit](https://github.com/open-webui/open-webui/commit/9fba2843b16a111b035949f5865e29ec6cf5ac9c)
|
||||
- 📏 **Exa result length cap.** Web search through Exa can be held to a number of characters per result, set beside its key in the admin web search settings or as "EXA_MAX_CONTENT_LENGTH". [Commit](https://github.com/open-webui/open-webui/commit/12b14124b9376eccf2ba7cf3dba92bc145493703)
|
||||
- 🇪🇺 **European web search option.** Staan can now be picked as the web search provider, giving deployments that need search inside the EU an option they do not have to host themselves, configured from the admin web search settings or through "STAAN_API_KEY", "STAAN_MARKET" and "STAAN_MAX_SNIPPETS". [#30138](https://github.com/open-webui/open-webui/pull/30138), [#26006](https://github.com/open-webui/open-webui/discussions/26006), [#30303](https://github.com/open-webui/open-webui/pull/30303), [#30301](https://github.com/open-webui/open-webui/issues/30301)
|
||||
- 🗣️ **Per-language names and descriptions.** A model, tool, skill, function, banner or arena entry can now hold its name, description, starter prompts and valve labels once per language, written in a searchable table in its own editor or brought in as a JSON file, which is refused where a translation drops one of the placeholders the original fills in. The interface shows the wording for the language it is set to, and falls back to the plain text where that language has none. [Commit](https://github.com/open-webui/open-webui/commit/7b6562e3358956ec71eed05352538a646d047734), [Commit](https://github.com/open-webui/open-webui/commit/858ab727d2d80bb3c9ef40f01a32de7a12ca75c9), [Commit](https://github.com/open-webui/open-webui/commit/3facfa61d413945d7144d26b827fcce1f3aef7c5), [Commit](https://github.com/open-webui/open-webui/commit/85b11a4f3504434bd9cd0748629b8a84817fb4f2)
|
||||
- ✏️ **Interface text you can reword.** An administrator can now replace the interface's own wording language by language from a panel in the admin settings, with a replacement refused where it drops one of the placeholders the original fills in. [Commit](https://github.com/open-webui/open-webui/commit/67ac1a4e937271a02bfb02a330aa99dc8192cbf4)
|
||||
- 🔓 **Turning off the sign-in form.** The box asking for an email address and a password can now be taken off the sign-in page from the authentication settings, where until now it could only be set before the server started, leaving single sign-on or a directory to sign people in. [Commit](https://github.com/open-webui/open-webui/commit/c4a349651e34bf5d0920bb5e26707ae9961ac39f)
|
||||
- 🏷️ **Custom file metadata.** Metadata attached to an uploaded file now travels with the pieces that file is split into and arrives with the retrieved sources, the oversized and internal fields left out, and an operator can name in "RAG_SOURCE_METADATA_KEYS" which of those fields the model itself gets to see alongside the text. [#29499](https://github.com/open-webui/open-webui/pull/29499), [#29486](https://github.com/open-webui/open-webui/issues/29486), [Commit](https://github.com/open-webui/open-webui/commit/894655f66b9563890e63c76090ddecc90a311eb2), [#29502](https://github.com/open-webui/open-webui/pull/29502), [#29696](https://github.com/open-webui/open-webui/pull/29696)
|
||||
- 🗑️ **Quick delete shortcut.** Holding Shift over a note in the list or grid, or over a row on the automations page, turns its trailing controls into a delete button, removing the entry in one click rather than the three the menu and its confirmation ask for. [#29635](https://github.com/open-webui/open-webui/pull/29635), [#29633](https://github.com/open-webui/open-webui/issues/29633), [#29640](https://github.com/open-webui/open-webui/pull/29640), [#29637](https://github.com/open-webui/open-webui/issues/29637)
|
||||
- 🔀 **Diffs are drawn as diffs.** A diff or patch block in a reply is now laid out as one, with the file and hunk headings, the old and new line numbers, and the added and removed lines picked out down to the part of the line that changed, and a button to switch to the plain text and back. [Commit](https://github.com/open-webui/open-webui/commit/254e29b9af634145dde0450451a36f0dfbd8f421)
|
||||
- ✒️ **A formatting switch for notes.** A note you can edit carries a Formatting switch in its menu: turned off, Markdown you type or paste stays as the characters you wrote, a paste keeps its plain text, and a web address is left as text, while formatting already in the note is untouched. [Commit](https://github.com/open-webui/open-webui/commit/d8f27e745bd31c89efa25cbe0f41bb0f70576fc5)
|
||||
- 🎛️ **Admin model list filters.** The models page in the admin settings can now be narrowed to the base models a connection offers or to the ones built in the workspace, alongside the filters for enabled, disabled, visible and hidden. [Commit](https://github.com/open-webui/open-webui/commit/9e634c0c56e0a060717f1e24a11b082daf664036)
|
||||
- 🔍 **Settings search by name.** The search box in settings now matches each setting's own name and description rather than a keyword list kept per page, in whichever language you are using and regardless of accents, it leaves out anything your permissions do not let you change, and opening a result no longer clears what you typed. [Commit](https://github.com/open-webui/open-webui/commit/7cbaabe02fc3c7c040008e82cf85f590bd0c346a), [Commit](https://github.com/open-webui/open-webui/commit/9d98ffcddf792be8d567ea39370d39fcac649acd), [Commit](https://github.com/open-webui/open-webui/commit/c82634b9d01adadbf9780ff84a0a8168fdc4fdea)
|
||||
- 🆘 **A model for every chat.** Opening a conversation whose model has since been retired no longer leaves it stranded with nothing selected; it falls back to your default model, then the configured default, then the first one available, and a chat that still has a live model keeps it. [#29757](https://github.com/open-webui/open-webui/pull/29757)
|
||||
- ⚡ **Non-blocking search.** Searching the text of chats and knowledge, and the grep a model runs over a knowledge base, now run beside the rest of the server rather than in front of it, so a long search no longer keeps other requests waiting. [#29621](https://github.com/open-webui/open-webui/pull/29621), [Commit](https://github.com/open-webui/open-webui/commit/d9c8de9c39fca7c4756e0f9bd599b09ebe435208)
|
||||
- ✴️ **Lighter shared note editing.** Notes written by several people at once no longer echo every keystroke back to the server once per watcher; the traffic between editors falls by half with two of them and keeps falling as more join, so shared notes stay smooth as the room grows. [#28185](https://github.com/open-webui/open-webui/pull/28185)
|
||||
- 🔢 **Sorted knowledge listings.** The "ls", "tree" and "find" commands a model runs over a knowledge base now sort by name and take "-t", "-S" and "-r" for newest first, largest first and reversed. [#29840](https://github.com/open-webui/open-webui/pull/29840)
|
||||
- 🪪 **Authentication type header.** A request to an OpenAI or Ollama connection now carries "X-OpenWebUI-Auth-Type", saying whether the person behind it signed in through the browser or called with an API key; its name is set with "FORWARD_USER_INFO_HEADER_AUTH_TYPE" and "{{AUTH_TYPE}}" works in custom headers. [Commit](https://github.com/open-webui/open-webui/commit/ee46e2664ab23bacc536fed21c06ab19067d695b)
|
||||
- 💡 **Follow-up ghost text.** Once a reply has finished, the first of the follow-up questions it suggests now sits greyed inside the empty message box as well as under the reply: Tab writes it out, and typing anything of your own clears it away. [Commit](https://github.com/open-webui/open-webui/commit/7aaa4a692e949724913834efc47c57572042703b)
|
||||
- 🪵 **Model refusal logging.** A request turned away with "Model not found" now writes a warning naming the account, the model and why it was refused, while the message the caller sees stays as vague as before. [Commit](https://github.com/open-webui/open-webui/commit/f5fcf4c89fb27ed39825875d573446fa84aa2a5b)
|
||||
- 🤲 **Cheaper chat requests behind Redis.** Where several servers share their websocket traffic through Redis, a chat request no longer pulls the whole shared model list from Redis to ask about a couple of models; each worker keeps its own copy and refetches only when the list actually changed, so the cost stops growing with the number of models configured. [#28176](https://github.com/open-webui/open-webui/pull/28176), [#28167](https://github.com/open-webui/open-webui/issues/28167)
|
||||
- 💽 **Long conversations stream cheaper.** Writing or reading a single message no longer loads, checks and rewrites the entire conversation to change a few hundred bytes, so streaming into a long chat stops getting more expensive as it grows, and the whole round trip a message event costs falls to a fraction on conversations of thousands of messages. [#28184](https://github.com/open-webui/open-webui/pull/28184), [#28169](https://github.com/open-webui/open-webui/issues/28169)
|
||||
- 🧱 **In-place streaming appends.** Streamed replies can be assembled with CPython's in-place string append, turned on with "ENABLE_CHAT_RESPONSE_STREAM_INPLACE_APPEND" and left off by default while it is rolled out gradually. [Commit](https://github.com/open-webui/open-webui/commit/113c56fc8c1986359556107a20e206159ca32f59), [Commit](https://github.com/open-webui/open-webui/commit/924a4a10fbd0be508a69faf66bf08ac9761bc24d), [#30066](https://github.com/open-webui/open-webui/pull/30066)
|
||||
- 📘 **Direct connection guidance.** The connections and integrations pages in the personal settings now say plainly that a direct connection leans on your browser session to keep requests running, which suits testing and temporary use rather than everyday work. [Commit](https://github.com/open-webui/open-webui/commit/5d66dd6ad291db17894953714c01fd4303093e7e), [Commit](https://github.com/open-webui/open-webui/commit/15e2259a2ae1b7dec1f93a52078304dd3a79a14c)
|
||||
- 🔄 **General improvements.** Various improvements were implemented across the application to enhance performance, stability, and security.
|
||||
- 🌐 **Translation updates.** Translations for Japanese, Traditional Chinese, Korean, Finnish, Russian, Ukrainian, German, Spanish, Portuguese (Brazil), Arabic, Arabic (Bahrain), Azerbaijani, Bulgarian, Bengali, Tibetan, Bosnian, Catalan, Cebuano, Czech, Danish, Basque, Croatian, Dutch, Italian, Turkish, Simplified Chinese, Bosnian, Catalan, Danish, Estonian, Finnish, Galician, Croatian, Kabyle, Norwegian Bokmål, Portuguese (Portugal), Romanian and Turkmen were enhanced and expanded.
|
||||
|
||||
### Fixed
|
||||
|
||||
- 🛡️ **Security Advisory**: This release includes security and access-control fixes. We recommend updating production deployments at your earliest convenience. Not all security fixes in this version may be enumerated in the fixed section. Some may be withheld for a short time to give administrators time to upgrade. [Advisories](https://github.com/open-webui/open-webui/security)
|
||||
- 🔑 **Tokens stay out of logs.** A failure part way through signing in with an identity provider no longer writes the credentials it was handed into the application log, recording the provider and the error it reported instead. [#29709](https://github.com/open-webui/open-webui/pull/29709)
|
||||
- 🔐 **Sign-in role mapping.** Roles sent by an identity provider were in some setups not applied, leaving an account at the default role, and a sign-in whose roles cannot be read is now refused rather than let through. [Commit](https://github.com/open-webui/open-webui/commit/10d1cfe6375f207acaa531e857edb575ded2cfc3)
|
||||
- 🚫 **Blocked sign-in groups saved.** Comma-separated group names typed into the blocked groups field now take effect once saved, where the save stored the text as written and the sign-in check then read it as nothing; a group name carrying a comma of its own survives a save too. [Commit](https://github.com/open-webui/open-webui/commit/3fc1146c13d7b4b7a67c7fd058014436f80e0dfd)
|
||||
- 🚪 **Signing out ends the session.** Signing out or having every token revoked left any live connection that account already had in place, and those are now cut at the same moment; a token already revoked can no longer be replayed to cut someone else's newer session. [Commit](https://github.com/open-webui/open-webui/commit/3a6d0fd203050401481bdb8621b827089d393817)
|
||||
- 🛡 **Diagrams and SVG stay on this origin.** Mermaid diagrams and SVG previews, attached or uploaded, no longer follow references pointing at another origin, a picture, style or configuration line among them, and a diagram that reaches outside is refused with an error; a file whose drawing leans on an external sprite now shows that part blank. [#30271](https://github.com/open-webui/open-webui/pull/30271)
|
||||
- 📲 **Terminal sessions follow access changes.** A terminal connection now re-checks your access to it every ten seconds for as long as it stays open, so removing someone's permission or deactivating their account ends their live terminal on the next check, on every worker and not only the one they signed in through. [Commit](https://github.com/open-webui/open-webui/commit/a1189a2d757407530a0c1c85a41b46cd6a4f7da5), [Commit](https://github.com/open-webui/open-webui/commit/e8bd0661d3774248c318286849041b94bb861909)
|
||||
- 🔒 **Connection model listing access.** The endpoints that list the models on a single Ollama or OpenAI connection could be reached by a signed-in account of any role, and now check the caller's role. [#29619](https://github.com/open-webui/open-webui/pull/29619)
|
||||
- 🔏 **Knowledge file access.** A file's access through a knowledge base now follows its actual attachment alone, rather than also a collection name left behind on the file record. [#29937](https://github.com/open-webui/open-webui/pull/29937)
|
||||
- 🌳 **Knowledge directories stay inside their base.** A directory id supplied to the knowledge file routes is now refused where it belongs to a different knowledge base, where a caller-supplied id could reach into a directory tree outside the one being addressed. [Commit](https://github.com/open-webui/open-webui/commit/a9541c18ca2a056f4ccfe56b9c733cba74b86b49), [#29887](https://github.com/open-webui/open-webui/pull/29887)
|
||||
- 🍪 **Cookie forwarding scope.** A connection authenticating as the signed-in person is handed this instance's browser cookies only where its new Forward cookies switch is on, which an instance relying on it has to turn on after upgrading. [Commit](https://github.com/open-webui/open-webui/commit/b71744b17823f7a3a9a64cb363e79a7164ed5b23), [Commit](https://github.com/open-webui/open-webui/commit/3cda47cdb449aac970426f0929c9e5f8bb8bd807), [Commit](https://github.com/open-webui/open-webui/commit/f9f815c86220f809fad5ee1300c44f55dea74c79)
|
||||
- 🗝️ **Revocation list fallback.** Where the revocation list is kept in Redis and Redis cannot be reached, a token is now accepted rather than the request failing, so signing out may not take effect until Redis is back. [Commit](https://github.com/open-webui/open-webui/commit/c1615bec2f5f143084b5fcc04f42ebfa0dade9df), [Commit](https://github.com/open-webui/open-webui/commit/6a85abb3f5e002f3069007213e16e4ef60776890)
|
||||
- 🗒 **Home page chat drafts.** A message typed into a chat started from the home page now comes back after a reload, text, uploads and all, where the draft was stored under a key the page never read again. [#29762](https://github.com/open-webui/open-webui/pull/29762), [#29760](https://github.com/open-webui/open-webui/issues/29760)
|
||||
- 🏞️ **S3-hosted chat images.** A chat image whose host echoes a Content-Encoding nothing asked for, an S3 or MinIO object uploaded with that metadata among them, no longer breaks the reply it sits in; it is sent as its link the way an unreachable image already was. [#29623](https://github.com/open-webui/open-webui/pull/29623)
|
||||
- 🪆 **Branch descent guard.** Stepping between reply branches walked a chat's children without tracking where it had already been, so a history that pointed back at itself spun forever and locked the tab, and a child id with no message behind it threw; every one of the eleven places that walk now shares a single guarded helper. [#30070](https://github.com/open-webui/open-webui/pull/30070)
|
||||
- 🗒️ **Note save on exit.** A note's title and its attachments save a moment after they change, and leaving the note inside that moment dropped the save, so the notes list kept showing the old title until the page was reloaded; the pending save now finishes before the next page opens. [#29745](https://github.com/open-webui/open-webui/pull/29745), [#29744](https://github.com/open-webui/open-webui/issues/29744)
|
||||
- 📷 **Attachments sent without text.** Sending an image or a file with nothing typed to a model that has skills attached replaced the empty message with the list of skill names, so the model answered with its own catalogue instead of looking at what you sent. [#30045](https://github.com/open-webui/open-webui/pull/30045), [#30040](https://github.com/open-webui/open-webui/issues/30040)
|
||||
- 🖍️ **SVG attachments read as text.** An SVG attached to a chat went up as a picture and came back rejected by every model that tried to decode it, and is now read as its source text instead, which needs a model that accepts file uploads. [#30102](https://github.com/open-webui/open-webui/pull/30102), [#30100](https://github.com/open-webui/open-webui/issues/30100)
|
||||
- 🩹 **Notes survive a small edit.** Asked to add a section or change a few lines, a model would send the whole note back and lose the rest of it with nothing to undo, because the editing tool never said when to edit a range instead; it now spells out the range rules the handler already enforced. [#30048](https://github.com/open-webui/open-webui/pull/30048)
|
||||
- 📋 **Note paste placement.** A plain paste replaced more of the note than was selected, and a paste into a code block broke out of the block instead of going inside it; both now land exactly where they were dropped. [Commit](https://github.com/open-webui/open-webui/commit/d8f27e745bd31c89efa25cbe0f41bb0f70576fc5)
|
||||
- 🧿 **Insert into note on read-only notes.** The Insert into note action no longer appears on a note you may only read, which offered it where it could not land. [#30223](https://github.com/open-webui/open-webui/pull/30223), [#30174](https://github.com/open-webui/open-webui/issues/30174)
|
||||
- 🤝 **Note sharing survives a save.** Saving a note you had been given access to took that sharing away, so the note went private and everyone else lost it. [#30175](https://github.com/open-webui/open-webui/pull/30175)
|
||||
- 🌍 **Collaborative link handling.** In a note two people are writing at once, a plain web address arriving from the other editor was turned into a link on your screen but not on theirs; it is now left as written. [Commit](https://github.com/open-webui/open-webui/commit/d8f27e745bd31c89efa25cbe0f41bb0f70576fc5)
|
||||
- 📑 **Tika 4 document formatting.** Where document extraction runs against a Tika server on version 4, the text now comes back as Markdown rather than a flat run of characters, so headings, lists and tables survive into what the model reads. [Commit](https://github.com/open-webui/open-webui/commit/ba34bee2d17e1c00238c85a8aab1a11a60db164e)
|
||||
- ⏳ **Session expiry accuracy.** A session with an identity provider now expires with the token it actually calls with, rather than whichever of its two tokens ran out first, which renewed it early and dropped it whenever that failed. [Commit](https://github.com/open-webui/open-webui/commit/aaaf26fb8ede28854dd660b11b85ba6bafe64007)
|
||||
- 🧩 **Tool steps in finished replies.** A reply that called tools showed each step as it ran, then lost them the moment the reply completed, because the provider's closing message replaced everything on screen rather than joining it; the closing message is now merged into what is already there, and a tool call is no longer mistaken for its own result. [Commit](https://github.com/open-webui/open-webui/commit/e1bfefdf9f8f2012cecfc7d81079bd6fff147932), [Commit](https://github.com/open-webui/open-webui/commit/31b272d3c93b87636c920f2aa68d6caaa07ae79a)
|
||||
- 🧭 **Streamed replies follow along.** A reply arriving over the response events now keeps the view at the bottom as it is written, as replies on the older path already did, unless you have scrolled up yourself. [Commit](https://github.com/open-webui/open-webui/commit/dfde08aa7391d924359b27f5768411d7a533a6ec), [Commit](https://github.com/open-webui/open-webui/commit/88e78b7819b28ffe91f3dff24d8c5d992004197e)
|
||||
- 🚿 **Tool results arrive when the tools are done.** The results of a model's tool calls now reach the reply as soon as the round that produced them finishes, instead of being held back with the rest of the streaming until the throttle let them through, and a channel reply or a continuation no longer carries the picture data a tool handed the model. [Commit](https://github.com/open-webui/open-webui/commit/478d1785fd27f400c614a5700f5b38ec9de412f9)
|
||||
- 🧗 **Tagged blocks while streaming.** A reply using reasoning, solution or code interpreter tags briefly showed the raw tag text in the message as it arrived, because the chunk carrying it reached the browser before the cleaned output did; the cleaned output is now sent the moment a tag is taken out, and the code an interpreter block writes fills that block rather than the message body. [Commit](https://github.com/open-webui/open-webui/commit/58b36765a7c20f5943a3180bd289de48876d0878)
|
||||
- 📗 **Excel in the code interpreter.** Reading or writing a spreadsheet in code the browser runs failed outright because the library that handles them never reached the browser, and it is now shipped alongside the rest. [#30140](https://github.com/open-webui/open-webui/pull/30140), [#30130](https://github.com/open-webui/open-webui/issues/30130)
|
||||
- 📀 **Bundled charts and formatting.** Drawing a chart with seaborn in code the browser runs fetched the library over the internet at that moment rather than taking it from what ships, and the code editor's Format button failed outright for anyone who is not an administrator; both now work from what comes with the application, offline included. [#30148](https://github.com/open-webui/open-webui/pull/30148), [#30145](https://github.com/open-webui/open-webui/issues/30145)
|
||||
- 🙋 **Typed answer submission.** In the card a model puts up to ask you a question, choosing one of its options on the last question sends your answers straight away, but typing your own into the Other box left Submit answers greyed out with no way to send it, and it now turns on as soon as that box has text. [#29494](https://github.com/open-webui/open-webui/pull/29494), [#29311](https://github.com/open-webui/open-webui/issues/29311)
|
||||
- 🥇 **A truthful Recommended badge.** The card a model puts up to ask you a question marks its first option Recommended, but nothing told the model that, so the badge fell on whichever option happened to be listed first; models are now asked to put the option they recommend there. [#30196](https://github.com/open-webui/open-webui/pull/30196), [#30195](https://github.com/open-webui/open-webui/issues/30195)
|
||||
- 🎚️ **Partial settings permissions.** An account barred from changing the interface settings can now save its system prompt, notifications, audio, keyboard shortcuts and pinned models, which were refused along with them. [Commit](https://github.com/open-webui/open-webui/commit/98a920168e2eea435ac15e1ad3d679946631e41d)
|
||||
- 🎧 **Speech file types kept.** Saving the audio settings emptied the list of file types accepted for speech recognition, so transcription then turned away the recordings it had taken before. [#30208](https://github.com/open-webui/open-webui/pull/30208)
|
||||
- 🚻 **Group picker without permission.** The picker for sharing a chat, a note or a knowledge base with a group was offered to accounts not allowed to share with groups, and is now hidden from them. [#30189](https://github.com/open-webui/open-webui/pull/30189)
|
||||
- 🗜️ **Per-field settings saves.** Only the settings actually changed are now stored, rather than the whole object, so another tab's older copy no longer overwrites them and a default an administrator changes still reaches everyone, and pinning a model, reordering the list or picking a default from outside the settings window now saves the same way. [Commit](https://github.com/open-webui/open-webui/commit/98a920168e2eea435ac15e1ad3d679946631e41d), [#30183](https://github.com/open-webui/open-webui/pull/30183)
|
||||
- ⚠️ **Failed settings feedback.** Settings that could not be saved were shown as saved anyway until the page was reloaded, because the interface stored them locally without waiting on the server; a failure now raises an error and leaves the panel as it was. [Commit](https://github.com/open-webui/open-webui/commit/98a920168e2eea435ac15e1ad3d679946631e41d)
|
||||
- 🔠 **Menu text scaling.** The entries in the menus that drop down across the interface now scale with the rest of it, rather than staying at a fixed size while the menu around them grew. [#29493](https://github.com/open-webui/open-webui/pull/29493), [#29488](https://github.com/open-webui/open-webui/issues/29488)
|
||||
- 🪞 **Account menu highlighting.** An account menu entry carrying a pin button beside it now lights up across the whole row in the shared colour, and a long label no longer pushes the pin out of the menu. [Commit](https://github.com/open-webui/open-webui/commit/e723dcda58f638a5da2398743d22d3e5854042bf)
|
||||
- 🔲 **Shift-click file selection.** Shift-clicking a file in the terminal's file browser now adds that range to what is already selected, and clicking through a selected file removes its range. [Commit](https://github.com/open-webui/open-webui/commit/f80ef8bd001d7ef650fb278d6fbd05bea4afad0f), [Commit](https://github.com/open-webui/open-webui/commit/4cc0d48b4d83503199bcc5f2322881722971a499), [Commit](https://github.com/open-webui/open-webui/commit/674760bfc1122e0a19fe299e05a86d1cbb528e6f)
|
||||
- 🛟 **Attachment name collisions.** Attaching a file to a message on an instance that puts attachments in a terminal's working directory replaced whatever file of that name was sitting there; the upload now takes the next free name, "report (1).pdf" beside "report.pdf", and the attachment shows the name it was saved under, though two uploads arriving at the same moment from different browsers can still land on the same name. [Commit](https://github.com/open-webui/open-webui/commit/d70053e44993f271d534fb87d2b40724b028fca2)
|
||||
- 📛 **Failed upload feedback.** A file that could not be written to the terminal now says so, rather than passing in silence while the browser refreshed as though it had arrived. [Commit](https://github.com/open-webui/open-webui/commit/f80ef8bd001d7ef650fb278d6fbd05bea4afad0f), [Commit](https://github.com/open-webui/open-webui/commit/4cc0d48b4d83503199bcc5f2322881722971a499), [Commit](https://github.com/open-webui/open-webui/commit/674760bfc1122e0a19fe299e05a86d1cbb528e6f)
|
||||
- ☑️ **Unchecked checkbox defaults.** A prompt variable written as a checkbox with a default of false opened the form already ticked, as did False, "false", 0 and "0", because any non-empty default counted as ticked; it is now ticked only where the value really is true. [#30037](https://github.com/open-webui/open-webui/pull/30037), [#30036](https://github.com/open-webui/open-webui/issues/30036)
|
||||
- 🎯 **Prefilled question timing.** Opening a chat from a link holding a question sent it before the box had it, so a question naming a variable went off with the variable unfilled; the send now waits for the text to be in place and filled in. [Commit](https://github.com/open-webui/open-webui/commit/3808eace6c2beb1904f0f04fdd443ec544f94acb), [Commit](https://github.com/open-webui/open-webui/commit/a23b579233276e40159eb615917b9aeb7d5ed5c9)
|
||||
- 📜 **Task list height.** The list of steps a model works through ran as long as it needed and pushed the rest of the reply down the page; it now stops at a quarter of the window's height and scrolls within itself. [Commit](https://github.com/open-webui/open-webui/commit/307b9b9133f0c7ad899f4b76226059da6f3177eb)
|
||||
- ⎋ **Escape key targeting.** The shortcut for closing a dialog always shut the settings window, whichever dialog was actually in front, so a dialog opened from within settings took both away at once; each dialog now answers the shortcut for itself, as it already did for the escape key. [#29830](https://github.com/open-webui/open-webui/pull/29830), [#29817](https://github.com/open-webui/open-webui/issues/29817)
|
||||
- 🎹 **Message pair shortcut.** The shortcut that adds an empty question and answer to a chat also sent whatever was typed in the message box, so the pair arrived alongside a message you had not meant to send yet. [#30167](https://github.com/open-webui/open-webui/pull/30167)
|
||||
- 🥁 **Collapsing the task list.** Folding away the list of tasks under a reply also sent whatever you had typed in the message box. [#30202](https://github.com/open-webui/open-webui/pull/30202)
|
||||
- 🕰️ **Temporary chat links.** A link carrying the temporary chat marker opened an ordinary chat, because the marker was written into the address but never read back when the page loaded, and such a link now opens the temporary chat it promises. [Commit](https://github.com/open-webui/open-webui/commit/67adde31936d1d2e656140f9edd898b1fbb18a1c)
|
||||
- ⌨️ **Recording a new shortcut.** Pressing a key combination to record it as a shortcut also ran whatever that combination was already bound to, so setting one up did the thing you were trying to rebind. [#30160](https://github.com/open-webui/open-webui/pull/30160)
|
||||
- 🔃 **Stale tab reload.** A tab still running the previous build met an error page after the server was updated instead of loading the new one, because every image built from Docker carried the same version stamp; the stamp now follows the build, and a stale tab reloads as it was meant to. [#29832](https://github.com/open-webui/open-webui/pull/29832), [#29831](https://github.com/open-webui/open-webui/issues/29831)
|
||||
- 📌 **Channel code headers.** The bar naming a piece of code in a channel thread or its pinned messages now sits flush at the top of the panel, clipped to the block, rather than floating over the code as it scrolls. [#29836](https://github.com/open-webui/open-webui/pull/29836), [#29835](https://github.com/open-webui/open-webui/issues/29835)
|
||||
- 📨 **Duplicated proxy headers.** A reply proxied from a terminal server or an OpenAI or Ollama connection no longer carries that server's own "Server" and "Date" headers, which had a reverse proxy in front logging a duplicate line for every one. [#29841](https://github.com/open-webui/open-webui/pull/29841), [#29824](https://github.com/open-webui/open-webui/issues/29824), [#29843](https://github.com/open-webui/open-webui/pull/29843)
|
||||
- 🎣 **Testing an image connection.** The Verify button beside an image generation connection saved the whole image configuration first and then tested whichever engine was active rather than the connection beside it, and it now tests exactly that connection and changes nothing. [Commit](https://github.com/open-webui/open-webui/commit/64bbdf7a73724986fac8bcf6e880fe32ef9ac495)
|
||||
- 🪝 **Responses API tool strictness.** A workspace tool, MCP server or OpenAPI server reaching a model on the Responses API was turned into a strict schema where it had never asked to be, so the model filled every optional field with empty strings, zeros and empty arrays, and search and filter tools were handed values where leaving them out was meant. [#30046](https://github.com/open-webui/open-webui/pull/30046), [#27750](https://github.com/open-webui/open-webui/issues/27750)
|
||||
- 🪃 **Responses API tool calling.** A forced tool choice sent to a connection on the Responses API went out in the wrong shape and was refused by providers that check it, and a tool call in a reply that was not streamed came back as empty text, so nothing reading the API ever saw it. [#30095](https://github.com/open-webui/open-webui/pull/30095), [#30085](https://github.com/open-webui/open-webui/issues/30085)
|
||||
- ⛓️ **Missing tools in links.** Opening a chat from a link whose "tools" or "tool-ids" parameter names a tool that no longer exists, or that the account cannot see, kept that id in the selection and sent it with the message; ids matching no tool the account has are now dropped and the rest of the link works as before. [#29803](https://github.com/open-webui/open-webui/pull/29803)
|
||||
- 🚫 **Model editor error messages.** A workspace model that cannot be loaded for editing now says so, instead of sending you back with "You do not have permission to edit this model" whatever the real reason. [#29694](https://github.com/open-webui/open-webui/pull/29694), [#29629](https://github.com/open-webui/open-webui/issues/29629)
|
||||
- 👥 **Directory updates that were dropped.** A change your identity provider sent as an add or a remove, or without naming the attribute it was changing, was accepted and then quietly discarded, so a rename or a deactivation never reached the account; those now take effect. [Commit](https://github.com/open-webui/open-webui/commit/ad9da981680c0fd9151c803a366a01545ea907c1)
|
||||
- 🗃️ **Directory request validation.** A provisioning request carrying the wrong kind of value, or an attribute Open WebUI does not support, now comes back as an error instead of passing in silence, and a sync that changes nothing no longer marks the account as touched. [Commit](https://github.com/open-webui/open-webui/commit/ad9da981680c0fd9151c803a366a01545ea907c1)
|
||||
- 🔘 **Model editor save button.** Saving a workspace model whose model list could not be refreshed afterwards now reports the error and frees the Save button, rather than leaving it disabled and spinning though the model had been saved. [Commit](https://github.com/open-webui/open-webui/commit/dc98e3023fc6e0113dbad76545cdb15e014db6ad), [Commit](https://github.com/open-webui/open-webui/commit/cc5479d16d5caf28ec19a16ffa1ee4f3dbc0f63f)
|
||||
- 🫱 **Rating a reply again.** Switching a reply's rating from one thumb to the other kept the score and the reason given the first time, and rating a reply whose feedback had since been deleted failed outright; both now record the rating you just gave. [#30075](https://github.com/open-webui/open-webui/pull/30075), [#30077](https://github.com/open-webui/open-webui/pull/30077)
|
||||
- 📶 **Sorting feedback by user.** The User column in the admin feedback history did nothing when clicked, and now sorts by who left the feedback. [#30191](https://github.com/open-webui/open-webui/pull/30191)
|
||||
- 🪂 **Closing the Edit User dialog.** Changes typed into the admin Edit User dialog and then abandoned showed on the row in the user list until the page was reloaded, and closing the dialog now drops them. [#30193](https://github.com/open-webui/open-webui/pull/30193)
|
||||
- 🪙 **Model defaults save.** Saving the model defaults in the admin settings put the selected, pinned and ordered models back as they stood when the page was opened, undoing anything changed in between. [#30206](https://github.com/open-webui/open-webui/pull/30206)
|
||||
- 👤 **A custom gender shown back.** An account whose gender is a wording of its own came back to an empty dropdown in the account form, and the form now shows Custom with that wording beside it. [#30215](https://github.com/open-webui/open-webui/pull/30215)
|
||||
- 🗓️ **Clearing a calendar event.** Emptying the repeat, the description or the location of a calendar event did not take and the old wording came back, and those fields can be cleared again. [#30204](https://github.com/open-webui/open-webui/pull/30204)
|
||||
- 📕 **Required prompt dropdowns.** A prompt's form could be sent with a required dropdown left unchosen, and now asks you to pick something first. [#30078](https://github.com/open-webui/open-webui/pull/30078)
|
||||
- 🎫 **Account form required fields.** The account form now refuses to save with a required field left empty, the way the admin user dialog does, and a date of birth is no longer demanded of accounts that never set one. [#30296](https://github.com/open-webui/open-webui/pull/30296), [#30295](https://github.com/open-webui/open-webui/issues/30295)
|
||||
- 🐑 **Model clones keep their base.** Cloning a model from the admin Models settings now carries the model it was built on, where the clone came out detached from it, and an arena model no longer offers Clone at all. [#30080](https://github.com/open-webui/open-webui/pull/30080), [#30079](https://github.com/open-webui/open-webui/issues/30079)
|
||||
- 🧤 **Model sharing survives a save.** Saving a model you cannot fully share no longer strips the access entries you cannot re-create, where an editor resending every stored entry had each one re-checked against what its author may grant, and a save from someone with narrow rights quietly took the model private for everyone else. [Commit](https://github.com/open-webui/open-webui/commit/754c4b5762ed0d79954ff28c4e06631004dc31b3), [#30093](https://github.com/open-webui/open-webui/pull/30093), [#30087](https://github.com/open-webui/open-webui/issues/30087)
|
||||
- 📠 **Prompt version saves.** Saving a new version of a prompt without Set as Production leaves the live prompt exactly as it was, where the draft quietly took its place, and the editor now shows the production text the moment a version is set. [#30231](https://github.com/open-webui/open-webui/pull/30231), [#30230](https://github.com/open-webui/open-webui/issues/30230), [#30233](https://github.com/open-webui/open-webui/pull/30233), [#30232](https://github.com/open-webui/open-webui/issues/30232)
|
||||
- 💬 **Model description round trips.** A workspace model whose description is switched from the default back to custom saves again, where the switch read the field as empty and the description was dropped on save. [#30249](https://github.com/open-webui/open-webui/pull/30249), [#30247](https://github.com/open-webui/open-webui/issues/30247)
|
||||
- 🎚 **Compaction threshold saves.** The context compaction threshold set in general settings now reaches the model parameters, where the value never left the page. [#30270](https://github.com/open-webui/open-webui/pull/30270), [#30269](https://github.com/open-webui/open-webui/issues/30269)
|
||||
- 📢 **Speech engine defaults.** Switching the text to speech engine now applies that engine's own default voice and model, where the change kept the previous engine's settings in place. [#30289](https://github.com/open-webui/open-webui/pull/30289), [#30288](https://github.com/open-webui/open-webui/issues/30288)
|
||||
- 🗞 **MinerU key in local mode.** The document settings save again in local mode with the MinerU key left empty, where the form demanded a key it did not need. [#30299](https://github.com/open-webui/open-webui/pull/30299), [#30298](https://github.com/open-webui/open-webui/issues/30298)
|
||||
- 🫂 **Group dialog reset.** The new-group dialog opens empty after a group is created, where the next one came up holding the group just made. [#30280](https://github.com/open-webui/open-webui/pull/30280), [#30279](https://github.com/open-webui/open-webui/issues/30279)
|
||||
- 🧵 **Thread reply notifications.** Clicking the notification for a reply written inside a thread dropped you at the bottom of the channel with the thread still shut and the reply nowhere in sight; it now opens the thread the reply belongs to. [#29856](https://github.com/open-webui/open-webui/pull/29856), [#29855](https://github.com/open-webui/open-webui/issues/29855)
|
||||
- 🔽 **Dropdown arrow spacing.** The arrow in the dropdowns drawn no wider than their contents, among them the provider on a new connection, no longer overlaps the last characters of the longest choice. [#29866](https://github.com/open-webui/open-webui/pull/29866), [#29865](https://github.com/open-webui/open-webui/issues/29865)
|
||||
- 🧊 **Lowercase header handling.** Headers from an upstream that writes them in lower case, as anything served by uvicorn does, are now matched without regard to case, so "Content-Encoding" is stripped and clients stop failing to decompress a body the server had already decoded. [#29843](https://github.com/open-webui/open-webui/pull/29843)
|
||||
- 🛑 **Complete chat stop.** Where a chat had more than one task in flight, stopping it, deleting it, or closing a note being worked on could stop the first and leave the rest running to the end, both because a task that had already finished ended the round early and because the list being worked through was rewritten underneath it as each one was cleared away; every task is now stopped in turn, so a reply that was calling tools stops calling them rather than running on to its own limit, though instances sharing their state through Redis were not affected. [Commit](https://github.com/open-webui/open-webui/commit/e35b907f737e625b900b3a03d61890f67ab0c4b0), [#29844](https://github.com/open-webui/open-webui/pull/29844), [#29816](https://github.com/open-webui/open-webui/issues/29816)
|
||||
- 🖊️ **Channel code blocks.** Where a model answers in a channel with structured output, the code inside it was drawn as plain highlighted text rather than in the editor every other message uses, so it could not be edited in place and a diff in it could not be opened for editing; it now renders the same way as everywhere else. [#29861](https://github.com/open-webui/open-webui/pull/29861)
|
||||
- 📐 **Code block edits on collapse.** An unsaved edit inside a code block stays on screen when the block is folded away and opened again, where collapsing it showed the saved text though the edit was still pending. [#30284](https://github.com/open-webui/open-webui/pull/30284), [#30283](https://github.com/open-webui/open-webui/issues/30283)
|
||||
- 🚨 **Code run errors with output.** A code run that ends with an error now shows the error alongside what it printed, where a run that printed anything at all showed its error nowhere. [#30286](https://github.com/open-webui/open-webui/pull/30286), [#30285](https://github.com/open-webui/open-webui/issues/30285)
|
||||
- 🔕 **Reactions on read-only channels.** The reaction picker no longer appears on a channel open to you as a viewer alone, matching the reply and menu options already hidden there. [#30241](https://github.com/open-webui/open-webui/pull/30241), [#30240](https://github.com/open-webui/open-webui/issues/30240)
|
||||
- 🎯 **Knowledge search accuracy on PostgreSQL.** A fresh install using PostgreSQL built its search index before a single piece of text existed to organise it around, so the index was never fit for the content that arrived afterwards and every search quietly returned the wrong passages; the index now waits until there is enough text to build on, and until then searches read everything exactly. An install already carrying such an index can restore it by rebuilding that one index. [#30143](https://github.com/open-webui/open-webui/pull/30143), [#30134](https://github.com/open-webui/open-webui/issues/30134)
|
||||
- 🗂 **Knowledge file filter on first click.** The File content filter in a knowledge base now narrows the listing the first time it is clicked, where the first click only armed the checkbox and everything stayed listed until it was clicked again. [#30211](https://github.com/open-webui/open-webui/pull/30211), [#30210](https://github.com/open-webui/open-webui/issues/30210)
|
||||
- 🔭 **Knowledge base search recall.** Where many knowledge bases are stored together, a search of one holding a small share of what is stored found only a small share of its matches, and the shortfall grew as the store did; the search now keeps looking until it has enough from the knowledge base you asked for, and "PGVECTOR_ITERATIVE_SCAN" switches that off or makes it strict. [#30142](https://github.com/open-webui/open-webui/pull/30142), [#30135](https://github.com/open-webui/open-webui/issues/30135)
|
||||
- 🪣 **Searches that found nothing.** A knowledge search that came back empty never handed its database connection back, so enough of them left knowledge search failing outright until the server was restarted; connections are returned now on PostgreSQL and on openGauss alike. [#30142](https://github.com/open-webui/open-webui/pull/30142), [#30133](https://github.com/open-webui/open-webui/issues/30133), [#30144](https://github.com/open-webui/open-webui/pull/30144)
|
||||
- 🧽 **Knowledge folder deletion.** Deleting a folder without moving what was in it up a level dropped the files from the listing but left their text in the search index and the files themselves in storage, so a model went on retrieving and citing pages from a folder that was no longer there; the text is now removed with the folder, and a file no other knowledge base holds is deleted with it unless file retention is switched on. [Commit](https://github.com/open-webui/open-webui/commit/17dbc6f001aeea25ae1df528cb79bb272eca4a77)
|
||||
- 🦀 **Regex searches over chat and knowledge files.** A pattern search now runs in time proportional to the text whatever the pattern, where a nasty expression could stall the search for good, and the patterns it accepts follow RE2's rules, with no lookarounds or backreferences and character classes matching ASCII only. [Commit](https://github.com/open-webui/open-webui/commit/97e013a66169c4fcd7f26ab9e41a4ff260ffd4da)
|
||||
- ⏰ **Automation schedule parsing.** An automation whose rule puts the time in "DTSTART" is now read at that hour, rather than listed at midnight and opened at nine, which moved when it ran as soon as it was saved again. [Commit](https://github.com/open-webui/open-webui/commit/540467b90a430e47e536c20710878b342de659fb)
|
||||
- ⏲️ **Recurrence counts and start dates.** A rule repeating a set number of times, ten or a hundred, is read as the limited repeat it is, where any count beginning with a one was treated as one-shot, and a rule carrying its start date on the same line as its repeat text now follows the start you picked, on schedules and calendar events alike. [#29262](https://github.com/open-webui/open-webui/pull/29262)
|
||||
- ⌛ **Stalled schedules and stuck tasks.** A schedule whose rule takes too long to work out no longer holds up the round that evaluates it and is skipped with a warning, and background tasks shared through Redis now expire once their worker falls silent, after "REDIS_TASK_TTL" seconds. [Commit](https://github.com/open-webui/open-webui/commit/5fb869db221d9599a576e8e2eea9c3314947d25e)
|
||||
- 📣 **Model mention chips.** A mention whose ID carried anything beyond letters, digits and a little punctuation, such as the brackets in some workspace model IDs, stayed on screen as the raw "<@…>" text both in the box you type in and in the message once sent; any ID without a space in it is now drawn as a chip. [#29864](https://github.com/open-webui/open-webui/pull/29864)
|
||||
- 😀 **Multi-codepoint emoji.** An emoji whose shortcode is several codepoints, the flags among them, is inserted whole, where only its first part reached the message. [#30213](https://github.com/open-webui/open-webui/pull/30213), [#30212](https://github.com/open-webui/open-webui/issues/30212)
|
||||
- 🔔 **Custom webhook names.** A webhook target named with a space or a slash is now tidied the way an automatic name always was, so it can still be edited, deleted, made the default or tested afterwards. [#29947](https://github.com/open-webui/open-webui/pull/29947)
|
||||
- 🪛 **Paginated MCP tool lists.** A server that hands its tools back a page at a time had only the first page read, so the rest were never offered to a model; the whole list is now collected before the tools are built. [Commit](https://github.com/open-webui/open-webui/commit/ffae4116a8d58a820f1770c41c69dafe4d51c881)
|
||||
- 💭 **Anthropic thinking blocks.** Open WebUI's own thinking blocks are no longer forwarded to an OpenAI-compatible backend, which since 0.11.0 made a strict server such as NVIDIA Dynamo refuse an Anthropic client's second turn; signed blocks still pass through. [#29849](https://github.com/open-webui/open-webui/pull/29849), [#29799](https://github.com/open-webui/open-webui/issues/29799)
|
||||
- 🗨️ **Channel model terminals.** A model with a terminal chosen in the workspace had that choice honoured in a chat but dropped where it answered in a channel or ran as a channel automation, so it worked without one; it now carries the same terminal everywhere, alongside the tools, filters and features it already carried. [Commit](https://github.com/open-webui/open-webui/commit/c78ad89934095c4e42e3f059d400a24fe5681de2)
|
||||
- 🧺 **Deleted channel messages leave no quotes.** Deleting a channel message now clears the quote of it sitting on every reply and drops it from the reply box, where a reply kept showing the deleted message and could still be sent addressed to it. [#30314](https://github.com/open-webui/open-webui/pull/30314), [#30313](https://github.com/open-webui/open-webui/issues/30313)
|
||||
- 🔌 **Terminal tools need a terminal.** A model was offered the tools that read your terminal and type into it whether or not the chat had a terminal switched on and connected in your browser, so it could reach for one that was not there; those tools are now handed over only for the terminal the chat has open. [Commit](https://github.com/open-webui/open-webui/commit/ca1eefe2937081b010ffef3985997d17d3332fa3), [Commit](https://github.com/open-webui/open-webui/commit/1ddba7e2c6f625fbc2c131eb24d3b9775ad898ad)
|
||||
- 🔤 **Custom header encoding.** The custom headers a connection sends are encoded once the values are filled in, so a person's name or group carrying anything beyond plain ASCII, a line break included, no longer breaks the request or reaches the other end as something else. [Commit](https://github.com/open-webui/open-webui/commit/7a4a4b93dca34f8ce0c481b180d3eea23797e984)
|
||||
- 🆎 **Chromium spellcheck corrections.** Picking a suggestion from the browser's own spelling menu in the message box did nothing, or put the misspelling straight back, because a highlight meant for the notes editor was being drawn over the selection and rebuilding the text underneath it, taking the browser's spelling marks with it; that highlight is now kept out of the message box and only drawn where the editor is not in use. [#29952](https://github.com/open-webui/open-webui/pull/29952), [#29944](https://github.com/open-webui/open-webui/issues/29944)
|
||||
- 🈁 **IME composition while renaming.** Confirming a chat rename with Enter or Escape part way through typing with an input method editor now finishes the composition without saving or cancelling the rename, where the key press acted at once. [Commit](https://github.com/open-webui/open-webui/commit/85146206f60a22385ed27dae30d4f00e7e2675eb)
|
||||
- 🐳 **Dotless host addresses.** With local web fetching turned on, an address pointing at a container name on the same network, "http://apprise:8000" and the like, is now accepted rather than refused as invalid. [#29945](https://github.com/open-webui/open-webui/pull/29945), [#28161](https://github.com/open-webui/open-webui/issues/28161)
|
||||
- 🪧 **False skill mentions.** Something written as "<$fh>" in a message, as Perl and other languages do, was taken for a mention of a skill and quietly removed before the model saw it, and drawn on screen as a chip; only mentions naming a skill that exists and is turned on are treated as mentions now. [Commit](https://github.com/open-webui/open-webui/commit/0edd731c7422870aa109fd0b758c6d68c06655da)
|
||||
- 🎒 **Skill settings survive saving.** Saving a skill from its editor cleared the tags and translations it carried and switched it back on where it had been disabled, and all of that now survives the save. [#30185](https://github.com/open-webui/open-webui/pull/30185)
|
||||
- 🫥 **Webhook avatar forwarding.** Turning "ENABLE_PROFILE_IMAGE_URL_FORWARDING" off stops browsers being sent on to outside picture addresses, and a channel webhook's picture was sent on regardless; it now serves the built-in picture like the rest, in the webhooks dialog as well as the message list, where the dialog had been sending every viewer's browser straight to the outside address. [#29889](https://github.com/open-webui/open-webui/pull/29889), [#29892](https://github.com/open-webui/open-webui/pull/29892)
|
||||
- 🪟 **Statistics window origin.** The window that shares chat statistics with the community accepted requests from any page that opened it and answered to anywhere; it now reads and replies only where the community site is at the other end. [#29918](https://github.com/open-webui/open-webui/pull/29918)
|
||||
- 🖼️ **Tool image rendering.** Where a tool answered with an image tucked inside an object or a list rather than on its own, the image was written into the conversation as its raw text, a single screenshot costing hundreds of thousands of tokens and crowding out everything else; such an image is now taken out wherever it sits and attached to the reply, so the model is handed the picture and you see it. An older fault that let every second image through untouched goes with it, and in a saved chat such an image is now kept as a file and referred to rather than written into the conversation itself, so the chat stays small. [#29665](https://github.com/open-webui/open-webui/pull/29665), [#29208](https://github.com/open-webui/open-webui/issues/29208), [Commit](https://github.com/open-webui/open-webui/commit/d372bec70427fe2d568e052ce5e1529e2ad41da9)
|
||||
- ⏱️ **Stopped reply state.** Pressing stop saved the reply as finished while the parts inside it were still marked as running, so a block went on reading "Thinking..." or "Executing..." and came back that way after every reload; those parts are now closed off as the reply is stopped, and an open tab settles at once rather than only after a reload, while a tool call still waiting for your approval keeps its prompt. [#29495](https://github.com/open-webui/open-webui/pull/29495), [#29281](https://github.com/open-webui/open-webui/issues/29281)
|
||||
- 🩺 **Knowledge sync errors.** A knowledge base sync that fails now names the file it happened on and what the browser said, rather than reporting nothing beyond "Error accessing directory". [#29507](https://github.com/open-webui/open-webui/pull/29507)
|
||||
- 🚧 **Terminal proxy restrictions.** Requests passed through to a terminal server are now refused where they aim at that server's administrative endpoints, are not followed on to somewhere else, and are turned away where the path carries characters a parser would rewrite. [Commit](https://github.com/open-webui/open-webui/commit/51bb8cb142f72503e861eeee25ae4dc73c26c36b)
|
||||
- ⛔ **Malformed tool calls.** Where a model asked for a tool with arguments that were not an object at all, a bare list or string, the reply stopped there; the model is now told what was wrong with the call and can try again. [Commit](https://github.com/open-webui/open-webui/commit/fed94c9f5af8a59660425d52df09e15fbedb25bc)
|
||||
- 📡 **Broken stream reporting.** Where something failed part way through streaming an answer out of "/api/chat/completions", the stream simply stopped, leaving a client waiting on an answer that would never finish; it now closes with an error and a proper end of stream. [Commit](https://github.com/open-webui/open-webui/commit/c0fb36c9b833a85a3a7364e195cf54f3d6c7a787)
|
||||
- 🧷 **Chat unblocked after an error.** A reply that failed, on a content filter or an exhausted quota, left the chat turning away everything you typed after it and stopped the message queue. Only the failed reply now ends, so the chat carries on and the other replies in a multi-model answer keep writing. [Commit](https://github.com/open-webui/open-webui/commit/dbb17a5725f9d7f844a6eee63ffca0bd077c7d94)
|
||||
- 🫙 **Empty failed replies in history.** An assistant turn that ended in an error with nothing written is no longer handed back to the model as part of the conversation when you send your next message. [Commit](https://github.com/open-webui/open-webui/commit/dbb17a5725f9d7f844a6eee63ffca0bd077c7d94)
|
||||
- 📭 **Empty page uploads.** Adding a web address to a knowledge base that came back without any text failed with a bare "Error uploading file" and, where the upload itself was refused, left the row sitting in the list; the reason now reaches you as it was given, and the row is taken away. [Commit](https://github.com/open-webui/open-webui/commit/6786ae1797eadaad7464a147213790e2d272822b)
|
||||
- 🎞️ **Tool embed scope.** The frames a tool call can ask to have shown, which run scripts of their own, were drawn wherever a message was rendered, a channel among them; they are now drawn only in the replies of the chat you are in, and never in a channel. [#29985](https://github.com/open-webui/open-webui/pull/29985)
|
||||
- 🚰 **Rejected picture addresses.** A model entry carrying a picture address the server refuses no longer leaves that address in memory, where anyone signed in could pile them up. [#29971](https://github.com/open-webui/open-webui/pull/29971)
|
||||
- 🖌 **Editing a stored image.** Asking a model to edit an image this instance already holds now works whatever host its address names, where a container name, a default port or a self-composed host made the edit fail with a generic loading error. [#29691](https://github.com/open-webui/open-webui/pull/29691), [#29220](https://github.com/open-webui/open-webui/issues/29220)
|
||||
- 🧪 **Memory replies carry less.** The memory tool no longer hands the model each memory's stored metadata, and a memory now records the model's id rather than the whole model entry. [Commit](https://github.com/open-webui/open-webui/commit/e9a0164690a8b1e190bdc8f4613e9d918b26d327)
|
||||
- 🔇 **Memory fully off.** With memory switched off, stored memories are no longer folded into a reply's context and the memory tools are no longer offered to the model, where both went on reaching it behind the switch. [#30228](https://github.com/open-webui/open-webui/pull/30228), [#30227](https://github.com/open-webui/open-webui/issues/30227)
|
||||
- 🧠 **Memory review behind the switch.** The background review that drafts new memories from a conversation no longer runs when memories are switched off or the account is barred from them, where it went on spending a task-model call every interval turn and failing at the write. [#30309](https://github.com/open-webui/open-webui/pull/30309)
|
||||
- 🏗️ **Terminal server save button.** Saving a terminal server now waits for the save to finish before the dialog closes and cannot be set off twice by a second click. [Commit](https://github.com/open-webui/open-webui/commit/1cdd7aa459d6e96905324b452600ff56369d8a4e)
|
||||
- 🗺️ **Terminal file panel paths.** The file panel beside a terminal now opens the file a model just wrote even when it is named with a relative path, where the panel could not match the name, jumped to the root and dragged the session's working directory with it. [#30282](https://github.com/open-webui/open-webui/pull/30282), [#30051](https://github.com/open-webui/open-webui/issues/30051)
|
||||
- 🍴 **Forked chat folder.** Forking a chat put the copy in the original's folder even where you cannot write to that folder; it is now created outside any folder unless you can. [#30069](https://github.com/open-webui/open-webui/pull/30069)
|
||||
- 🪢 **Dropping a folder in place.** Dragging a folder onto the folder it already sits in failed with "Folder already exists", and is now taken for the no-op it is. [#30169](https://github.com/open-webui/open-webui/pull/30169)
|
||||
- 🗝️ **Read-only folders read-only everywhere.** A folder shared with you as a viewer no longer shows its edit controls on the empty-chat page, where they appeared and a save went through or failed depending on rights the page never checked. [Commit](https://github.com/open-webui/open-webui/commit/ee3ece1e2b8c9a38c94faf354ca020d66aba801d)
|
||||
- 🚿 **Deleting a folder, keeping chats.** Removing a folder while keeping the chats inside it was refused for an account not allowed to delete chats, even though nothing was being deleted, and it now goes through. [#30163](https://github.com/open-webui/open-webui/pull/30163)
|
||||
- 🏷️ **Tag cleanup after deletion.** An administrator deleting someone else's chat tidied unused tags out of their own account rather than the owner's, leaving the owner with tags nothing points at. [#30171](https://github.com/open-webui/open-webui/pull/30171)
|
||||
- 🌱 **Forking past an unfinished reply.** A chat that held an interrupted reply anywhere in it refused every fork from then on and never recovered; forking now waits only on a reply actually being generated, and a turn paused for tool approval still forks with its prompt showing. [#30131](https://github.com/open-webui/open-webui/pull/30131), [#30128](https://github.com/open-webui/open-webui/issues/30128)
|
||||
- 💾 **Deleted tool memory.** Deleting a tool or a function left the whole of its code in memory for as long as the server ran; it is now let go of along with the rest. [#29983](https://github.com/open-webui/open-webui/pull/29983)
|
||||
- 🐌 **Sign-in rate limiting.** Counting sign-in attempts through Redis no longer stops the whole worker until Redis answers, so a slow Redis stops freezing every other request with it. [#29977](https://github.com/open-webui/open-webui/pull/29977)
|
||||
- ♻️ **Session pool cleanup.** The task that clears out abandoned websocket sessions now carries on through an error from Redis instead of ending for good, and stops properly at shutdown. [#29976](https://github.com/open-webui/open-webui/pull/29976), [#29979](https://github.com/open-webui/open-webui/pull/29979)
|
||||
- 🛰️ **Direct connections across workers.** A reply streamed over a direct connection no longer goes quiet part way through where several servers share their websocket traffic through Redis; the events it lives on now travel between workers the way the rest already did. [Commit](https://github.com/open-webui/open-webui/commit/0180efecf362d487e0c30f040f5948c325fbe337)
|
||||
- 🫧 **Empty document ids.** A save arriving for a document with no id at all was filed against that empty id and never cleared, so anyone signed in could pile them up; nothing is filed for it now. [#29980](https://github.com/open-webui/open-webui/pull/29980)
|
||||
- 🔬 **Page fetch CPU spin.** Fetching a page through the browser-driven loader never returned where that page opened a WebSocket, holding a worker thread at full CPU for the life of the process and costing another core on every further fetch, which left the whole instance slow. [#30050](https://github.com/open-webui/open-webui/pull/30050), [#30024](https://github.com/open-webui/open-webui/issues/30024)
|
||||
- 🔁 **Duplicate search tracebacks.** A vector database outage wrote a full traceback twice for every collection and query pair, turning one outage into hundreds of identical stack traces per message on every replica; a single record now names every collection that failed. [#29981](https://github.com/open-webui/open-webui/pull/29981)
|
||||
- 🧯 **Page fetch logging.** Fetching a page through the browser-driven loader filled the log with tracebacks where the page closed while it was still pulling pieces of itself, as sites behind Cloudflare and similar do; those requests are now let go of before the page closes. [#29325](https://github.com/open-webui/open-webui/pull/29325), [#28869](https://github.com/open-webui/open-webui/issues/28869)
|
||||
- 📤 **Tool export scope.** Exporting all tools at once returned every tool the account could see, the source of a tool shared for reading included; it now returns only the tools it may edit, matching the single-tool export and the way models already export. [#29310](https://github.com/open-webui/open-webui/pull/29310)
|
||||
- 🗳️ **Partial workspace exports.** Exporting the prompts or the models from the workspace wrote out only the page you happened to be looking at, so most of them were quietly left out of the file; both now export everything you are allowed to. [#30187](https://github.com/open-webui/open-webui/pull/30187)
|
||||
- 🧳 **Imported chats keep more.** A chat brought back from an export arrived unpinned and unarchived and without the variables it was saved with, and all three now survive the round trip. [#30155](https://github.com/open-webui/open-webui/pull/30155), [#30151](https://github.com/open-webui/open-webui/pull/30151)
|
||||
- 🗂️ **Unarchiving from search.** The menu on a search result offered to archive a chat that was already archived and said it had been archived when it had been brought back, and it now names and reports whichever of the two it did. [#30177](https://github.com/open-webui/open-webui/pull/30177)
|
||||
- 🙈 **Folder filters with no match.** A chat search narrowed by a folder name that matches no folder now finds nothing, where the folder filter was quietly dropped and every chat came back. [#30273](https://github.com/open-webui/open-webui/pull/30273), [#29959](https://github.com/open-webui/open-webui/discussions/29959)
|
||||
- 🧹 **Sidebar after bulk actions.** Archiving, deleting or unarchiving every chat at once, or importing a batch of them, left the folders and the pinned chats in the sidebar showing what was no longer there until the page was reloaded, and a bulk action that failed no longer reports success. [Commit](https://github.com/open-webui/open-webui/commit/8b3ee2827241ccc952a3073b2a6bbfad5df01827)
|
||||
- 🔐 **Model pictures follow model access.** The picture belonging to a model is now shown only to people who can see that model, where anyone signed in could fetch it and tell an existing model from an unknown one by which picture came back. [#29700](https://github.com/open-webui/open-webui/pull/29700)
|
||||
- 🚪 **Webhook pictures follow channel access.** The picture belonging to a channel webhook is now shown only to people with access to that channel, where anyone signed in could fetch it or be sent on to wherever it pointed, and it is refused outright where channels are turned off. [#29703](https://github.com/open-webui/open-webui/pull/29703)
|
||||
- 📎 **Safer Word document previews.** Previewing a Word document no longer renders an HTML sub-document embedded inside it, and a link in one opens only where it points at a web address, a mail address or a telephone number. [#29699](https://github.com/open-webui/open-webui/pull/29699)
|
||||
- 🚦 **Citation link schemes.** A source attached to a reply now opens only where it points at a web address, falling back to the panel that shows the source rather than following anything else. [#29701](https://github.com/open-webui/open-webui/pull/29701)
|
||||
- 🈚 **Citation chips inside formatted text.** A citation inside bold, italic or linked text now renders its chip, where the formatting took it and the citation vanished from the sentence. [#30278](https://github.com/open-webui/open-webui/pull/30278), [#30277](https://github.com/open-webui/open-webui/issues/30277)
|
||||
- 🐍 **Saving a tool or function.** Saving a tool or function in the admin pages no longer fails with a missing module error from the built-in code formatter, which was not installing everything it needed. [#29503](https://github.com/open-webui/open-webui/pull/29503)
|
||||
- 🛠️ **Tool request duplication.** A tool that writes through an address carrying part of its input no longer has that part repeated in the body of the request as well, which servers checking their input strictly turned away, so those calls now go through. [#29717](https://github.com/open-webui/open-webui/pull/29717), [#29716](https://github.com/open-webui/open-webui/issues/29716)
|
||||
- 🍎 **Answers survive on Apple Silicon.** Asking a question that searches a knowledge base with a locally run reranking model no longer takes the whole server down on a Mac, losing the answer and the connection with it. [#29735](https://github.com/open-webui/open-webui/pull/29735), [#29722](https://github.com/open-webui/open-webui/issues/29722)
|
||||
- 📚 **Web results stop being cited.** Pages a web search only listed are no longer offered to the model as things to cite, which had it attaching a result id to text from a different result and the citations panel resolving that to a title that looked right. [#29631](https://github.com/open-webui/open-webui/pull/29631), [#29627](https://github.com/open-webui/open-webui/issues/29627)
|
||||
- 🔎 **SearchApi errors, news and links.** Web search through searchapi.io now reports a bad key instead of coming back empty, reads the news results it returns alongside its ordinary ones, and hands the web loader the resolved destination link, so citations stop pointing at a redirect page. [#30308](https://github.com/open-webui/open-webui/pull/30308), [#30305](https://github.com/open-webui/open-webui/issues/30305)
|
||||
- 🔼 **Honest version checks.** An instance that cannot reach the release listing now says the check failed, instead of reporting whatever it is running as the newest version and recording nothing about it. [#29626](https://github.com/open-webui/open-webui/pull/29626), [#29580](https://github.com/open-webui/open-webui/issues/29580)
|
||||
- 🏟️ **Arena models report their errors.** A message to an arena model whose provider answers with an error now shows that error in the chat, where it used to fail on something unrelated and leave the real reason unsaid, and titles and tags no longer break the same way. [#29662](https://github.com/open-webui/open-webui/pull/29662), [#29658](https://github.com/open-webui/open-webui/issues/29658)
|
||||
- 🏳️ **Nameless tool calls fail once.** A model endpoint that sends a tool call with no name at all now has that call fail on the spot, rather than the missing name being kept, stored with the message and sent back on the next turn for the endpoint to reject. [#29690](https://github.com/open-webui/open-webui/pull/29690), [#29686](https://github.com/open-webui/open-webui/issues/29686)
|
||||
- 🧹 **Direct connections stop leaking listeners.** A server talking to a direct connection no longer leaves a listener behind for every request that ends any way but a clean finish, which grew without limit while a connection kept failing. [#29509](https://github.com/open-webui/open-webui/pull/29509)
|
||||
- 🚀 **Cheaper model refreshes.** The model registry shared through Redis is now written only when the models themselves change, rather than on every refresh because of the countdown Ollama attaches to a model it holds in memory. [Commit](https://github.com/open-webui/open-webui/commit/649c012ecf308a994ea180127f7f8f94d0aec311)
|
||||
- ✍️ **Continued reply text.** Asking for the rest of a cut-off reply in a temporary chat replaced what was on screen with only the new text, because the message being continued was read back from the saved chat it did not have. It is now taken from the request before the model is called, the continuation joins the same message instead of arriving as a second one, on a connection whose provider is set to llama.cpp the model is told to carry on from the text it is handed rather than repeat it back, opening the result in the message editor no longer shows a line break where the two halves meet, and, where haptic feedback is switched on, a continuation buzzes as it streams like any other reply. [Commit](https://github.com/open-webui/open-webui/commit/77d2000eb79e1cb6ae2004d40e8cca8c9e754cd0), [Commit](https://github.com/open-webui/open-webui/commit/6c7aa3543d21442241f6c53add5ff623ec816c44), [Commit](https://github.com/open-webui/open-webui/commit/57fc344873edc0db9e9f7ff3e9fb167cd80e3ef2), [Commit](https://github.com/open-webui/open-webui/commit/7eefeef4f17118f81c87acb464ad562803a6f26c), [Commit](https://github.com/open-webui/open-webui/commit/d418840aa9c4b77f613308cdaa4f2c6a062a8715), [Commit](https://github.com/open-webui/open-webui/commit/3795d5b29253d4b8d7a0adbab457c7317c40f3c6), [Commit](https://github.com/open-webui/open-webui/commit/57acc2b68f2f9e40b53aa7e609fdd52a4b0d15c4)
|
||||
- 🔗 **Cancelled edits keep attachments.** Cancelling the edit of a message no longer strips the files attached to it, where dropping the edit took the attachments down with it. [#30281](https://github.com/open-webui/open-webui/pull/30281), [#30192](https://github.com/open-webui/open-webui/issues/30192)
|
||||
- 🏎️ **Faster media page reads.** The browser-driven loader pulled every image, video and font a page referenced down through the server before any text was extracted, so a page carrying a few dozen audio players took ten seconds or timed out. Those requests are now dropped before they are made, and the same page comes back in under three seconds, having pulled 4 MB where it used to pull 55. [#29742](https://github.com/open-webui/open-webui/pull/29742), [#29741](https://github.com/open-webui/open-webui/issues/29741)
|
||||
- 📝 **Starting a note from search.** Starting a note from the search box now works when you are already on the notes page, keeps the whole of what you typed including characters such as ampersands and hashes, and no longer makes a further note each time the browser back button is pressed. [#29645](https://github.com/open-webui/open-webui/pull/29645), [#29642](https://github.com/open-webui/open-webui/issues/29642)
|
||||
- 📱 **Apple device replies.** An assistant reply no longer comes up blank in a home screen app, an in-app browser or a desktop-class window on Apple devices, where the check that avoided the drawing fault only recognised Safari itself. [#29734](https://github.com/open-webui/open-webui/pull/29734), [#29688](https://github.com/open-webui/open-webui/issues/29688), [#26712](https://github.com/open-webui/open-webui/issues/26712)
|
||||
- 📊 **Single source relevance.** A reply drawing on a single source now shows how relevant that source is, where the figure appeared only once a second source joined it and so looked as though it came and went. [#29647](https://github.com/open-webui/open-webui/pull/29647), [#29646](https://github.com/open-webui/open-webui/issues/29646)
|
||||
- 🔧 **Arduino sketches upload to knowledge.** A sketch file now reaches the plain text reader like the C++ and header files beside it, rather than being handed to a document extraction server that could make nothing of it and failing the upload. [#29673](https://github.com/open-webui/open-webui/pull/29673), [#29670](https://github.com/open-webui/open-webui/issues/29670)
|
||||
- 📰 **Docling conversion failures.** A file that Docling refuses or fails to convert now fails the upload with the reason Docling gave, rather than breaking with a raw error or quietly filing a placeholder that was then indexed and handed to the model in place of the file. [#30107](https://github.com/open-webui/open-webui/pull/30107), [#29808](https://github.com/open-webui/open-webui/issues/29808)
|
||||
- 📄 **Uploaded text kept as written.** A file whose text contains escape sequences such as the one standing for a non-breaking space is now stored and read by the model exactly as it was written, rather than having some of them rewritten depending on where in the file they sat. [#29736](https://github.com/open-webui/open-webui/pull/29736), [#29732](https://github.com/open-webui/open-webui/issues/29732)
|
||||
- 🔦 **Readable slash command labels.** The entries in the slash command menu no longer show as white text on a white background in the light theme. [#29512](https://github.com/open-webui/open-webui/pull/29512), [#29510](https://github.com/open-webui/open-webui/issues/29510)
|
||||
- ⌨️ **Literal arrow sequences.** A sequence such as three hyphens after a less-than sign is now shown as the characters it is made of rather than drawn as an arrow, which had text look changed when it never was. [#29595](https://github.com/open-webui/open-webui/pull/29595), [#29594](https://github.com/open-webui/open-webui/issues/29594)
|
||||
- 🖌️ **Editing an image you uploaded.** An image already held by Open WebUI can now be used with image editing, where fetching its own link back over the network could fail on a private network or without a sign-in. [Commit](https://github.com/open-webui/open-webui/commit/50413f34824ea49d5b94d3a97f3fe4bb2e881e38)
|
||||
- 🖼 **Playground image edits.** Editing an image in the Images playground now works, where every attempt came back rejected since the request carried its fields under a heading the endpoint never read. [Commit](https://github.com/open-webui/open-webui/commit/c07fa08b995e8d1a1fc2d94a88d8cea691bdc5ee)
|
||||
- 🎨 **A tidier attach webpage dialog.** The row holding the Add button no longer carries a grey band of its own between the address box and the button, matching every other dialog. [#29664](https://github.com/open-webui/open-webui/pull/29664), [#29663](https://github.com/open-webui/open-webui/issues/29663)
|
||||
- 🖱️ **A plain note date.** The date under a note title no longer shows a pointing hand or announces itself as something to press, having never done anything when clicked. [#29708](https://github.com/open-webui/open-webui/pull/29708)
|
||||
- 🌇 **Folder backgrounds on creation.** The background image picked in the Create Folder dialog now arrives on the folder when it is created from the sidebar, where the image was discarded unless the dialog was opened from an existing folder. [#30218](https://github.com/open-webui/open-webui/pull/30218), [#30217](https://github.com/open-webui/open-webui/issues/30217)
|
||||
- 🖇 **Delete Chat shortcut everywhere.** The keyboard shortcut that deletes the open chat now works wherever the chat was opened from, where it did nothing unless the chat's row happened to be on screen in the sidebar at that moment. [#30165](https://github.com/open-webui/open-webui/pull/30165), [#30164](https://github.com/open-webui/open-webui/issues/30164)
|
||||
- 🚮 **Retired chat variables.** A model whose system prompt no longer declares a variable a previous prompt did stops asking for it, where every new chat kept opening the dialog and refusing to send until a value was entered. [#30173](https://github.com/open-webui/open-webui/pull/30173), [#30172](https://github.com/open-webui/open-webui/issues/30172)
|
||||
- 👁️ **Compact hover previews in Safari.** Holding over a chat in the sidebar shows the small preview every other browser shows, rather than one laid out at the width and spacing of a full conversation. [#29734](https://github.com/open-webui/open-webui/pull/29734)
|
||||
- 🔖 **Visible title generation faults.** When a new chat's automatic title cannot be generated, the reason now reaches the log at the default level, rather than only under debug logging where a broken feature looked the same as a switched-off one. [#30106](https://github.com/open-webui/open-webui/pull/30106), [#29533](https://github.com/open-webui/open-webui/issues/29533)
|
||||
|
||||
### Changed
|
||||
|
||||
- 🪶 **Slim starts with nothing configured.** The slim image leaves out the local models and the libraries around them, and still starts and holds a conversation on its defaults; the features that leaned on those models each need an external service of their own. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1)
|
||||
- 🗄️ **Slim database support.** The slim image runs on SQLite, its default, or on PostgreSQL; pointed at MySQL, MariaDB or another engine, or started with AWS RDS IAM logins switched on, it stops with an error instead of starting, and those deployments need the standard image. [Commit](https://github.com/open-webui/open-webui/commit/d27aa72ab4a7b5632b4ad49e8467081ad3d7ebb4)
|
||||
- 📁 **Slim file storage.** The slim image keeps files on local storage, its default; configured for an S3, Google Cloud or Azure bucket, it stops with an error instead of starting, and those deployments need the standard image. [Commit](https://github.com/open-webui/open-webui/commit/d27aa72ab4a7b5632b4ad49e8467081ad3d7ebb4)
|
||||
- 🧮 **Slim searches only through pgvector.** Knowledge search on the slim image needs PostgreSQL with pgvector, and configured for another vector store the instance still starts, and the failure arrives the first time something is searched rather than at startup. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1), [Commit](https://github.com/open-webui/open-webui/commit/d27aa72ab4a7b5632b4ad49e8467081ad3d7ebb4)
|
||||
- 🧠 **Slim embedding requirements.** The slim image carries no embedding or reranking model, so knowledge needs OpenAI, Ollama or Azure OpenAI embeddings and an external reranker, falling back to plain cosine scoring where none is set. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1)
|
||||
- ✂️ **Slim document splitting.** Splitting a document along a downloaded tokenizer is unavailable on the slim image, which leaves splitting by character or by token count. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1)
|
||||
- 📃 **Slim document readers.** The slim image reads text, Markdown, CSV, HTML and XML files as they are; uploading a PDF, a Word file or a presentation fails unless one of the external document extractors is configured. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1)
|
||||
- 🎙️ **Slim speech requirements.** The slim image carries neither local Whisper nor local voices, so speech to text and text to speech need an external engine before they will work. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1)
|
||||
- 🕸️ **Slim web page fetching.** The slim image carries no headless browser, so a web page is fetched over plain HTTP or through an external loader, and a page that draws itself with JavaScript comes back with less of its content than on the standard image. [Commit](https://github.com/open-webui/open-webui/commit/cb942bb94c8dc7941336088fb3392e2398ff56c1), [Commit](https://github.com/open-webui/open-webui/commit/d27aa72ab4a7b5632b4ad49e8467081ad3d7ebb4)
|
||||
- 🔎 **Slim leaves out DDGS.** DDGS, the metasearch provider that needs no key of its own, is not carried in the slim image, so web search there needs a provider with a key. [Commit](https://github.com/open-webui/open-webui/commit/0fa4dea5ff64ea663f162d07c9a1175c39e4aad0)
|
||||
- 📥 **Slim code interpreter packages.** The slim image leaves out the code interpreter's packages, so the browser fetches numpy, pandas, matplotlib, scikit-learn and the rest from "cdn.jsdelivr.net" and the interpreter stops working where that is blocked. [Commit](https://github.com/open-webui/open-webui/commit/98fcb844e1b19f7dd6289af26273cdec5447dc52)
|
||||
- 🧰 **Slim git requirements.** The slim image no longer carries git, so a tool or function whose requirements point at a "git+https://" address fails to install and needs the standard image or a published package. [Commit](https://github.com/open-webui/open-webui/commit/30eed1251301f74e0dfeaad09c41e81320f790fc)
|
||||
- 🧺 **LangChain community removal.** The readers for text, HTML, Word, CSV, PDF and Azure Document Intelligence are now written here rather than taken from "langchain-community", which is no longer installed; a tool or function importing it has to name it in its own requirements from now on. [Commit](https://github.com/open-webui/open-webui/commit/05484aa055a868a49842e1ddff169c19c98b755b)
|
||||
- 🧾 **Undeclared package imports.** Packages that sat in the image only by accident, among them nltk, pymongo, the Google Drive client and the Gemini SDK, are no longer installed, so a tool or function importing one must name it in its own requirements. [#29725](https://github.com/open-webui/open-webui/pull/29725), [#29726](https://github.com/open-webui/open-webui/pull/29726), [Commit](https://github.com/open-webui/open-webui/commit/a1c02098aa2687c72482a59117efe643b785df51)
|
||||
- 🆔 **Model ID whitespace.** A workspace model whose ID contains a space or a tab is now refused in the editor, through the API and on import; one already stored goes on answering but cannot be saved again until it is recreated. [Commit](https://github.com/open-webui/open-webui/commit/8a19e2f867063256bb2836649e2fe80af41ef748)
|
||||
- 🖇️ **Link scheme rendering.** A link in a reply, a citation or a web search result is now rendered only where it points at a web address, a mail address, a telephone number or somewhere inside this instance; anything else, an "ftp://" address or an application link such as "obsidian://" or "vscode://" among them, is shown as the text it is. Two old oddities go with it: a source written as "HTTP://" now becomes a link, and a filename merely containing the letters http no longer becomes one that leads nowhere. [#29890](https://github.com/open-webui/open-webui/pull/29890)
|
||||
- 🧲 **Integrations tab is opt-in.** The Integrations tab in personal settings, where tool and terminal connections of your own are managed, is now hidden until an administrator turns on Direct Integrations under Integrations or sets "ENABLE_DIRECT_INTEGRATIONS", and hiding it leaves existing connections working. [Commit](https://github.com/open-webui/open-webui/commit/6d8e63e3666e1b1aa5540ffda0816be5f40c5271), [Commit](https://github.com/open-webui/open-webui/commit/c82634b9d01adadbf9780ff84a0a8168fdc4fdea)
|
||||
- 📡 **Image connection check endpoint.** The endpoint that checks an image generation connection has moved and now takes the connection to test in the request itself, so anything calling the old address needs updating. [Commit](https://github.com/open-webui/open-webui/commit/64bbdf7a73724986fac8bcf6e880fe32ef9ac495)
|
||||
|
||||
## [0.11.3] - 2026-08-31
|
||||
|
||||
### Added
|
||||
|
|
@ -287,6 +534,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
- 🛎️ **Losing all your settings.** Your interface settings are no longer wiped by a session that failed to load them, which could happen with no action on your part and cleared everything from your theme to your model parameters; saving now changes only the settings you actually changed, and a session that cannot load them tells you instead of carrying on as though you had none. [#27766](https://github.com/open-webui/open-webui/issues/27766), [Commit](https://github.com/open-webui/open-webui/commit/ad8c79f68657bd3bcf5db6be650e498bb904b36b)
|
||||
- 🖲️ **Losing the collapsed sidebar.** With the sidebar collapsed, opening a chat no longer pushes the narrow sidebar strip off the edge of the screen, which left no way to reopen the sidebar short of shrinking the window to phone size. [#28501](https://github.com/open-webui/open-webui/pull/28501), [#28500](https://github.com/open-webui/open-webui/issues/28500)
|
||||
- 🧯 **Timers that fail without saying so.** A timer whose reply cannot be generated, such as one set against a model that has since been removed, is now recorded as failed with the reason, instead of being marked as completed while the reply never arrives. [#27785](https://github.com/open-webui/open-webui/pull/27785), [#27783](https://github.com/open-webui/open-webui/issues/27783)
|
||||
- 🎲 **Timers stop for retired owners.** A timer whose owner has been deleted or demoted to pending is recorded as an error instead of running, so a retired account no longer answers through a timer it set while active. [#30220](https://github.com/open-webui/open-webui/pull/30220)
|
||||
- 🖊️ **Message buttons in channels.** The buttons that appear when you hover a channel message now sit above the message rather than over its content, so they can be clicked on a message that starts with a code block or a table, and so the code and table controls stay clickable too. [#27737](https://github.com/open-webui/open-webui/pull/27737), [#27736](https://github.com/open-webui/open-webui/issues/27736)
|
||||
- 🔡 **Searching for non-English tags and text.** Searching workspace models by tag, or prompts and automations by their contents, now finds entries containing characters outside the English alphabet, where roughly half were missed depending on which settings were in force when each one was saved. [#28399](https://github.com/open-webui/open-webui/pull/28399)
|
||||
- 🔭 **Searching the calendar without an end date.** Asking a model to search your calendar without naming an end date now works on PostgreSQL, where the open-ended range was too large for the database to accept and the search failed outright. [Commit](https://github.com/open-webui/open-webui/commit/9550731cc17759f6862595b8cd849ae48695b5c1), [#27717](https://github.com/open-webui/open-webui/issues/27717)
|
||||
|
|
|
|||
|
|
@ -34,10 +34,21 @@ Examples of unacceptable behavior include:
|
|||
- **Spamming and promotional exploitation.** Sharing irrelevant product promotions or self-promotion in the community is not allowed unless it directly contributes value to the discussion.
|
||||
- Posting low-effort, hard to read, essay-length AI generated comments or other forms of low-quality, hard to parse content that puts the burden of understanding on the reader.
|
||||
|
||||
### How We Develop the Project
|
||||
|
||||
Development is led by the maintainers, and code pull requests are reserved for work we explicitly request or exceptional contributions we choose to consider at our discretion. We use actionable reports and concrete use cases to understand problems, then evaluate, revise, and implement the appropriate approach internally. We assess each change against the project's architecture, existing behavior, quality standards, and future direction before settling on an implementation. Resolving a reported problem requires that broader context, and a working external patch usually requires substantial rewriting to meet the project's standards. Reviewing the patch, explaining the required changes, and coordinating successive revisions usually takes more effort than developing the solution internally. Fragmented commit histories, branches that have not been rebased, unresolved conflicts, and lengthy or unverified AI-generated comments add cleanup and discussion that delay the underlying work. Maintainers remain responsible for testing, documenting, supporting, and maintaining every accepted change, so we choose the approach based on the whole product and its ongoing maintenance. Clear reports, reproduction details, and relevant context give us what we need to make those decisions and develop the solution. A polished implementation, clean commit history, or completed checklist does not establish an exception to this process, and opening an issue or discussion is not an invitation to submit a PR. Wait for an explicit maintainer request before investing in a PR; unsolicited submissions are generally closed without review, and requested PRs remain subject to maintainer judgment.
|
||||
|
||||
### Feedback and Community Engagement
|
||||
|
||||
- **Constructive feedback is encouraged, but hostile or entitled behavior will result in immediate action.** If you disagree with elements of the project, we encourage you to offer meaningful improvements or fork the project if necessary. Healthy discussions and technical disagreements are welcome only when handled with professionalism.
|
||||
- **Respect contributors' time and efforts.** No one is entitled to personalized or on-demand assistance. This is a community built on collaboration and shared effort; demanding or demeaning behavior undermines that trust and will not be allowed.
|
||||
Participation should help maintainers understand a concrete problem while respecting the project's priorities and available capacity. Please follow the [issue templates](.github/ISSUE_TEMPLATE) and [pull request policy](.github/pull_request_template.md) before submitting anything.
|
||||
|
||||
- **Make reports actionable.** Search existing issues and discussions, check the latest version and whether the problem is already addressed on `dev`, and use the appropriate template. Bug reports should describe a reproducible problem, the affected workflow, expected and actual behavior, and relevant evidence. Feature requests should explain the user-facing need; broader product, UX, architecture, or maintenance questions belong in Discussions. Report security concerns privately through the [security reporting process](https://github.com/open-webui/open-webui/security).
|
||||
- **Share the problem before investing in code.** Start with an actionable issue or discussion and leave implementation planning to the maintainers. An issue or discussion alone is not an invitation to submit a PR. Please wait for an explicit request before opening one; any exception is at the maintainers' discretion. Implementation notes, local diffs, or patches may be shared as reference in the relevant issue or discussion.
|
||||
- **Respect maintainers' discretion.** Submitting an issue, proposal, or pull request does not create an obligation to respond, review, implement, or merge it. Maintainers set the project's direction and defer or close submissions based on scope, quality, maintenance cost, or available capacity. Unsolicited pull requests are generally closed without review.
|
||||
- **Keep discussion focused and concise.** Provide new information when it helps evaluate the problem. Repeated bumps, duplicate submissions, unsolicited direct messages seeking attention, or pressure for timelines place an unnecessary burden on contributors.
|
||||
- **Respect decisions and boundaries.** Technical disagreement is welcome when expressed professionally. Reopening a declined request or continuing to press for a different outcome without new, relevant information is not constructive. You are free to explore a different direction in your own fork.
|
||||
|
||||
Participants are expected to respect maintainers' decisions and the contribution process. Harassment, hostility, or repeated disregard for these boundaries will result in enforcement under this Code of Conduct.
|
||||
|
||||
### Zero Tolerance: No Warnings, Immediate Action
|
||||
|
||||
|
|
|
|||
58
Dockerfile
58
Dockerfile
|
|
@ -26,6 +26,9 @@ ARG GID=0
|
|||
######## WebUI frontend ########
|
||||
FROM --platform=$BUILDPLATFORM node:22-alpine3.20 AS build
|
||||
ARG BUILD_HASH
|
||||
ARG USE_SLIM
|
||||
ARG UID
|
||||
ARG GID
|
||||
|
||||
# Set Node.js options (heap limit Allocation failed - JavaScript heap out of memory)
|
||||
# ENV NODE_OPTIONS="--max-old-space-size=4096"
|
||||
|
|
@ -40,7 +43,14 @@ RUN npm ci --force
|
|||
|
||||
COPY . .
|
||||
ENV APP_BUILD_HASH=${BUILD_HASH}
|
||||
RUN npm run build
|
||||
RUN npm run build && \
|
||||
if [ "$USE_SLIM" = "true" ]; then find build -type f -name '*.map' -delete; fi
|
||||
|
||||
# Prepare backend ownership before the final copy so static assets occupy one layer.
|
||||
# Group 0 write access lets arbitrary OpenShift UIDs update these assets at startup.
|
||||
RUN chown -R $UID:$GID /app/backend && \
|
||||
chgrp -R 0 /app/backend/open_webui/static && \
|
||||
chmod -R g=u /app/backend/open_webui/static
|
||||
|
||||
######## WebUI backend ########
|
||||
FROM python:3.11-slim-bookworm AS base
|
||||
|
|
@ -123,24 +133,33 @@ RUN echo -n 00000000-0000-0000-0000-000000000000 > $HOME/.cache/chroma/telemetry
|
|||
# Make sure the user has access to the app and root directory
|
||||
RUN chown -R $UID:$GID /app $HOME
|
||||
|
||||
# Install common system dependencies
|
||||
# Slim cannot bundle a local model server or GPU runtime.
|
||||
RUN if [ "$USE_SLIM" = "true" ] && { [ "$USE_CUDA" = "true" ] || [ "$USE_OLLAMA" = "true" ]; }; then \
|
||||
echo "USE_SLIM cannot be combined with USE_CUDA or USE_OLLAMA" >&2; exit 1; fi
|
||||
|
||||
# Keep the slim runtime free of local document/audio processing tools.
|
||||
# Git-based tool requirements require the standard image.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
git build-essential pandoc gcc curl jq ca-certificates \
|
||||
libmariadb-dev \
|
||||
python3-dev \
|
||||
ffmpeg libsm6 libxext6 zstd \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
curl jq ca-certificates \
|
||||
&& if [ "$USE_SLIM" != "true" ]; then \
|
||||
apt-get install -y --no-install-recommends \
|
||||
git build-essential pandoc gcc libmariadb-dev ffmpeg libsm6 libxext6; \
|
||||
fi && if [ "$USE_OLLAMA" = "true" ]; then \
|
||||
apt-get install -y --no-install-recommends zstd; \
|
||||
fi && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# install python dependencies
|
||||
COPY --chown=$UID:$GID ./backend/requirements.txt ./requirements.txt
|
||||
COPY --chown=$UID:$GID ./backend/requirements*.txt ./
|
||||
|
||||
# Set UV_LINK_MODE to copy to prevent 0-byte file corruption in QEMU arm64 cross-builds
|
||||
ENV UV_LINK_MODE=copy
|
||||
|
||||
RUN set -e; \
|
||||
pip3 install --no-cache-dir uv; \
|
||||
if [ "$USE_CUDA" = "true" ]; then \
|
||||
RUN --mount=from=ghcr.io/astral-sh/uv:0.12.10,source=/uv,target=/bin/uv \
|
||||
set -e; \
|
||||
if [ "$USE_SLIM" = "true" ]; then \
|
||||
uv pip install --system -r requirements-slim.txt --no-cache-dir; \
|
||||
elif [ "$USE_CUDA" = "true" ]; then \
|
||||
# If you use CUDA the whisper and embedding model will be downloaded on first use
|
||||
# fix: pin torch<=2.9.1 - torch 2.10.0 aarch64 wheels cause SIGILL on ARM devices (RPi 4 Cortex-A72) #21349
|
||||
pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/$USE_CUDA_DOCKER_VER --no-cache-dir; \
|
||||
|
|
@ -149,7 +168,6 @@ RUN set -e; \
|
|||
python -c "import os; from sentence_transformers import SentenceTransformer; SentenceTransformer(os.environ.get('AUXILIARY_EMBEDDING_MODEL', 'TaylorAI/bge-micro-v2'), device='cpu')"; \
|
||||
python -c "import os; from faster_whisper import WhisperModel; WhisperModel(os.environ['WHISPER_MODEL'], device='cpu', compute_type='int8', download_root=os.environ['WHISPER_MODEL_DIR'])"; \
|
||||
python -c "import os; import tiktoken; tiktoken.get_encoding(os.environ['TIKTOKEN_ENCODING_NAME'])"; \
|
||||
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
|
||||
else \
|
||||
pip3 install 'torch<=2.9.1' torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu --no-cache-dir; \
|
||||
uv pip install --system -r requirements.txt --no-cache-dir; \
|
||||
|
|
@ -158,7 +176,6 @@ RUN set -e; \
|
|||
python -c "import os; from sentence_transformers import SentenceTransformer; SentenceTransformer(os.environ.get('AUXILIARY_EMBEDDING_MODEL', 'TaylorAI/bge-micro-v2'), device='cpu')"; \
|
||||
python -c "import os; from faster_whisper import WhisperModel; WhisperModel(os.environ['WHISPER_MODEL'], device='cpu', compute_type='int8', download_root=os.environ['WHISPER_MODEL_DIR'])"; \
|
||||
python -c "import os; import tiktoken; tiktoken.get_encoding(os.environ['TIKTOKEN_ENCODING_NAME'])"; \
|
||||
python -c "import nltk; nltk.download('punkt_tab', download_dir='/usr/local/share/nltk_data')"; \
|
||||
fi; \
|
||||
fi; \
|
||||
mkdir -p /app/backend/data; chown -R $UID:$GID /app/backend/data/; \
|
||||
|
|
@ -187,19 +204,8 @@ COPY --chown=$UID:$GID --from=build /app/build /app/build
|
|||
COPY --chown=$UID:$GID --from=build /app/CHANGELOG.md /app/CHANGELOG.md
|
||||
COPY --chown=$UID:$GID --from=build /app/package.json /app/package.json
|
||||
|
||||
# copy backend files
|
||||
COPY --chown=$UID:$GID ./backend .
|
||||
|
||||
# The backend rewrites its bundled static assets (favicons, splash, manifest,
|
||||
# loader.js, ...) under open_webui/static at startup. Make that directory
|
||||
# writable by an arbitrary UID -- which under OpenShift's restricted SCC is
|
||||
# always a member of GID 0 -- so those writes don't fail with EACCES and crash
|
||||
# the boot log with "[Errno 13] Permission denied". `chmod -R g=u` mirrors the
|
||||
# owner bits onto the group (the Red Hat arbitrary-UID idiom). This is applied
|
||||
# unconditionally because it targets a directory the app writes on every start;
|
||||
# the broader, opt-in USE_PERMISSION_HARDENING below covers the rest of /app.
|
||||
RUN chgrp -R 0 /app/backend/open_webui/static && \
|
||||
chmod -R g=u /app/backend/open_webui/static
|
||||
# copy backend files with the ownership and static permissions prepared above
|
||||
COPY --from=build /app/backend .
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,8 @@ For more information, be sure to check out our [Open WebUI Documentation](https:
|
|||
|
||||
- 🤖 **Models & Agents**: Wrap any base model with custom instructions, tools, and knowledge to build specialized agents. Supports dynamic variables, per-user/group access control, and community preset imports via [Open WebUI Community](https://openwebui.com/).
|
||||
|
||||
- ⚡ **Agentic Execution with [Open Terminal](https://github.com/open-webui/open-terminal)**: Give your agents a terminal and filesystem to carry out multi-step tasks. Let them analyze data, run scripts, fix errors, and produce files directly in chat. Scale to teams with **[Terminals (Enterprise)](https://github.com/open-webui/terminals)** for per-user isolated environments, resource limits, and automatic lifecycle management.
|
||||
|
||||
- 📝 **Notes**: A dedicated workspace for content outside conversations. Draft with a rich editor, use AI to rewrite selected text, and attach notes to any chat for full-context injection.
|
||||
|
||||
- 📢 **Channels**: Real-time shared spaces where your team and AI models collaborate in one timeline. Tag models to draft or critique, with threads, reactions, pins, and access control.
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ from authlib.integrations.starlette_client import OAuth
|
|||
from pydantic import BaseModel
|
||||
|
||||
from open_webui.env import (
|
||||
USE_SLIM,
|
||||
DATA_DIR,
|
||||
DATABASE_URL,
|
||||
ENABLE_ADMIN_CHAT_ACCESS,
|
||||
|
|
@ -227,6 +228,7 @@ if CUSTOM_NAME:
|
|||
####################################
|
||||
|
||||
ENABLE_DIRECT_CONNECTIONS = os.getenv('ENABLE_DIRECT_CONNECTIONS', 'False').lower() == 'true'
|
||||
ENABLE_DIRECT_INTEGRATIONS = os.getenv('ENABLE_DIRECT_INTEGRATIONS', 'False').lower() == 'true'
|
||||
|
||||
####################################
|
||||
# OLLAMA_BASE_URL
|
||||
|
|
@ -498,12 +500,12 @@ CODE_INTERPRETER_PYODIDE_PROMPT = """
|
|||
# Vector Database
|
||||
####################################
|
||||
|
||||
VECTOR_DB = os.getenv('VECTOR_DB', 'chroma')
|
||||
VECTOR_DB = os.getenv('VECTOR_DB', 'pgvector' if USE_SLIM else 'chroma')
|
||||
|
||||
# Chroma
|
||||
CHROMA_DATA_PATH = f'{DATA_DIR}/vector_db'
|
||||
|
||||
if VECTOR_DB == 'chroma':
|
||||
if VECTOR_DB == 'chroma' and not USE_SLIM:
|
||||
import chromadb
|
||||
|
||||
CHROMA_TENANT = os.getenv('CHROMA_TENANT', chromadb.DEFAULT_TENANT)
|
||||
|
|
@ -644,7 +646,7 @@ SSL_ASSERT_FINGERPRINT = os.getenv('SSL_ASSERT_FINGERPRINT', None)
|
|||
ELASTICSEARCH_INDEX_PREFIX = os.getenv('ELASTICSEARCH_INDEX_PREFIX', 'open_webui_collections')
|
||||
# Pgvector
|
||||
PGVECTOR_DB_URL = os.getenv('PGVECTOR_DB_URL', DATABASE_URL)
|
||||
if VECTOR_DB == 'pgvector' and not PGVECTOR_DB_URL.startswith('postgres'):
|
||||
if not USE_SLIM and VECTOR_DB == 'pgvector' and not PGVECTOR_DB_URL.startswith('postgres'):
|
||||
raise ValueError(
|
||||
'Pgvector requires setting PGVECTOR_DB_URL or using Postgres with vector extension as the primary database.'
|
||||
)
|
||||
|
|
@ -739,6 +741,10 @@ else:
|
|||
except Exception:
|
||||
PGVECTOR_IVFFLAT_LISTS = 100
|
||||
|
||||
PGVECTOR_ITERATIVE_SCAN = os.getenv('PGVECTOR_ITERATIVE_SCAN', 'relaxed_order').strip().lower()
|
||||
if PGVECTOR_ITERATIVE_SCAN not in ('off', 'relaxed_order', 'strict_order'):
|
||||
PGVECTOR_ITERATIVE_SCAN = 'relaxed_order'
|
||||
|
||||
# openGauss
|
||||
OPENGAUSS_DB_URL = os.getenv('OPENGAUSS_DB_URL', DATABASE_URL)
|
||||
|
||||
|
|
@ -805,7 +811,7 @@ ORACLE_DB_POOL_MAX = int(os.getenv('ORACLE_DB_POOL_MAX', 10))
|
|||
ORACLE_DB_POOL_INCREMENT = int(os.getenv('ORACLE_DB_POOL_INCREMENT', 1))
|
||||
|
||||
|
||||
if VECTOR_DB == 'oracle23ai':
|
||||
if not USE_SLIM and VECTOR_DB == 'oracle23ai':
|
||||
if not ORACLE_DB_USER or not ORACLE_DB_PASSWORD or not ORACLE_DB_DSN:
|
||||
raise ValueError('Oracle23ai requires setting ORACLE_DB_USER, ORACLE_DB_PASSWORD, and ORACLE_DB_DSN.')
|
||||
if ORACLE_DB_USE_WALLET and (not ORACLE_WALLET_DIR or not ORACLE_WALLET_PASSWORD):
|
||||
|
|
@ -1270,6 +1276,9 @@ AZURE_AI_SEARCH_ENDPOINT = os.getenv('AZURE_AI_SEARCH_ENDPOINT', '')
|
|||
AZURE_AI_SEARCH_INDEX_NAME = os.getenv('AZURE_AI_SEARCH_INDEX_NAME', '')
|
||||
|
||||
EXA_API_KEY = os.getenv('EXA_API_KEY', '')
|
||||
EXA_MAX_CONTENT_LENGTH = int(os.environ['EXA_MAX_CONTENT_LENGTH']) if os.getenv('EXA_MAX_CONTENT_LENGTH') else None
|
||||
if EXA_MAX_CONTENT_LENGTH is not None and EXA_MAX_CONTENT_LENGTH <= 0:
|
||||
raise ValueError('EXA_MAX_CONTENT_LENGTH must be a positive integer or unset')
|
||||
|
||||
PERPLEXITY_API_KEY = os.getenv('PERPLEXITY_API_KEY', '')
|
||||
|
||||
|
|
@ -1293,6 +1302,12 @@ TAVILY_API_KEY = os.getenv('TAVILY_API_KEY', '')
|
|||
|
||||
TAVILY_EXTRACT_DEPTH = os.getenv('TAVILY_EXTRACT_DEPTH', 'basic')
|
||||
|
||||
STAAN_API_KEY = os.getenv('STAAN_API_KEY', '')
|
||||
|
||||
STAAN_MARKET = os.getenv('STAAN_MARKET', 'en-us')
|
||||
|
||||
STAAN_MAX_SNIPPETS = int(os.getenv('STAAN_MAX_SNIPPETS', '0'))
|
||||
|
||||
PLAYWRIGHT_WS_URL = os.getenv('PLAYWRIGHT_WS_URL', '')
|
||||
|
||||
PLAYWRIGHT_TIMEOUT = int(os.getenv('PLAYWRIGHT_TIMEOUT', '10000'))
|
||||
|
|
@ -1660,43 +1675,14 @@ DEFAULT_MODELS = os.getenv('DEFAULT_MODELS', None)
|
|||
|
||||
DEFAULT_PINNED_MODELS = os.getenv('DEFAULT_PINNED_MODELS', None)
|
||||
|
||||
# None uses the frontend's localized defaults; an empty list disables suggestions.
|
||||
try:
|
||||
default_prompt_suggestions = JSONCodec.loads(os.getenv('DEFAULT_PROMPT_SUGGESTIONS', '[]'))
|
||||
DEFAULT_PROMPT_SUGGESTIONS = JSONCodec.loads(os.getenv('DEFAULT_PROMPT_SUGGESTIONS', 'null'))
|
||||
except Exception as e:
|
||||
log.exception(f'Error loading DEFAULT_PROMPT_SUGGESTIONS: {e}')
|
||||
default_prompt_suggestions = []
|
||||
if default_prompt_suggestions == []:
|
||||
default_prompt_suggestions = [
|
||||
{
|
||||
'title': ['Help me study', 'vocabulary for a college entrance exam'],
|
||||
'content': "Help me study vocabulary: write a sentence for me to fill in the blank, and I'll try to pick the correct option.",
|
||||
},
|
||||
{
|
||||
'title': ['Give me ideas', "for what to do with my kids' art"],
|
||||
'content': "What are 5 creative things I could do with my kids' art? I don't want to throw them away, but it's also so much clutter.",
|
||||
},
|
||||
{
|
||||
'title': ['Tell me a fun fact', 'about the Roman Empire'],
|
||||
'content': 'Tell me a random fun fact about the Roman Empire',
|
||||
},
|
||||
{
|
||||
'title': ['Show me a code snippet', "of a website's sticky header"],
|
||||
'content': "Show me a code snippet of a website's sticky header in CSS and JavaScript.",
|
||||
},
|
||||
{
|
||||
'title': [
|
||||
'Explain options trading',
|
||||
"if I'm familiar with buying and selling stocks",
|
||||
],
|
||||
'content': "Explain options trading in simple terms if I'm familiar with buying and selling stocks.",
|
||||
},
|
||||
{
|
||||
'title': ['Overcome procrastination', 'give me tips'],
|
||||
'content': 'Could you start by asking me about instances when I procrastinate the most and then give me some suggestions to overcome it?',
|
||||
},
|
||||
]
|
||||
DEFAULT_PROMPT_SUGGESTIONS = None
|
||||
|
||||
DEFAULT_PROMPT_SUGGESTIONS = default_prompt_suggestions
|
||||
DEFAULT_PROMPT_SUGGESTIONS_I18N = {}
|
||||
|
||||
try:
|
||||
model_order_list = JSONCodec.loads(os.getenv('MODEL_ORDER_LIST', '[]'))
|
||||
|
|
@ -2162,6 +2148,7 @@ else:
|
|||
|
||||
|
||||
class BannerModel(BaseModel):
|
||||
i18n: dict[str, dict[str, str]] | None = None
|
||||
id: str
|
||||
type: str
|
||||
title: str | None = None
|
||||
|
|
@ -2834,6 +2821,7 @@ LDAP_ATTRIBUTE_FOR_GROUPS = os.getenv('LDAP_ATTRIBUTE_FOR_GROUPS', 'memberOf')
|
|||
|
||||
DEFAULT_CONFIG = {
|
||||
'direct.enable': ENABLE_DIRECT_CONNECTIONS,
|
||||
'direct.integrations.enable': ENABLE_DIRECT_INTEGRATIONS,
|
||||
'ollama.enable': ENABLE_OLLAMA_API,
|
||||
'ollama.base_urls': OLLAMA_BASE_URLS,
|
||||
'ollama.api_configs': OLLAMA_API_CONFIGS,
|
||||
|
|
@ -2998,6 +2986,7 @@ DEFAULT_CONFIG = {
|
|||
'web.search.azure_ai_search_endpoint': AZURE_AI_SEARCH_ENDPOINT,
|
||||
'web.search.azure_ai_search_index_name': AZURE_AI_SEARCH_INDEX_NAME,
|
||||
'web.search.exa_api_key': EXA_API_KEY,
|
||||
'web.search.exa_max_content_length': EXA_MAX_CONTENT_LENGTH,
|
||||
'web.search.perplexity_api_key': PERPLEXITY_API_KEY,
|
||||
'web.search.perplexity_model': PERPLEXITY_MODEL,
|
||||
'web.search.perplexity_search_context_usage': PERPLEXITY_SEARCH_CONTEXT_USAGE,
|
||||
|
|
@ -3009,6 +2998,9 @@ DEFAULT_CONFIG = {
|
|||
'web.search.sougou_api_sk': SOUGOU_API_SK,
|
||||
'web.search.tavily_api_key': TAVILY_API_KEY,
|
||||
'web.search.tavily_extract_depth': TAVILY_EXTRACT_DEPTH,
|
||||
'web.search.staan_api_key': STAAN_API_KEY,
|
||||
'web.search.staan_market': STAAN_MARKET,
|
||||
'web.search.staan_max_snippets': STAAN_MAX_SNIPPETS,
|
||||
'web.loader.playwright_ws_url': PLAYWRIGHT_WS_URL,
|
||||
'web.loader.playwright_timeout': PLAYWRIGHT_TIMEOUT,
|
||||
'web.loader.firecrawl_api_key': FIRECRAWL_API_KEY,
|
||||
|
|
@ -3095,7 +3087,9 @@ DEFAULT_CONFIG = {
|
|||
'ui.default_models': DEFAULT_MODELS,
|
||||
'ui.default_pinned_models': DEFAULT_PINNED_MODELS,
|
||||
'ui.default_interface_settings': DEFAULT_INTERFACE_SETTINGS,
|
||||
'ui.i18n': {},
|
||||
'ui.prompt_suggestions': DEFAULT_PROMPT_SUGGESTIONS,
|
||||
'ui.prompt_suggestions_i18n': DEFAULT_PROMPT_SUGGESTIONS_I18N,
|
||||
'ui.model_order_list': MODEL_ORDER_LIST,
|
||||
'models.default_metadata': DEFAULT_MODEL_METADATA,
|
||||
'models.default_params': DEFAULT_MODEL_PARAMS,
|
||||
|
|
|
|||
|
|
@ -7,7 +7,9 @@ import pkgutil
|
|||
import re
|
||||
import shutil
|
||||
import sys
|
||||
import threading
|
||||
import traceback
|
||||
from contextlib import nullcontext
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
from uuid import uuid4
|
||||
|
|
@ -40,12 +42,13 @@ except ImportError:
|
|||
print('dotenv not installed, skipping...')
|
||||
|
||||
DOCKER = os.getenv('DOCKER', 'False').lower() == 'true'
|
||||
USE_SLIM = os.getenv('USE_SLIM_DOCKER', 'False').lower() == 'true'
|
||||
|
||||
USE_CUDA = os.getenv('USE_CUDA_DOCKER', 'false')
|
||||
DEVICE_TYPE = 'cpu'
|
||||
_cuda_error: Optional[str] = None
|
||||
|
||||
if USE_CUDA.lower() == 'true':
|
||||
if not USE_SLIM and USE_CUDA.lower() == 'true':
|
||||
try:
|
||||
import torch # noqa: E402
|
||||
|
||||
|
|
@ -57,7 +60,7 @@ if USE_CUDA.lower() == 'true':
|
|||
os.environ['USE_CUDA_DOCKER'] = 'false'
|
||||
USE_CUDA = 'false'
|
||||
|
||||
if sys.platform == 'darwin' and DEVICE_TYPE == 'cpu':
|
||||
if not USE_SLIM and sys.platform == 'darwin' and DEVICE_TYPE == 'cpu':
|
||||
try:
|
||||
import torch # noqa: E402
|
||||
|
||||
|
|
@ -66,6 +69,9 @@ if sys.platform == 'darwin' and DEVICE_TYPE == 'cpu':
|
|||
except Exception:
|
||||
pass
|
||||
|
||||
# Torch MPS inference is not thread-safe and a concurrent call kills the whole process.
|
||||
MPS_INFERENCE_LOCK = threading.Lock() if DEVICE_TYPE == 'mps' else nullcontext()
|
||||
|
||||
####################################
|
||||
# LOGGING
|
||||
####################################
|
||||
|
|
@ -245,8 +251,6 @@ if FROM_INIT_PY:
|
|||
|
||||
STATIC_DIR = Path(os.getenv('STATIC_DIR', OPEN_WEBUI_DIR / 'static'))
|
||||
|
||||
FONTS_DIR = Path(os.getenv('FONTS_DIR', OPEN_WEBUI_DIR / 'static' / 'fonts'))
|
||||
|
||||
FRONTEND_BUILD_DIR = Path(os.getenv('FRONTEND_BUILD_DIR', BASE_DIR / 'build')).resolve()
|
||||
|
||||
if FROM_INIT_PY:
|
||||
|
|
@ -367,6 +371,9 @@ ENABLE_QUERIES_CACHE = os.getenv('ENABLE_QUERIES_CACHE', 'False').lower() == 'tr
|
|||
ENABLE_ADMIN_CHAT_ACCESS = os.getenv('ENABLE_ADMIN_CHAT_ACCESS', 'True').lower() == 'true'
|
||||
RAG_SYSTEM_CONTEXT = os.getenv('RAG_SYSTEM_CONTEXT', 'False').lower() == 'true'
|
||||
|
||||
# Empty by default: chunk metadata also holds internal bookkeeping (file hashes, collection names, scores).
|
||||
RAG_SOURCE_METADATA_KEYS = [key.strip() for key in os.getenv('RAG_SOURCE_METADATA_KEYS', '').split(',') if key.strip()]
|
||||
|
||||
####################################
|
||||
# REDIS
|
||||
####################################
|
||||
|
|
@ -381,6 +388,14 @@ try:
|
|||
except ValueError:
|
||||
REDIS_RESPONSE_STREAM_TTL = 3600
|
||||
|
||||
# Seconds a task survives without a heartbeat. 0 disables expiry.
|
||||
try:
|
||||
REDIS_TASK_TTL = int(os.getenv('REDIS_TASK_TTL', '300'))
|
||||
if REDIS_TASK_TTL != 0 and REDIS_TASK_TTL < 60:
|
||||
REDIS_TASK_TTL = 300
|
||||
except ValueError:
|
||||
REDIS_TASK_TTL = 300
|
||||
|
||||
REDIS_SENTINEL_HOSTS = os.getenv('REDIS_SENTINEL_HOSTS', '')
|
||||
REDIS_SENTINEL_PORT = os.getenv('REDIS_SENTINEL_PORT', '26379')
|
||||
|
||||
|
|
@ -839,7 +854,7 @@ MINERU_MAX_MARKDOWN_BYTES = (
|
|||
# When enabled, skips pydub-based preprocessing (format conversion, compression,
|
||||
# and chunked splitting) before sending files to processing engines. Useful when
|
||||
# the upstream provider handles these steps or when ffmpeg is unavailable.
|
||||
BYPASS_PYDUB_PREPROCESSING = os.getenv('BYPASS_PYDUB_PREPROCESSING', 'False').lower() == 'true'
|
||||
BYPASS_PYDUB_PREPROCESSING = USE_SLIM or os.getenv('BYPASS_PYDUB_PREPROCESSING', 'False').lower() == 'true'
|
||||
|
||||
# When disabled (default), the OpenAI catch-all proxy endpoint (/{path:path})
|
||||
# is blocked. Enable only if you need direct passthrough to upstream OpenAI-
|
||||
|
|
@ -979,6 +994,7 @@ FORWARD_USER_INFO_HEADER_USER_NAME = os.getenv('FORWARD_USER_INFO_HEADER_USER_NA
|
|||
FORWARD_USER_INFO_HEADER_USER_ID = os.getenv('FORWARD_USER_INFO_HEADER_USER_ID', 'X-OpenWebUI-User-Id')
|
||||
FORWARD_USER_INFO_HEADER_USER_EMAIL = os.getenv('FORWARD_USER_INFO_HEADER_USER_EMAIL', 'X-OpenWebUI-User-Email')
|
||||
FORWARD_USER_INFO_HEADER_USER_ROLE = os.getenv('FORWARD_USER_INFO_HEADER_USER_ROLE', 'X-OpenWebUI-User-Role')
|
||||
FORWARD_USER_INFO_HEADER_AUTH_TYPE = os.getenv('FORWARD_USER_INFO_HEADER_AUTH_TYPE', 'X-OpenWebUI-Auth-Type')
|
||||
FORWARD_SESSION_INFO_HEADER_MESSAGE_ID = os.getenv('FORWARD_SESSION_INFO_HEADER_MESSAGE_ID', 'X-OpenWebUI-Message-Id')
|
||||
FORWARD_SESSION_INFO_HEADER_CHAT_ID = os.getenv('FORWARD_SESSION_INFO_HEADER_CHAT_ID', 'X-OpenWebUI-Chat-Id')
|
||||
|
||||
|
|
@ -1037,6 +1053,13 @@ ENABLE_CHAT_RESPONSE_BASE64_IMAGE_URL_CONVERSION = (
|
|||
)
|
||||
ENABLE_API_OUTLET_FILTERS = os.getenv('ENABLE_API_OUTLET_FILTERS', 'True').lower() == 'true'
|
||||
|
||||
# Opt in to CPython's in-place string append optimization for streamed responses.
|
||||
# Off by default for a staged rollout. Only a host already out of memory can lose
|
||||
# text here; the default path (a full copy per chunk) raises there too.
|
||||
ENABLE_CHAT_RESPONSE_STREAM_INPLACE_APPEND = (
|
||||
os.getenv('ENABLE_CHAT_RESPONSE_STREAM_INPLACE_APPEND', 'False').lower() == 'true'
|
||||
)
|
||||
|
||||
# When enabled, uses a hardcoded extension-to-MIME dictionary as a last-resort
|
||||
# fallback when both mimetypes.guess_type() and file.meta.content_type fail to
|
||||
# determine the content type. This can help on minimal container images (e.g.
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ from open_webui.env import (
|
|||
DATABASE_URL,
|
||||
ENABLE_DB_MIGRATIONS,
|
||||
OPEN_WEBUI_DIR,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from sqlalchemy import Dialect, MetaData, create_engine, event, types
|
||||
|
|
@ -142,6 +143,17 @@ class JSONField(types.TypeDecorator): # TEXT-backed JSON storage
|
|||
return JSONField(length=self.impl.length)
|
||||
|
||||
|
||||
if USE_SLIM:
|
||||
if make_url(DATABASE_URL).get_backend_name() not in ('sqlite', 'postgresql', 'postgres'):
|
||||
raise ValueError(
|
||||
'Slim requires SQLite or PostgreSQL for DATABASE_URL. Use the standard image for other databases.'
|
||||
)
|
||||
if DATABASE_ENABLE_IAM_TOKEN_AUTH:
|
||||
raise ValueError(
|
||||
'AWS RDS IAM authentication requires the standard image. Slim supports PostgreSQL database credentials.'
|
||||
)
|
||||
|
||||
|
||||
# Normalize SSL params from the URL once; the sync engine needs them
|
||||
# reattached in canonical libpq form for psycopg2.
|
||||
_url_without_ssl, _ssl_dict = extract_ssl_params_from_url(DATABASE_URL)
|
||||
|
|
|
|||
|
|
@ -74,7 +74,9 @@ from open_webui.config import (
|
|||
seed_registered_defaults,
|
||||
)
|
||||
from open_webui.constants import ERROR_MESSAGES, TASKS
|
||||
from open_webui.utils.recurrence import RecurrenceEvaluationTimeout
|
||||
from open_webui.env import (
|
||||
USE_SLIM,
|
||||
AIOHTTP_CLIENT_SESSION_SSL,
|
||||
AUDIT_EXCLUDED_PATHS,
|
||||
AUDIT_INCLUDED_PATHS,
|
||||
|
|
@ -106,12 +108,14 @@ from open_webui.env import (
|
|||
MAX_BODY_LOG_SIZE,
|
||||
# Redis
|
||||
REDIS_KEY_PREFIX,
|
||||
REDIS_TASK_TTL,
|
||||
REDIS_URL,
|
||||
RESET_CONFIG_ON_START,
|
||||
SAFE_MODE,
|
||||
SCIM_TOKEN,
|
||||
VERSION,
|
||||
WEBSOCKET_HEARTBEAT_INTERVAL,
|
||||
WEBSOCKET_MANAGER,
|
||||
# Admin Account Runtime Creation
|
||||
WEBUI_ADMIN_EMAIL,
|
||||
WEBUI_ADMIN_NAME,
|
||||
|
|
@ -188,6 +192,7 @@ from open_webui.socket.main import (
|
|||
get_user_id_from_session_pool,
|
||||
periodic_session_pool_cleanup,
|
||||
periodic_usage_pool_cleanup,
|
||||
redis_event_listener,
|
||||
)
|
||||
from open_webui.socket.main import (
|
||||
app as socket_app,
|
||||
|
|
@ -199,6 +204,7 @@ from open_webui.tasks import (
|
|||
list_task_ids_by_item_id,
|
||||
list_tasks,
|
||||
redis_task_command_listener,
|
||||
redis_task_heartbeat,
|
||||
stop_item_tasks,
|
||||
stop_task,
|
||||
) # Import from tasks.py
|
||||
|
|
@ -385,6 +391,11 @@ async def lifespan(app: FastAPI):
|
|||
|
||||
if app.state.redis is not None:
|
||||
app.state.redis_task_command_listener = asyncio.create_task(redis_task_command_listener(app))
|
||||
if REDIS_TASK_TTL > 0:
|
||||
app.state.redis_task_heartbeat = asyncio.create_task(redis_task_heartbeat(app))
|
||||
|
||||
if WEBSOCKET_MANAGER == 'redis':
|
||||
app.state.redis_event_listener = asyncio.create_task(redis_event_listener())
|
||||
|
||||
app.state.periodic_usage_pool_cleanup = asyncio.create_task(periodic_usage_pool_cleanup())
|
||||
app.state.periodic_session_pool_cleanup = asyncio.create_task(periodic_session_pool_cleanup())
|
||||
|
|
@ -472,6 +483,12 @@ async def lifespan(app: FastAPI):
|
|||
if hasattr(app.state, 'redis_task_command_listener'):
|
||||
app.state.redis_task_command_listener.cancel()
|
||||
|
||||
if hasattr(app.state, 'redis_task_heartbeat'):
|
||||
app.state.redis_task_heartbeat.cancel()
|
||||
|
||||
if hasattr(app.state, 'redis_event_listener'):
|
||||
app.state.redis_event_listener.cancel()
|
||||
|
||||
app.state.periodic_usage_pool_cleanup.cancel()
|
||||
app.state.periodic_session_pool_cleanup.cancel()
|
||||
app.state.scheduler_worker_loop.cancel()
|
||||
|
|
@ -494,6 +511,12 @@ app = FastAPI(
|
|||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(RecurrenceEvaluationTimeout)
|
||||
async def recurrence_timeout_handler(request: Request, exc: RecurrenceEvaluationTimeout):
|
||||
return JSONResponse(status_code=400, content={'detail': str(exc)})
|
||||
|
||||
|
||||
# Used by readiness checks to gate traffic until startup work is done.
|
||||
app.state.startup_complete = False
|
||||
|
||||
|
|
@ -1463,8 +1486,8 @@ async def chat_completion(
|
|||
async def run_initial_title_generation():
|
||||
try:
|
||||
await background_tasks_handler(title_ctx)
|
||||
except Exception as e:
|
||||
log.debug('Error generating initial chat title: %s', e)
|
||||
except Exception:
|
||||
log.exception('Error generating initial chat title')
|
||||
|
||||
asyncio.create_task(run_initial_title_generation())
|
||||
else:
|
||||
|
|
@ -1623,6 +1646,9 @@ async def chat_completion(
|
|||
|
||||
async def process_chat(request, form_data, user, metadata, model, tasks=None):
|
||||
try:
|
||||
ctx = None
|
||||
if metadata.get('assistant_message_id'):
|
||||
ctx = await build_chat_response_context(request, form_data, user, model, metadata, tasks, [])
|
||||
form_data, metadata, events = await process_chat_payload(request, form_data, user, metadata, model)
|
||||
|
||||
if await drain_approved_tool_calls(request, form_data, user, model, metadata):
|
||||
|
|
@ -1633,12 +1659,15 @@ async def chat_completion(
|
|||
# When the upstream provider returns an error (e.g. HTTP 400
|
||||
# content-filter, quota exceeded), generate_chat_completion
|
||||
# returns a JSONResponse instead of raising. Detect this and
|
||||
# raise so the except-block below emits chat:message:error +
|
||||
# chat:tasks:cancel, unblocking the frontend.
|
||||
# raise so the except-block below emits a terminal
|
||||
# chat:message:error, unblocking the frontend.
|
||||
if isinstance(response, JSONResponse) and response.status_code >= 400:
|
||||
raise Exception(get_response_error_detail(response))
|
||||
|
||||
ctx = await build_chat_response_context(request, form_data, user, model, metadata, tasks, events)
|
||||
if ctx is None:
|
||||
ctx = await build_chat_response_context(request, form_data, user, model, metadata, tasks, events)
|
||||
else:
|
||||
ctx.update(form_data=form_data, metadata=metadata, events=events)
|
||||
|
||||
return await process_chat_response(response, ctx)
|
||||
except asyncio.CancelledError:
|
||||
|
|
@ -1667,6 +1696,7 @@ async def chat_completion(
|
|||
{
|
||||
'parentId': metadata.get('user_message_id', None),
|
||||
'error': {'content': error_detail},
|
||||
'done': True,
|
||||
},
|
||||
)
|
||||
|
||||
|
|
@ -1675,12 +1705,9 @@ async def chat_completion(
|
|||
await event_emitter(
|
||||
{
|
||||
'type': 'chat:message:error',
|
||||
'data': {'error': {'content': error_detail}},
|
||||
'data': {'error': {'content': error_detail}, 'done': True},
|
||||
}
|
||||
)
|
||||
await event_emitter(
|
||||
{'type': 'chat:tasks:cancel'},
|
||||
)
|
||||
|
||||
except Exception:
|
||||
pass
|
||||
|
|
@ -2235,6 +2262,7 @@ async def get_app_config(request: Request):
|
|||
'auth.enable_api_keys',
|
||||
'ui.enable_password_change_form',
|
||||
'direct.enable',
|
||||
'direct.integrations.enable',
|
||||
'folders.enable',
|
||||
'folders.max_file_count',
|
||||
'channels.enable',
|
||||
|
|
@ -2260,7 +2288,9 @@ async def get_app_config(request: Request):
|
|||
'ui.default_models',
|
||||
'ui.default_pinned_models',
|
||||
'ui.default_interface_settings',
|
||||
'ui.i18n',
|
||||
'ui.prompt_suggestions',
|
||||
'ui.prompt_suggestions_i18n',
|
||||
'code_execution.engine',
|
||||
'code_interpreter.engine',
|
||||
'audio.tts.engine',
|
||||
|
|
@ -2283,6 +2313,7 @@ async def get_app_config(request: Request):
|
|||
'name': app.state.WEBUI_NAME,
|
||||
'version': VERSION,
|
||||
'default_locale': str(DEFAULT_LOCALE),
|
||||
'i18n': config.get('ui.i18n') or {},
|
||||
'oauth': {
|
||||
# Hide providers (and thus the login buttons / auto-redirect) when OAuth
|
||||
# is disabled, without clearing the admin's provider configuration.
|
||||
|
|
@ -2294,6 +2325,7 @@ async def get_app_config(request: Request):
|
|||
'auto_redirect': config.get('oauth.auto_redirect'),
|
||||
},
|
||||
'features': {
|
||||
'slim': USE_SLIM,
|
||||
# --- Public: required by login/signup page pre-auth ---
|
||||
'auth': WEBUI_AUTH,
|
||||
'auth_trusted_header': bool(WEBUI_AUTH_TRUSTED_EMAIL_HEADER),
|
||||
|
|
@ -2317,6 +2349,7 @@ async def get_app_config(request: Request):
|
|||
'enable_public_active_users_count': ENABLE_PUBLIC_ACTIVE_USERS_COUNT,
|
||||
'enable_easter_eggs': ENABLE_EASTER_EGGS,
|
||||
'enable_direct_connections': config.get('direct.enable'),
|
||||
'enable_direct_integrations': config.get('direct.integrations.enable', False),
|
||||
'enable_plugins': ENABLE_PLUGINS,
|
||||
'enable_folders': config.get('folders.enable'),
|
||||
'folder_max_file_count': config.get('folders.max_file_count'),
|
||||
|
|
@ -2361,6 +2394,7 @@ async def get_app_config(request: Request):
|
|||
'default_models': config.get('ui.default_models'),
|
||||
'default_pinned_models': config.get('ui.default_pinned_models'),
|
||||
'default_prompt_suggestions': config.get('ui.prompt_suggestions'),
|
||||
'default_prompt_suggestions_i18n': config.get('ui.prompt_suggestions_i18n'),
|
||||
**({'user_count': user_count} if user_count is not None else {}),
|
||||
'code': {
|
||||
'engine': config.get('code_execution.engine'),
|
||||
|
|
@ -2575,8 +2609,8 @@ async def get_app_latest_release_version(user=Depends(get_verified_user)):
|
|||
|
||||
return {'current': VERSION, 'latest': latest_version[1:]}
|
||||
except Exception as e:
|
||||
log.debug(e)
|
||||
return {'current': VERSION, 'latest': VERSION}
|
||||
log.warning(f'Version update check failed: {e}')
|
||||
return {'current': VERSION, 'latest': None}
|
||||
|
||||
|
||||
@app.get('/api/changelog')
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ from typing import Optional
|
|||
import bcrypt
|
||||
from open_webui.internal.db import Base, JSONField, get_async_db_context
|
||||
from open_webui.models.users import User, UserModel, UserProfileImageResponse, Users
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import BaseModel, field_validator
|
||||
from sqlalchemy import Boolean, Column, String, Text, delete, select, update
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
|
@ -87,7 +87,7 @@ class SignupForm(BaseModel):
|
|||
@classmethod
|
||||
def check_profile_image_url(cls, v: str | None) -> str | None:
|
||||
if v is not None:
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
return v
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -312,6 +312,7 @@ class AutomationTable:
|
|||
rows = result.scalars().all()
|
||||
|
||||
from open_webui.utils.automations import next_run_ns
|
||||
from open_webui.utils.recurrence import RecurrenceEvaluationTimeout
|
||||
|
||||
# Batch-fetch user timezones so rescheduling respects each
|
||||
# user's local timezone instead of falling back to server time.
|
||||
|
|
@ -323,13 +324,20 @@ class AutomationTable:
|
|||
tz_result = await db.execute(select(User.id, User.timezone).where(User.id.in_(user_ids)))
|
||||
timezone_by_user_id = {uid: tz for uid, tz in tz_result.all()}
|
||||
|
||||
claimed = []
|
||||
for row in rows:
|
||||
try:
|
||||
next_run_at = await next_run_ns(row.data.get('rrule', ''), tz=timezone_by_user_id.get(row.user_id))
|
||||
except RecurrenceEvaluationTimeout:
|
||||
log.warning('Skipping automation %s: recurrence evaluation timed out', row.id)
|
||||
continue
|
||||
row.last_run_at = now_ns
|
||||
row.next_run_at = next_run_ns(row.data.get('rrule', ''), tz=timezone_by_user_id.get(row.user_id))
|
||||
row.next_run_at = next_run_at
|
||||
claimed.append(row)
|
||||
|
||||
await db.commit()
|
||||
|
||||
return [AutomationModel.model_validate(r) for r in rows]
|
||||
return [AutomationModel.model_validate(r) for r in claimed]
|
||||
|
||||
|
||||
####################
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ from open_webui.constants import ERROR_MESSAGES
|
|||
from open_webui.models.access_grants import AccessGrantModel, AccessGrants
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.users import User, UserModel, UserResponse
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
from pydantic import BaseModel, ConfigDict, Field
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
BigInteger,
|
||||
|
|
@ -179,6 +179,20 @@ class CalendarUpdateForm(BaseModel):
|
|||
access_grants: Optional[list[dict]] = None
|
||||
|
||||
|
||||
async def validate_calendar_rrule(value: Optional[str]) -> None:
|
||||
if value:
|
||||
from open_webui.utils.recurrence import rrule_interval_seconds
|
||||
|
||||
try:
|
||||
interval = await rrule_interval_seconds(value)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e)) from e
|
||||
if interval is not None and interval < MIN_CALENDAR_RRULE_INTERVAL_SECONDS:
|
||||
raise ValueError(ERROR_MESSAGES.CALENDAR_RRULE_TOO_FREQUENT)
|
||||
|
||||
|
||||
class CalendarEventForm(BaseModel):
|
||||
calendar_id: str
|
||||
title: str
|
||||
|
|
@ -193,22 +207,6 @@ class CalendarEventForm(BaseModel):
|
|||
meta: Optional[dict] = None
|
||||
attendees: Optional[list[dict]] = None
|
||||
|
||||
@field_validator('rrule')
|
||||
@classmethod
|
||||
def reject_sub_daily_rrule(cls, value: Optional[str]) -> Optional[str]:
|
||||
if value:
|
||||
from open_webui.utils.automations import rrule_interval_seconds
|
||||
|
||||
try:
|
||||
interval = rrule_interval_seconds(value)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if interval is not None and interval < MIN_CALENDAR_RRULE_INTERVAL_SECONDS:
|
||||
raise ValueError(ERROR_MESSAGES.CALENDAR_RRULE_TOO_FREQUENT)
|
||||
return value
|
||||
|
||||
|
||||
class CalendarEventUpdateForm(BaseModel):
|
||||
calendar_id: Optional[str] = None
|
||||
|
|
@ -225,22 +223,6 @@ class CalendarEventUpdateForm(BaseModel):
|
|||
is_cancelled: Optional[bool] = None
|
||||
attendees: Optional[list[dict]] = None
|
||||
|
||||
@field_validator('rrule')
|
||||
@classmethod
|
||||
def reject_sub_daily_rrule(cls, value: Optional[str]) -> Optional[str]:
|
||||
if value:
|
||||
from open_webui.utils.automations import rrule_interval_seconds
|
||||
|
||||
try:
|
||||
interval = rrule_interval_seconds(value)
|
||||
except ValueError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if interval is not None and interval < MIN_CALENDAR_RRULE_INTERVAL_SECONDS:
|
||||
raise ValueError(ERROR_MESSAGES.CALENDAR_RRULE_TOO_FREQUENT)
|
||||
return value
|
||||
|
||||
|
||||
class RSVPForm(BaseModel):
|
||||
status: str # 'accepted' | 'declined' | 'tentative' | 'pending'
|
||||
|
|
@ -465,6 +447,7 @@ class CalendarEventTable:
|
|||
async def insert_new_event(
|
||||
self, user_id: str, form_data: CalendarEventForm, db: Optional[AsyncSession] = None
|
||||
) -> Optional[CalendarEventModel]:
|
||||
await validate_calendar_rrule(form_data.rrule)
|
||||
async with get_async_db_context(db) as db:
|
||||
now = int(time.time_ns())
|
||||
event = CalendarEvent(
|
||||
|
|
@ -695,6 +678,7 @@ class CalendarEventTable:
|
|||
async def update_event_by_id(
|
||||
self, id: str, form_data: CalendarEventUpdateForm, db: Optional[AsyncSession] = None
|
||||
) -> Optional[CalendarEventModel]:
|
||||
await validate_calendar_rrule(form_data.rrule)
|
||||
async with get_async_db_context(db) as db:
|
||||
result = await db.execute(select(CalendarEvent).filter(CalendarEvent.id == id))
|
||||
event = result.scalars().first()
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ from open_webui.models.access_grants import (
|
|||
)
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.users import User
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
|
|
@ -253,7 +253,7 @@ class ChannelWebhookForm(BaseModel):
|
|||
def check_profile_image_url(cls, v: Optional[str]) -> Optional[str]:
|
||||
if v is None:
|
||||
return v
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
|
||||
|
||||
class ChannelTable:
|
||||
|
|
|
|||
|
|
@ -258,6 +258,7 @@ class ChatForm(BaseModel):
|
|||
class ChatImportForm(ChatForm):
|
||||
meta: dict | None = {}
|
||||
pinned: bool | None = False
|
||||
archived: bool | None = False
|
||||
current_message_id: str | None = None
|
||||
created_at: int | None = None
|
||||
updated_at: int | None = None
|
||||
|
|
@ -267,11 +268,6 @@ class ChatsImportForm(BaseModel):
|
|||
chats: list[ChatImportForm]
|
||||
|
||||
|
||||
class ChatTitleMessagesForm(BaseModel):
|
||||
title: str
|
||||
messages: list[dict]
|
||||
|
||||
|
||||
class ChatTitleForm(BaseModel):
|
||||
title: str
|
||||
|
||||
|
|
@ -309,6 +305,7 @@ class ChatTitleIdResponse(BaseModel):
|
|||
last_read_at: int | None = None
|
||||
snippet: str | None = None
|
||||
active: bool = False
|
||||
archived: bool = False
|
||||
|
||||
|
||||
class SharedChatResponse(BaseModel):
|
||||
|
|
@ -420,9 +417,6 @@ class ChatTable:
|
|||
"""
|
||||
Clean a Chat SQLAlchemy model's title + chat JSON,
|
||||
and return True if anything changed.
|
||||
|
||||
The message write paths (upsert/status/delete) rely on this
|
||||
leaving the blob clean and sanitize only the data they add.
|
||||
"""
|
||||
changed = False
|
||||
|
||||
|
|
@ -649,6 +643,7 @@ class ChatTable:
|
|||
'meta': form_data.meta,
|
||||
'variables': form_data.variables or {},
|
||||
'pinned': form_data.pinned,
|
||||
'archived': form_data.archived,
|
||||
'folder_id': form_data.folder_id,
|
||||
'current_message_id': form_data.current_message_id or self.get_current_message_id(form_data.chat),
|
||||
'created_at': (form_data.created_at if form_data.created_at else int(time.time())),
|
||||
|
|
@ -1108,11 +1103,16 @@ class ChatTable:
|
|||
if messages_map and message_id in messages_map:
|
||||
return messages_map[message_id]
|
||||
|
||||
chat = await self.get_chat_by_id(id)
|
||||
if chat is None:
|
||||
# Messages the frontend saved straight into the chat blob have no chat_message row yet.
|
||||
async with get_async_db_context() as session:
|
||||
result = await session.execute(select(Chat.chat[('history', 'messages')]).filter_by(id=id))
|
||||
row = result.one_or_none()
|
||||
|
||||
if row is None:
|
||||
return None
|
||||
|
||||
return chat.chat.get('history', {}).get('messages', {}).get(message_id, {})
|
||||
messages = row[0] or {}
|
||||
return self._clean_null_bytes(messages.get(message_id, {}))
|
||||
|
||||
async def get_message_metadata(
|
||||
self,
|
||||
|
|
@ -1160,7 +1160,6 @@ class ChatTable:
|
|||
if chat_item is None:
|
||||
return None
|
||||
|
||||
self._sanitize_chat_row(chat_item)
|
||||
chat = chat_item.chat or {}
|
||||
self._repair_chat_current_id(chat)
|
||||
|
||||
|
|
@ -1168,7 +1167,7 @@ class ChatTable:
|
|||
saved_message = self.upsert_message_to_history(history, message_id, message)
|
||||
chat['history'] = history
|
||||
chat_item.chat = chat # chat is a fresh dict when the column was empty
|
||||
chat_item.title = chat.get('title', 'New Chat')
|
||||
chat_item.title = self._clean_null_bytes(chat.get('title', 'New Chat'))
|
||||
chat_item.current_message_id = self.get_current_message_id(chat)
|
||||
flag_modified(chat_item, 'chat')
|
||||
|
||||
|
|
@ -1206,7 +1205,6 @@ class ChatTable:
|
|||
if chat_item is None:
|
||||
return None
|
||||
|
||||
self._sanitize_chat_row(chat_item)
|
||||
chat = chat_item.chat or {}
|
||||
self._repair_chat_current_id(chat)
|
||||
|
||||
|
|
@ -1214,7 +1212,7 @@ class ChatTable:
|
|||
deleted_ids = self.delete_message_from_history(history, message_id)
|
||||
if not deleted_ids:
|
||||
chat_item.chat = chat
|
||||
chat_item.title = chat.get('title', 'New Chat')
|
||||
chat_item.title = self._clean_null_bytes(chat.get('title', 'New Chat'))
|
||||
chat_item.current_message_id = self.get_current_message_id(chat)
|
||||
flag_modified(chat_item, 'chat')
|
||||
await session.commit()
|
||||
|
|
@ -1223,7 +1221,7 @@ class ChatTable:
|
|||
messages = history.get('messages') or {}
|
||||
chat['history'] = history
|
||||
chat_item.chat = chat
|
||||
chat_item.title = chat.get('title', 'New Chat')
|
||||
chat_item.title = self._clean_null_bytes(chat.get('title', 'New Chat'))
|
||||
chat_item.current_message_id = self.get_current_message_id(chat)
|
||||
flag_modified(chat_item, 'chat')
|
||||
chat_item.updated_at = int(time.time())
|
||||
|
|
@ -1253,7 +1251,6 @@ class ChatTable:
|
|||
if chat_item is None:
|
||||
return None
|
||||
|
||||
self._sanitize_chat_row(chat_item)
|
||||
chat = chat_item.chat or {}
|
||||
self._repair_chat_current_id(chat)
|
||||
history = chat.get('history', {})
|
||||
|
|
@ -1265,7 +1262,7 @@ class ChatTable:
|
|||
|
||||
chat['history'] = history
|
||||
chat_item.chat = chat
|
||||
chat_item.title = chat.get('title', 'New Chat')
|
||||
chat_item.title = self._clean_null_bytes(chat.get('title', 'New Chat'))
|
||||
chat_item.current_message_id = self.get_current_message_id(chat)
|
||||
flag_modified(chat_item, 'chat')
|
||||
await session.commit()
|
||||
|
|
@ -1990,10 +1987,8 @@ class ChatTable:
|
|||
]
|
||||
|
||||
# Extract folder names
|
||||
folders = await Folders.search_folders_by_names(
|
||||
user_id,
|
||||
[word.replace('folder:', '') for word in search_text_words if word.startswith('folder:')],
|
||||
)
|
||||
folder_names = [word.replace('folder:', '') for word in search_text_words if word.startswith('folder:')]
|
||||
folders = await Folders.search_folders_by_names(user_id, folder_names)
|
||||
folder_ids = [folder.id for folder in folders]
|
||||
|
||||
is_pinned = None
|
||||
|
|
@ -2037,7 +2032,7 @@ class ChatTable:
|
|||
else:
|
||||
stmt = stmt.filter(Chat.share_id.is_(None))
|
||||
|
||||
if folder_ids:
|
||||
if folder_names:
|
||||
stmt = stmt.filter(Chat.folder_id.in_(folder_ids))
|
||||
|
||||
# Check if the database dialect is either 'sqlite' or 'postgresql'
|
||||
|
|
@ -2540,18 +2535,15 @@ class ChatTable:
|
|||
except Exception:
|
||||
return False
|
||||
|
||||
async def move_chats_by_user_id_and_folder_id(
|
||||
async def move_chats_by_folder_id(
|
||||
self,
|
||||
user_id: str,
|
||||
folder_id: str,
|
||||
new_folder_id: str | None,
|
||||
db: AsyncSession | None = None,
|
||||
) -> bool:
|
||||
try:
|
||||
async with get_async_db_context(db) as session:
|
||||
await session.execute(
|
||||
update(Chat).filter_by(user_id=user_id, folder_id=folder_id).values(folder_id=new_folder_id)
|
||||
)
|
||||
await session.execute(update(Chat).filter_by(folder_id=folder_id).values(folder_id=new_folder_id))
|
||||
await session.commit()
|
||||
|
||||
return True
|
||||
|
|
|
|||
|
|
@ -625,6 +625,7 @@ class KnowledgeTable:
|
|||
db=db,
|
||||
),
|
||||
breadcrumbs=await self.get_directory_breadcrumbs(
|
||||
knowledge_id,
|
||||
filter.get('directory_id') if filter else None,
|
||||
db=db,
|
||||
),
|
||||
|
|
@ -908,6 +909,7 @@ class KnowledgeTable:
|
|||
|
||||
async def get_directory_breadcrumbs(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
directory_id: Optional[str],
|
||||
db: Optional[AsyncSession] = None,
|
||||
) -> list[KnowledgeDirectoryModel]:
|
||||
|
|
@ -922,7 +924,10 @@ class KnowledgeTable:
|
|||
|
||||
while current_id and current_id not in seen:
|
||||
seen.add(current_id)
|
||||
result = await db.execute(select(KnowledgeDirectory).filter_by(id=current_id))
|
||||
# Scoped by knowledge base so a caller-supplied id cannot walk another one's tree.
|
||||
result = await db.execute(
|
||||
select(KnowledgeDirectory).filter_by(id=current_id, knowledge_id=knowledge_id)
|
||||
)
|
||||
directory = result.scalars().first()
|
||||
if not directory:
|
||||
break
|
||||
|
|
@ -1069,6 +1074,26 @@ class KnowledgeTable:
|
|||
for child_id in child_ids:
|
||||
await self._delete_files_in_subtree(child_id, db=db)
|
||||
|
||||
async def get_files_by_id_and_directory_id(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
directory_id: str,
|
||||
db: Optional[AsyncSession] = None,
|
||||
) -> list[FileModel]:
|
||||
"""Get all files in a directory and its subdirectories."""
|
||||
async with get_async_db_context(db) as db:
|
||||
directory_ids = [directory_id]
|
||||
for parent_id in directory_ids:
|
||||
result = await db.execute(select(KnowledgeDirectory.id).filter_by(parent_id=parent_id))
|
||||
directory_ids.extend(result.scalars().all())
|
||||
result = await db.execute(
|
||||
select(File)
|
||||
.join(KnowledgeFile, File.id == KnowledgeFile.file_id)
|
||||
.filter(KnowledgeFile.knowledge_id == knowledge_id)
|
||||
.filter(KnowledgeFile.directory_id.in_(directory_ids))
|
||||
)
|
||||
return [FileModel.model_validate(file) for file in result.scalars().all()]
|
||||
|
||||
async def move_file_to_directory(
|
||||
self,
|
||||
knowledge_id: str,
|
||||
|
|
|
|||
|
|
@ -10,17 +10,13 @@ from open_webui.models.access_grants import AccessGrantModel, AccessGrants
|
|||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.users import User, UserModel, UserResponse, Users
|
||||
from open_webui.utils.misc import json_text_variants
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationInfo, field_validator, model_validator
|
||||
from sqlalchemy import BigInteger, Boolean, Column, String, Text, cast, delete, func, or_, select, update
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Track invalid profile_image_url values we've already warned about so we
|
||||
# don't flood the logs on every DB read (the validator fires per-row).
|
||||
_warned_profile_urls: set[str] = set()
|
||||
|
||||
|
||||
def normalize_model_tags(tags: Any) -> list[dict[str, str]]:
|
||||
if not isinstance(tags, list):
|
||||
|
|
@ -79,26 +75,24 @@ class ModelMeta(BaseModel):
|
|||
"""Metadata for a workspace model entry (profile, description, tags, capabilities)."""
|
||||
|
||||
profile_image_url: str | None = None
|
||||
background_image_url: str | None = None
|
||||
description: str | None = Field(default=None, description='User-facing description of the model.')
|
||||
i18n: dict[str, Any] | None = None
|
||||
capabilities: dict | None = None
|
||||
knowledge: list[Any] | None = None
|
||||
|
||||
model_config = ConfigDict(extra='allow')
|
||||
|
||||
@field_validator('profile_image_url', mode='before')
|
||||
@field_validator('profile_image_url', 'background_image_url', mode='before')
|
||||
@classmethod
|
||||
def check_profile_image_url(cls, v: str | None) -> str | None:
|
||||
def check_image_url(cls, v: str | None, info: ValidationInfo) -> str | None:
|
||||
if v is None:
|
||||
return v
|
||||
try:
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v, file_only=info.field_name == 'background_image_url')
|
||||
except ValueError:
|
||||
if v not in _warned_profile_urls:
|
||||
_warned_profile_urls.add(v)
|
||||
log.warning(
|
||||
'Clearing invalid profile_image_url stored in DB (likely a legacy SVG data-URI): %.80s…',
|
||||
v,
|
||||
)
|
||||
if info.field_name == 'background_image_url':
|
||||
raise
|
||||
return None
|
||||
|
||||
@field_validator('knowledge', mode='before')
|
||||
|
|
@ -175,7 +169,7 @@ class ModelAccessListResponse(BaseModel):
|
|||
class ModelForm(BaseModel):
|
||||
model_config = ConfigDict(extra='ignore')
|
||||
|
||||
id: str
|
||||
id: str = Field(pattern=r'^\S+$')
|
||||
base_model_id: str | None = None
|
||||
name: str
|
||||
meta: ModelMeta
|
||||
|
|
@ -248,11 +242,14 @@ class ModelsTable:
|
|||
return models
|
||||
|
||||
async def get_models(
|
||||
self, writable_by_user_id: str | None = None, db: AsyncSession | None = None
|
||||
self, writable_by_user_id: str | None = None, db: AsyncSession | None = None, ids: list[str] | None = None
|
||||
) -> list[ModelUserResponse]:
|
||||
async with get_async_db_context(db) as db:
|
||||
stmt = select(Model).filter(Model.base_model_id != None)
|
||||
|
||||
if ids is not None:
|
||||
stmt = stmt.filter(Model.id.in_(ids))
|
||||
|
||||
if writable_by_user_id:
|
||||
user_group_ids = {
|
||||
group.id for group in await Groups.get_groups_by_member_id(writable_by_user_id, db=db)
|
||||
|
|
@ -290,13 +287,15 @@ class ModelsTable:
|
|||
)
|
||||
return models
|
||||
|
||||
async def get_model_owners_attaching_file(self, file_id: str, db: AsyncSession | None = None) -> dict[str, str]:
|
||||
"""Map of model id to owner id for workspace models whose knowledge attaches this file."""
|
||||
async def get_model_owner_ids_by_file_id(
|
||||
self, file_id: str, db: AsyncSession | None = None, include_background: bool = False
|
||||
) -> dict[str, str]:
|
||||
"""Return model IDs mapped to owner IDs for models referencing the file."""
|
||||
async with get_async_db_context(db) as db:
|
||||
# File ids are server-generated uuids, so the text match can only over-match.
|
||||
result = await db.execute(
|
||||
select(Model.id, Model.user_id, Model.meta).filter(
|
||||
Model.base_model_id.is_not(None), cast(Model.meta, String).like(f'%"{file_id}"%')
|
||||
Model.base_model_id.is_not(None), cast(Model.meta, String).like(f'%{file_id}%')
|
||||
)
|
||||
)
|
||||
return {
|
||||
|
|
@ -306,6 +305,7 @@ class ModelsTable:
|
|||
isinstance(item, dict) and item.get('type') == 'file' and item.get('id') == file_id
|
||||
for item in meta.get('knowledge') or []
|
||||
)
|
||||
or (include_background and meta.get('background_image_url') == f'/api/v1/files/{file_id}/content')
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
|
|
@ -448,11 +448,13 @@ class ModelsTable:
|
|||
|
||||
return ModelListResponse(items=models, total=total)
|
||||
|
||||
async def get_model_meta_by_id(self, id: str, db: AsyncSession | None = None) -> tuple[dict, int | None]:
|
||||
"""Return (meta, updated_at) for a model, skipping access grant resolution."""
|
||||
async def get_model_meta_by_id(
|
||||
self, id: str, db: AsyncSession | None = None
|
||||
) -> tuple[dict, str, int | None] | None:
|
||||
"""Return (meta, user_id, updated_at) for a model, skipping access grant resolution."""
|
||||
try:
|
||||
async with get_async_db_context(db) as db:
|
||||
result = await db.execute(select(Model.meta, Model.updated_at).filter_by(id=id))
|
||||
result = await db.execute(select(Model.meta, Model.user_id, Model.updated_at).filter_by(id=id))
|
||||
return result.first()
|
||||
except Exception:
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -506,14 +506,16 @@ class PromptsTable:
|
|||
)
|
||||
|
||||
# Update prompt fields
|
||||
prompt.name = form_data.name
|
||||
prompt.command = form_data.command
|
||||
prompt.content = form_data.content
|
||||
prompt.data = form_data.data or prompt.data
|
||||
prompt.meta = form_data.meta or prompt.meta
|
||||
|
||||
if form_data.tags is not None:
|
||||
prompt.tags = form_data.tags
|
||||
if form_data.is_production:
|
||||
prompt.name = form_data.name
|
||||
prompt.content = form_data.content
|
||||
prompt.data = form_data.data or prompt.data
|
||||
prompt.meta = form_data.meta or prompt.meta
|
||||
|
||||
if form_data.tags is not None:
|
||||
prompt.tags = form_data.tags
|
||||
|
||||
if form_data.access_grants is not None:
|
||||
await AccessGrants.set_access_grants('prompt', prompt.id, form_data.access_grants, db=session)
|
||||
|
|
@ -531,7 +533,7 @@ class PromptsTable:
|
|||
'command': prompt.command,
|
||||
'data': form_data.data or {},
|
||||
'meta': form_data.meta or {},
|
||||
'tags': prompt.tags or [],
|
||||
'tags': form_data.tags if form_data.tags is not None else (prompt.tags or []),
|
||||
'access_grants': [grant.model_dump() for grant in current_access_grants],
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -33,6 +33,7 @@ class Skill(Base):
|
|||
|
||||
|
||||
class SkillMeta(BaseModel):
|
||||
i18n: dict[str, dict[str, str]] | None = None
|
||||
tags: Optional[list[str]] = []
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ class Tool(Base): # database table definition
|
|||
|
||||
|
||||
class ToolMeta(BaseModel):
|
||||
i18n: dict[str, dict[str, str]] | None = None
|
||||
description: str | None = None
|
||||
manifest: dict | None = {}
|
||||
has_user_valves: bool = False
|
||||
|
|
|
|||
|
|
@ -4,12 +4,18 @@ from __future__ import annotations
|
|||
|
||||
import datetime
|
||||
import time
|
||||
from typing import Optional
|
||||
from typing import Literal, Optional
|
||||
from open_webui.env import DATABASE_USER_ACTIVE_STATUS_UPDATE_INTERVAL
|
||||
from open_webui.internal.db import Base, JSONField, get_async_db_context
|
||||
from open_webui.utils.misc import throttle
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from pydantic import (
|
||||
BaseModel,
|
||||
ConfigDict,
|
||||
Field,
|
||||
field_validator,
|
||||
model_validator,
|
||||
)
|
||||
from sqlalchemy import (
|
||||
JSON,
|
||||
BigInteger,
|
||||
|
|
@ -36,6 +42,97 @@ from sqlalchemy.ext.asyncio import AsyncSession
|
|||
####################
|
||||
|
||||
|
||||
class InterfaceTitleSettings(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
auto: bool | None = None
|
||||
|
||||
|
||||
class InterfaceImageCompressionSize(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
width: int | float | Literal[''] | None = None
|
||||
height: int | float | Literal[''] | None = None
|
||||
|
||||
|
||||
class InterfaceFloatingActionButton(BaseModel):
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
id: str
|
||||
label: str
|
||||
input: bool
|
||||
prompt: str
|
||||
|
||||
|
||||
class InterfaceSettings(BaseModel):
|
||||
"""Fields owned by the Interface settings panel; not the entire user UI dict."""
|
||||
|
||||
model_config = ConfigDict(extra='forbid')
|
||||
|
||||
autoTags: bool | None = None
|
||||
autoFollowUps: bool | None = None
|
||||
highContrastMode: bool | None = None
|
||||
detectArtifacts: bool | None = None
|
||||
responseAutoCopy: bool | None = None
|
||||
showUsername: bool | None = None
|
||||
showUpdateToast: bool | None = None
|
||||
showChangelog: bool | None = None
|
||||
showEmojiInCall: bool | None = None
|
||||
voiceInterruption: bool | None = None
|
||||
displayMultiModelResponsesInTabs: bool | None = None
|
||||
chatFadeStreamingText: bool | None = None
|
||||
richTextInput: bool | None = None
|
||||
showFormattingToolbar: bool | None = None
|
||||
insertPromptAsRichText: bool | None = None
|
||||
promptAutocomplete: bool | None = None
|
||||
insertSuggestionPrompt: bool | None = None
|
||||
keepFollowUpPrompts: bool | None = None
|
||||
insertFollowUpPrompt: bool | None = None
|
||||
regenerateMenu: bool | None = None
|
||||
enableMessageQueue: bool | None = None
|
||||
largeTextAsFile: bool | None = None
|
||||
copyFormatted: bool | None = None
|
||||
collapseCodeBlocks: bool | None = None
|
||||
renderMarkdownInUserMessages: bool | None = None
|
||||
renderMarkdownInAssistantMessages: bool | None = None
|
||||
expandDetails: bool | None = None
|
||||
chatHoverPreview: bool | None = None
|
||||
renderMarkdownInPreviews: bool | None = None
|
||||
chatBubble: bool | None = None
|
||||
widescreenMode: bool | None = None
|
||||
splitLargeChunks: bool | None = None
|
||||
scrollOnBranchChange: bool | None = None
|
||||
scrollOnResponseGeneration: bool | None = None
|
||||
showFilesOnTerminalSelect: bool | None = None
|
||||
temporaryChatByDefault: bool | None = None
|
||||
userLocation: bool | None = None
|
||||
showChatTitleInTab: bool | None = None
|
||||
iframeSandboxAllowScripts: bool | None = None
|
||||
iframeSandboxAllowSameOrigin: bool | None = None
|
||||
iframeSandboxAllowForms: bool | None = None
|
||||
iframeSandboxAllowDownloads: bool | None = None
|
||||
terminalPreviewAllowSameOrigin: bool | None = None
|
||||
stylizedPdfExport: bool | None = None
|
||||
hapticFeedback: bool | None = None
|
||||
ctrlEnterToSend: bool | None = None
|
||||
showFloatingActionButtons: bool | None = None
|
||||
imageCompression: bool | None = None
|
||||
imageCompressionInChannels: bool | None = None
|
||||
|
||||
landingPageMode: Literal['', 'chat'] | None = None
|
||||
chatDirection: Literal['LTR', 'RTL', 'auto'] | None = None
|
||||
terminalFileDisplay: Literal['sidebar', 'inline'] | None = None
|
||||
defaultUploadContext: Literal['full', 'focused'] | None = None
|
||||
webSearch: Literal['always'] | None = None
|
||||
models: list[str] | None = None
|
||||
backgroundImageUrl: str | None = None
|
||||
fontFamily: str | None = None
|
||||
textScale: float | None = None
|
||||
title: InterfaceTitleSettings | None = None
|
||||
imageCompressionSize: InterfaceImageCompressionSize | None = None
|
||||
floatingActionButtons: list[InterfaceFloatingActionButton] | None = None
|
||||
|
||||
|
||||
class UserSettings(BaseModel):
|
||||
ui: dict | None = {}
|
||||
model_config = ConfigDict(extra='allow')
|
||||
|
|
@ -180,7 +277,7 @@ class UpdateProfileForm(BaseModel):
|
|||
@field_validator('profile_image_url')
|
||||
@classmethod
|
||||
def check_profile_image_url(cls, v: str) -> str:
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
|
||||
|
||||
class UserGroupIdsModel(UserModel):
|
||||
|
|
@ -270,7 +367,7 @@ class UserUpdateForm(BaseModel):
|
|||
def check_profile_image_url(cls, v: str | None) -> str | None:
|
||||
if v is None:
|
||||
return v
|
||||
return validate_profile_image_url(v)
|
||||
return validate_image_url(v)
|
||||
|
||||
|
||||
class UsersTable:
|
||||
|
|
@ -286,7 +383,7 @@ class UsersTable:
|
|||
db: AsyncSession | None = None,
|
||||
) -> UserModel | None:
|
||||
try:
|
||||
profile_image_url = validate_profile_image_url(profile_image_url)
|
||||
profile_image_url = validate_image_url(profile_image_url)
|
||||
except ValueError:
|
||||
profile_image_url = '/user.png'
|
||||
|
||||
|
|
@ -657,7 +754,7 @@ class UsersTable:
|
|||
db: AsyncSession | None = None,
|
||||
) -> UserModel | None:
|
||||
try:
|
||||
profile_image_url = validate_profile_image_url(profile_image_url)
|
||||
profile_image_url = validate_image_url(profile_image_url)
|
||||
except ValueError:
|
||||
profile_image_url = '/user.png'
|
||||
|
||||
|
|
@ -706,7 +803,9 @@ class UsersTable:
|
|||
return None
|
||||
scim = dict(user.scim or {})
|
||||
scim[provider] = {'external_id': external_id}
|
||||
user.scim = scim
|
||||
if scim != user.scim:
|
||||
user.scim = scim
|
||||
user.updated_at = int(time.time())
|
||||
await session.commit()
|
||||
return UserModel.model_validate(user)
|
||||
|
||||
|
|
@ -729,7 +828,18 @@ class UsersTable:
|
|||
if not user:
|
||||
return None
|
||||
user_settings = dict(user.settings or {})
|
||||
updated = dict(updated)
|
||||
ui_settings = updated.pop('ui', None)
|
||||
user_settings.update(updated)
|
||||
if ui_settings is not None:
|
||||
# UI updates are field-level patches: omission keeps a value; null resets it.
|
||||
current_ui_settings = dict(user_settings.get('ui') or {})
|
||||
for key, value in ui_settings.items():
|
||||
if value is None:
|
||||
current_ui_settings.pop(key, None)
|
||||
else:
|
||||
current_ui_settings[key] = value
|
||||
user_settings['ui'] = current_ui_settings
|
||||
user.settings = user_settings
|
||||
await session.commit()
|
||||
return UserModel.model_validate(user)
|
||||
|
|
|
|||
121
backend/open_webui/retrieval/loaders/local.py
Normal file
121
backend/open_webui/retrieval/loaders/local.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
from importlib import import_module
|
||||
from pathlib import Path
|
||||
|
||||
from bs4 import BeautifulSoup
|
||||
from langchain_core.documents import Document
|
||||
|
||||
|
||||
class TextLoader:
|
||||
def __init__(self, file_path, encoding=None):
|
||||
self.file_path = str(file_path)
|
||||
self.encoding = encoding
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
try:
|
||||
text = Path(self.file_path).read_text(encoding=self.encoding)
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Error loading {self.file_path}') from e
|
||||
return [
|
||||
Document(
|
||||
page_content=text,
|
||||
metadata={'source': self.file_path},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class HTMLLoader(TextLoader):
|
||||
def load(self) -> list[Document]:
|
||||
with open(self.file_path, encoding=self.encoding) as file:
|
||||
soup = BeautifulSoup(file, 'lxml')
|
||||
return [
|
||||
Document(
|
||||
page_content=soup.get_text(),
|
||||
metadata={'source': self.file_path, 'title': str(soup.title.string) if soup.title else ''},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class DocxLoader(TextLoader):
|
||||
def load(self) -> list[Document]:
|
||||
import docx2txt
|
||||
|
||||
return [
|
||||
Document(
|
||||
page_content=docx2txt.process(Path(self.file_path).expanduser()),
|
||||
metadata={'source': self.file_path},
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
class UnstructuredLoader:
|
||||
def __init__(self, file_path, file_format, mode='single', **kwargs):
|
||||
# Match the optional-package check; format dependencies are loaded when parsing.
|
||||
import_module('unstructured')
|
||||
self.file_path = file_path
|
||||
self.file_format = file_format
|
||||
self.mode = mode
|
||||
self.kwargs = kwargs
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
file_format = self.file_format
|
||||
if file_format in ('doc', 'ppt', 'pptx'):
|
||||
from unstructured.file_utils.filetype import detect_filetype
|
||||
|
||||
legacy_format = 'doc' if file_format == 'doc' else 'ppt'
|
||||
try:
|
||||
import_module('magic')
|
||||
except ImportError:
|
||||
is_legacy = Path(self.file_path).suffix == f'.{legacy_format}'
|
||||
else:
|
||||
is_legacy = detect_filetype(self.file_path).name.lower() == legacy_format
|
||||
file_format = legacy_format if is_legacy else legacy_format + 'x'
|
||||
elif file_format == 'msg':
|
||||
from unstructured.file_utils.filetype import detect_filetype
|
||||
|
||||
detected = detect_filetype(self.file_path).name
|
||||
if detected not in ('EML', 'MSG'):
|
||||
raise ValueError(f'Unsupported email file type: {detected}')
|
||||
file_format = 'email' if detected == 'EML' else 'msg'
|
||||
|
||||
module = import_module(f'unstructured.partition.{file_format}')
|
||||
elements = getattr(module, f'partition_{file_format}')(filename=self.file_path, **self.kwargs)
|
||||
metadata = {'source': str(self.file_path)}
|
||||
if self.mode == 'elements':
|
||||
return [
|
||||
Document(
|
||||
page_content=str(element),
|
||||
metadata={
|
||||
**metadata,
|
||||
**element.metadata.to_dict(),
|
||||
'category': element.category,
|
||||
'element_id': element.id,
|
||||
},
|
||||
)
|
||||
for element in elements
|
||||
]
|
||||
return [Document(page_content='\n\n'.join(map(str, elements)), metadata=metadata)]
|
||||
|
||||
|
||||
class DocumentIntelligenceLoader:
|
||||
def __init__(self, file_path, api_endpoint, api_key=None, azure_credential=None, api_model='prebuilt-layout'):
|
||||
if (api_key is None) == (azure_credential is None):
|
||||
raise ValueError('Provide exactly one of api_key or azure_credential.')
|
||||
self.file_path = file_path
|
||||
self.api_endpoint = api_endpoint
|
||||
self.api_key = api_key
|
||||
self.azure_credential = azure_credential
|
||||
self.api_model = api_model
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
from azure.ai.documentintelligence import DocumentIntelligenceClient
|
||||
from azure.core.credentials import AzureKeyCredential
|
||||
|
||||
credential = self.azure_credential if self.azure_credential is not None else AzureKeyCredential(self.api_key)
|
||||
with DocumentIntelligenceClient(self.api_endpoint, credential) as client, open(self.file_path, 'rb') as file:
|
||||
result = client.begin_analyze_document(
|
||||
self.api_model,
|
||||
body=file,
|
||||
content_type='application/octet-stream',
|
||||
output_content_format='markdown',
|
||||
).result()
|
||||
return [Document(page_content=result.content, metadata=result.as_dict())]
|
||||
|
|
@ -7,27 +7,29 @@ import zipfile
|
|||
|
||||
import ftfy
|
||||
import requests
|
||||
from fastapi import HTTPException
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from langchain_community.document_loaders import (
|
||||
AzureAIDocumentIntelligenceLoader,
|
||||
BSHTMLLoader,
|
||||
CSVLoader,
|
||||
Docx2txtLoader,
|
||||
PyPDFLoader,
|
||||
TextLoader,
|
||||
)
|
||||
from langchain_core.documents import Document
|
||||
from open_webui.env import (
|
||||
AIOHTTP_CLIENT_SESSION_SSL,
|
||||
GLOBAL_LOG_LEVEL,
|
||||
USE_SLIM,
|
||||
MINERU_MAX_MARKDOWN_BYTES,
|
||||
REQUESTS_VERIFY,
|
||||
)
|
||||
from open_webui.retrieval.loaders.datalab_marker import DatalabMarkerLoader
|
||||
from open_webui.retrieval.loaders.external_document import ExternalDocumentLoader
|
||||
from open_webui.retrieval.loaders.local import (
|
||||
DocumentIntelligenceLoader,
|
||||
DocxLoader,
|
||||
HTMLLoader,
|
||||
TextLoader,
|
||||
UnstructuredLoader,
|
||||
)
|
||||
from open_webui.retrieval.loaders.mineru import MinerULoader
|
||||
from open_webui.retrieval.loaders.mistral import MistralLoader
|
||||
from open_webui.retrieval.loaders.paddleocr_vl import PADDLEOCR_VL_SUPPORTED_EXTENSIONS, PaddleOCRVLLoader
|
||||
from open_webui.retrieval.loaders.pdf import PDFLoader
|
||||
from open_webui.utils.headers import get_user_groups_for_custom_headers
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
|
||||
|
|
@ -50,6 +52,7 @@ known_source_ext = [
|
|||
'h',
|
||||
'c',
|
||||
'cs',
|
||||
'ino',
|
||||
'sql',
|
||||
'log',
|
||||
'ini',
|
||||
|
|
@ -88,6 +91,7 @@ known_source_ext = [
|
|||
'yaml',
|
||||
'yml',
|
||||
'toml',
|
||||
'svg',
|
||||
]
|
||||
|
||||
known_archive_ext = {'docx', 'epub', 'odt', 'pptx', 'xlsx'}
|
||||
|
|
@ -160,7 +164,21 @@ class CSVLoaderWithSummary:
|
|||
self.encoding = encoding
|
||||
|
||||
def load(self) -> list[Document]:
|
||||
docs = CSVLoader(self.file_path, encoding=self.encoding).load()
|
||||
docs = []
|
||||
try:
|
||||
with open(self.file_path, newline='', encoding=self.encoding) as file:
|
||||
for index, row in enumerate(csv.DictReader(file)):
|
||||
fields = []
|
||||
for key, value in row.items():
|
||||
if isinstance(value, str):
|
||||
value = value.strip()
|
||||
elif isinstance(value, list):
|
||||
value = ','.join(v.strip() for v in value)
|
||||
fields.append(f'{key.strip() if key is not None else key}: {value}')
|
||||
content = '\n'.join(fields)
|
||||
docs.append(Document(page_content=content, metadata={'source': self.file_path, 'row': index}))
|
||||
except Exception as e:
|
||||
raise RuntimeError(f'Error loading {self.file_path}') from e
|
||||
if os.getenv('ENABLE_RAG_CSV_SUMMARY', 'False').lower() == 'true':
|
||||
summary = get_csv_summary(self.filename, self.file_path, self.encoding)
|
||||
if summary:
|
||||
|
|
@ -215,7 +233,7 @@ class TikaLoader:
|
|||
if self.extract_images == True:
|
||||
headers['X-Tika-PDFextractInlineImages'] = 'true'
|
||||
|
||||
endpoint_path = 'tika/json/text' if self.server_version == '4' else 'tika/text'
|
||||
endpoint_path = 'tika/json/md' if self.server_version == '4' else 'tika/text'
|
||||
content_key = 'tk:content' if self.server_version == '4' else 'X-TIKA:content'
|
||||
endpoint = f'{self.url.rstrip("/")}/{endpoint_path}'
|
||||
|
||||
|
|
@ -273,8 +291,17 @@ class DoclingLoader:
|
|||
)
|
||||
if r.ok:
|
||||
result = r.json()
|
||||
# Docling reports failed and skipped conversions inside HTTP 200 responses.
|
||||
conversion_status = result.get('status')
|
||||
if conversion_status in ['failure', 'skipped']:
|
||||
error_details = (
|
||||
'; '.join(filter(None, (error.get('error_message') for error in result.get('errors', []))))
|
||||
or 'no error message provided'
|
||||
)
|
||||
raise Exception(f'Error calling Docling: conversion status {conversion_status} - {error_details}')
|
||||
|
||||
document_data = result.get('document', {})
|
||||
md_content = document_data.get('md_content', '')
|
||||
md_content = document_data.get('md_content') or ''
|
||||
text = md_content or '<No text content found>'
|
||||
|
||||
metadata = {'Content-Type': self.mime_type} if self.mime_type else {}
|
||||
|
|
@ -313,7 +340,11 @@ class Loader:
|
|||
def load(self, filename: str, file_content_type: str, file_path: str) -> list[Document]:
|
||||
loader = self._get_loader(filename, file_content_type, file_path)
|
||||
docs = loader.load()
|
||||
return [Document(page_content=ftfy.fix_text(doc.page_content), metadata=doc.metadata) for doc in docs]
|
||||
# ftfy's auto mode unescapes entities on every line before the first literal '<', rewriting the document.
|
||||
return [
|
||||
Document(page_content=ftfy.fix_text(doc.page_content, unescape_html=False), metadata=doc.metadata)
|
||||
for doc in docs
|
||||
]
|
||||
|
||||
async def aload(self, filename: str, file_content_type: str, file_path: str) -> list[Document]:
|
||||
"""
|
||||
|
|
@ -613,14 +644,14 @@ class Loader:
|
|||
)
|
||||
):
|
||||
if self.kwargs.get('DOCUMENT_INTELLIGENCE_KEY') != '':
|
||||
loader = AzureAIDocumentIntelligenceLoader(
|
||||
loader = DocumentIntelligenceLoader(
|
||||
file_path=file_path,
|
||||
api_endpoint=self.kwargs.get('DOCUMENT_INTELLIGENCE_ENDPOINT'),
|
||||
api_key=self.kwargs.get('DOCUMENT_INTELLIGENCE_KEY'),
|
||||
api_model=self.kwargs.get('DOCUMENT_INTELLIGENCE_MODEL'),
|
||||
)
|
||||
else:
|
||||
loader = AzureAIDocumentIntelligenceLoader(
|
||||
loader = DocumentIntelligenceLoader(
|
||||
file_path=file_path,
|
||||
api_endpoint=self.kwargs.get('DOCUMENT_INTELLIGENCE_ENDPOINT'),
|
||||
azure_credential=DefaultAzureCredential(),
|
||||
|
|
@ -666,8 +697,21 @@ class Loader:
|
|||
file_path=file_path,
|
||||
)
|
||||
else:
|
||||
if USE_SLIM:
|
||||
if file_ext == 'csv':
|
||||
return CSVLoaderWithSummary(file_path, filename, self._detect_text_encoding(file_path))
|
||||
if file_ext in ['htm', 'html']:
|
||||
return HTMLLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
if file_ext in ['txt', 'md', 'markdown', 'rst', 'xml'] or self._is_text_file(
|
||||
file_ext, file_content_type
|
||||
):
|
||||
return TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
raise HTTPException(
|
||||
503,
|
||||
'This file type requires an external document extractor in slim. Configure one that supports it.',
|
||||
)
|
||||
if file_ext == 'pdf':
|
||||
loader = PyPDFLoader(
|
||||
loader = PDFLoader(
|
||||
file_path,
|
||||
extract_images=self.kwargs.get('PDF_EXTRACT_IMAGES'),
|
||||
mode=self.kwargs.get('PDF_LOADER_MODE', 'page'),
|
||||
|
|
@ -680,9 +724,7 @@ class Loader:
|
|||
)
|
||||
elif file_ext == 'rst':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredRSTLoader
|
||||
|
||||
loader = UnstructuredRSTLoader(file_path, mode='elements')
|
||||
loader = UnstructuredLoader(file_path, 'rst', mode='elements')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -692,9 +734,7 @@ class Loader:
|
|||
loader = TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
elif file_ext == 'xml':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredXMLLoader
|
||||
|
||||
loader = UnstructuredXMLLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'xml')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -703,14 +743,12 @@ class Loader:
|
|||
)
|
||||
loader = TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
elif file_ext in ['htm', 'html']:
|
||||
loader = BSHTMLLoader(file_path, open_encoding='unicode_escape')
|
||||
loader = HTMLLoader(file_path, encoding='unicode_escape')
|
||||
elif file_ext == 'md':
|
||||
loader = TextLoader(file_path, encoding=self._detect_text_encoding(file_path))
|
||||
elif file_content_type == 'application/epub+zip':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredEPubLoader
|
||||
|
||||
loader = UnstructuredEPubLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'epub')
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .epub files requires the 'unstructured' package. "
|
||||
|
|
@ -720,12 +758,10 @@ class Loader:
|
|||
file_content_type == 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'
|
||||
or file_ext == 'docx'
|
||||
):
|
||||
loader = Docx2txtLoader(file_path)
|
||||
loader = DocxLoader(file_path)
|
||||
elif file_ext == 'doc' or file_content_type == 'application/msword':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredWordDocumentLoader
|
||||
|
||||
loader = UnstructuredWordDocumentLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'doc')
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .doc files requires the 'unstructured' package. "
|
||||
|
|
@ -736,9 +772,7 @@ class Loader:
|
|||
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
|
||||
] or file_ext in ['xls', 'xlsx']:
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredExcelLoader
|
||||
|
||||
loader = UnstructuredExcelLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'xlsx')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -751,9 +785,7 @@ class Loader:
|
|||
'application/vnd.openxmlformats-officedocument.presentationml.presentation',
|
||||
] or file_ext in ['ppt', 'pptx']:
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredPowerPointLoader
|
||||
|
||||
loader = UnstructuredPowerPointLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'ppt' if file_ext == 'ppt' else 'pptx')
|
||||
except ImportError:
|
||||
log.warning(
|
||||
"The 'unstructured' package is not installed. "
|
||||
|
|
@ -763,12 +795,8 @@ class Loader:
|
|||
loader = PptxLoader(file_path)
|
||||
elif file_ext == 'msg':
|
||||
try:
|
||||
from langchain_community.document_loaders import (
|
||||
UnstructuredEmailLoader,
|
||||
)
|
||||
|
||||
# unstructured parses .msg via python-oxmsg; avoids extract_msg's beautifulsoup4<4.14 conflict
|
||||
loader = UnstructuredEmailLoader(file_path, process_attachments=False)
|
||||
loader = UnstructuredLoader(file_path, 'msg', process_attachments=False)
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .msg files requires the 'unstructured' package. "
|
||||
|
|
@ -776,9 +804,7 @@ class Loader:
|
|||
)
|
||||
elif file_ext == 'odt':
|
||||
try:
|
||||
from langchain_community.document_loaders import UnstructuredODTLoader
|
||||
|
||||
loader = UnstructuredODTLoader(file_path)
|
||||
loader = UnstructuredLoader(file_path, 'odt')
|
||||
except ImportError:
|
||||
raise ValueError(
|
||||
"Processing .odt files requires the 'unstructured' package. "
|
||||
|
|
|
|||
101
backend/open_webui/retrieval/loaders/pdf.py
Normal file
101
backend/open_webui/retrieval/loaders/pdf.py
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
import datetime as dt
|
||||
import io
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
from langchain_core.document_loaders import BaseLoader
|
||||
from langchain_core.documents import Document
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class PDFLoader(BaseLoader):
|
||||
def __init__(self, file_path, *, extract_images=False, mode='page'):
|
||||
if mode not in ('single', 'page'):
|
||||
raise ValueError("PDF mode must be 'single' or 'page'")
|
||||
self.file_path = str(Path(file_path).expanduser())
|
||||
self.extract_images = extract_images
|
||||
self.mode = mode
|
||||
self.ocr = None
|
||||
|
||||
def lazy_load(self):
|
||||
from pypdf import PdfReader
|
||||
|
||||
with open(self.file_path, 'rb') as file:
|
||||
reader = PdfReader(file)
|
||||
metadata = {'producer': 'PyPDF', 'creator': 'PyPDF', 'creationdate': ''}
|
||||
for key, value in (reader.metadata or {}).items():
|
||||
key = key.removeprefix('/').lower()
|
||||
value = value if type(value) in (str, int) else str(value)
|
||||
if key in ('creationdate', 'moddate') and isinstance(value, str):
|
||||
try:
|
||||
value = dt.datetime.strptime(value.replace("'", ''), 'D:%Y%m%d%H%M%S%z').isoformat()
|
||||
except ValueError:
|
||||
pass
|
||||
metadata[key] = (
|
||||
value.strip()
|
||||
if isinstance(value, str) and key not in ('creationdate', 'moddate', 'page_count', 'file_path')
|
||||
else value
|
||||
)
|
||||
metadata.update(source=self.file_path, total_pages=len(reader.pages))
|
||||
labels = reader.page_labels if self.mode == 'page' else None
|
||||
texts = []
|
||||
for index, page in enumerate(reader.pages):
|
||||
text = page.extract_text()
|
||||
if self.extract_images:
|
||||
image_text = self._extract_images(page)
|
||||
if image_text:
|
||||
text = self._merge_image_text(text, image_text)
|
||||
text = text.strip()
|
||||
if self.mode == 'page':
|
||||
yield Document(page_content=text, metadata={**metadata, 'page': index, 'page_label': labels[index]})
|
||||
else:
|
||||
texts.append(text)
|
||||
if self.mode == 'single':
|
||||
yield Document(page_content='\n\f'.join(texts), metadata=metadata)
|
||||
|
||||
@staticmethod
|
||||
def _merge_image_text(text, image_text):
|
||||
# Insert before the final paragraphs/footer where possible, matching existing chunks.
|
||||
position, separator = len(text), '\n\n'
|
||||
for _ in range(2):
|
||||
for delimiter in ('\n\n\n', '\n\n'):
|
||||
found = text.rfind(delimiter, 0, position)
|
||||
if found >= 0:
|
||||
position, separator = found, delimiter
|
||||
break
|
||||
else:
|
||||
break
|
||||
return text[:position] + separator + image_text + text[position:]
|
||||
|
||||
def _extract_images(self, page):
|
||||
import numpy as np
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
if '/Resources' not in page or '/XObject' not in page['/Resources']:
|
||||
return ''
|
||||
texts = []
|
||||
xobjects = page['/Resources']['/XObject']
|
||||
for name in xobjects:
|
||||
try:
|
||||
stream = xobjects[name]
|
||||
if stream.get('/Subtype') != '/Image':
|
||||
continue
|
||||
try:
|
||||
# Encoded images, including CMYK JPEGs, can go straight to Pillow.
|
||||
image = Image.open(io.BytesIO(stream.get_data()))
|
||||
except UnidentifiedImageError:
|
||||
image = stream.decode_as_image()
|
||||
pixels = np.array(image.convert('RGB'))
|
||||
except Exception as e:
|
||||
log.warning('Skipping unreadable PDF image %s: %s', name, e)
|
||||
continue
|
||||
|
||||
if self.ocr is None:
|
||||
from rapidocr import RapidOCR
|
||||
|
||||
self.ocr = RapidOCR()
|
||||
result = self.ocr(pixels)
|
||||
if result and result.txts:
|
||||
texts.append('\n'.join(result.txts).strip())
|
||||
return '\n\n' + '\n'.join(filter(None, texts)) + '\n\n' if any(texts) else ''
|
||||
|
|
@ -10,13 +10,13 @@ from typing import Awaitable, Optional, Union
|
|||
from urllib.parse import quote
|
||||
|
||||
import aiohttp
|
||||
import numpy as np
|
||||
import requests
|
||||
from huggingface_hub import snapshot_download
|
||||
from fastapi import HTTPException
|
||||
from langchain_classic.retrievers import (
|
||||
ContextualCompressionRetriever,
|
||||
EnsembleRetriever,
|
||||
)
|
||||
from langchain_community.retrievers import BM25Retriever
|
||||
from langchain_core.documents import Document
|
||||
from open_webui.config import (
|
||||
RAG_EMBEDDING_CONTENT_PREFIX,
|
||||
|
|
@ -32,7 +32,10 @@ from open_webui.env import (
|
|||
BYPASS_RETRIEVAL_ACCESS_CONTROL,
|
||||
ENABLE_FORWARD_USER_INFO_HEADERS,
|
||||
ENABLE_RETRIEVAL_UNSCOPED_COLLECTIONS,
|
||||
MPS_INFERENCE_LOCK,
|
||||
OFFLINE_MODE,
|
||||
RAG_SOURCE_METADATA_KEYS,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.models.access_grants import AccessGrants
|
||||
from open_webui.models.chats import Chats
|
||||
|
|
@ -45,7 +48,7 @@ from open_webui.models.users import UserModel
|
|||
from open_webui.retrieval.loaders.youtube import YoutubeLoader
|
||||
from open_webui.retrieval.vector.async_client import ASYNC_VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.external import retrieve_external_knowledge
|
||||
from open_webui.retrieval.vector.factory import VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import get_vector_db_client
|
||||
from open_webui.retrieval.vector.main import GetResult, SearchResult
|
||||
from open_webui.retrieval.web.utils import get_web_loader
|
||||
from open_webui.utils.access_control.files import get_owner_accessible_folder_files, has_access_to_file
|
||||
|
|
@ -62,6 +65,15 @@ from langchain_core.callbacks import CallbackManagerForRetrieverRun
|
|||
from langchain_core.retrievers import BaseRetriever
|
||||
|
||||
|
||||
class BM25Retriever(BaseRetriever):
|
||||
docs: list[Document]
|
||||
vectorizer: Any
|
||||
k: int
|
||||
|
||||
def _get_relevant_documents(self, query: str, *, run_manager: CallbackManagerForRetrieverRun) -> list[Document]:
|
||||
return self.vectorizer.get_top_n(query.split(), self.docs, n=self.k)
|
||||
|
||||
|
||||
def is_youtube_url(url: str) -> bool:
|
||||
youtube_regex = r'^(https?://)?(www\.)?(youtube\.com|youtu\.be)/.+$'
|
||||
return re.match(youtube_regex, url) is not None
|
||||
|
|
@ -330,27 +342,23 @@ class VectorSearchRetriever(BaseRetriever):
|
|||
|
||||
|
||||
def query_doc(collection_name: str, query_embedding: list[float], k: int, user: UserModel = None):
|
||||
try:
|
||||
log.debug('query_doc:doc %s', collection_name)
|
||||
result = VECTOR_DB_CLIENT.search(
|
||||
collection_name=collection_name,
|
||||
vectors=[query_embedding],
|
||||
limit=k,
|
||||
)
|
||||
log.debug('query_doc:doc %s', collection_name)
|
||||
result = get_vector_db_client().search(
|
||||
collection_name=collection_name,
|
||||
vectors=[query_embedding],
|
||||
limit=k,
|
||||
)
|
||||
|
||||
if result:
|
||||
log.info('query_doc:result %s %s', result.ids, result.metadatas)
|
||||
if result:
|
||||
log.info('query_doc:result %s %s', result.ids, result.metadatas)
|
||||
|
||||
return result
|
||||
except Exception as e:
|
||||
log.exception(f'Error querying doc {collection_name} with limit {k}: {e}')
|
||||
raise e
|
||||
return result
|
||||
|
||||
|
||||
def get_doc(collection_name: str, user: UserModel = None):
|
||||
try:
|
||||
log.debug('get_doc:doc %s', collection_name)
|
||||
result = VECTOR_DB_CLIENT.get(collection_name=collection_name)
|
||||
result = get_vector_db_client().get(collection_name=collection_name)
|
||||
|
||||
if result:
|
||||
log.info('query_doc:result %s %s', result.ids, result.metadatas)
|
||||
|
|
@ -495,122 +503,116 @@ async def query_doc_with_hybrid_search(
|
|||
enable_enriched_texts: bool = False,
|
||||
native_hybrid_search: bool = True,
|
||||
) -> dict:
|
||||
try:
|
||||
if native_hybrid_search and not enable_enriched_texts:
|
||||
native_result = await query_doc_with_native_hybrid_search(
|
||||
collection_name=collection_name,
|
||||
query=query,
|
||||
embedding_function=embedding_function,
|
||||
k=k,
|
||||
reranking_function=reranking_function,
|
||||
k_reranker=k_reranker,
|
||||
r=r,
|
||||
hybrid_bm25_weight=hybrid_bm25_weight,
|
||||
)
|
||||
if native_result is not None:
|
||||
return native_result
|
||||
|
||||
if collection_result is None:
|
||||
collection_result = await ASYNC_VECTOR_DB_CLIENT.get(collection_name=collection_name)
|
||||
|
||||
# First check if collection_result has the required attributes
|
||||
if (
|
||||
not collection_result
|
||||
or not hasattr(collection_result, 'documents')
|
||||
or not hasattr(collection_result, 'metadatas')
|
||||
):
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
# Now safely check the documents content after confirming attributes exist
|
||||
if (
|
||||
not collection_result.documents
|
||||
or len(collection_result.documents) == 0
|
||||
or not collection_result.documents[0]
|
||||
):
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
log.debug('query_doc_with_hybrid_search:doc %s', collection_name)
|
||||
|
||||
original_texts = collection_result.documents[0]
|
||||
bm25_metadatas = [
|
||||
{**meta, CHUNK_HASH_KEY: _content_hash(original_texts[idx])}
|
||||
for idx, meta in enumerate(collection_result.metadatas[0])
|
||||
]
|
||||
|
||||
bm25_texts = get_enriched_texts(collection_result) if enable_enriched_texts else original_texts
|
||||
|
||||
bm25_retriever = BM25Retriever.from_texts(
|
||||
texts=bm25_texts,
|
||||
metadatas=bm25_metadatas,
|
||||
)
|
||||
bm25_retriever.k = k
|
||||
|
||||
vector_search_retriever = VectorSearchRetriever(
|
||||
if native_hybrid_search and not enable_enriched_texts:
|
||||
native_result = await query_doc_with_native_hybrid_search(
|
||||
collection_name=collection_name,
|
||||
query=query,
|
||||
embedding_function=embedding_function,
|
||||
top_k=k,
|
||||
)
|
||||
|
||||
# Use CHUNK_HASH_KEY for dedup so enriched BM25 texts don't defeat RRF
|
||||
if hybrid_bm25_weight <= 0:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[vector_search_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
elif hybrid_bm25_weight >= 1:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
else:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever, vector_search_retriever],
|
||||
weights=[hybrid_bm25_weight, 1.0 - hybrid_bm25_weight],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
|
||||
compressor = RerankCompressor(
|
||||
embedding_function=embedding_function,
|
||||
top_n=k_reranker,
|
||||
k=k,
|
||||
reranking_function=reranking_function,
|
||||
r_score=r,
|
||||
k_reranker=k_reranker,
|
||||
r=r,
|
||||
hybrid_bm25_weight=hybrid_bm25_weight,
|
||||
)
|
||||
if native_result is not None:
|
||||
return native_result
|
||||
|
||||
if collection_result is None:
|
||||
collection_result = await ASYNC_VECTOR_DB_CLIENT.get(collection_name=collection_name)
|
||||
|
||||
# First check if collection_result has the required attributes
|
||||
if (
|
||||
not collection_result
|
||||
or not hasattr(collection_result, 'documents')
|
||||
or not hasattr(collection_result, 'metadatas')
|
||||
):
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
# Now safely check the documents content after confirming attributes exist
|
||||
if not collection_result.documents or len(collection_result.documents) == 0 or not collection_result.documents[0]:
|
||||
log.warning(f'query_doc_with_hybrid_search:no_docs {collection_name}')
|
||||
return {'documents': [], 'metadatas': [], 'distances': []}
|
||||
|
||||
log.debug('query_doc_with_hybrid_search:doc %s', collection_name)
|
||||
|
||||
original_texts = collection_result.documents[0]
|
||||
bm25_metadatas = [
|
||||
{**meta, CHUNK_HASH_KEY: _content_hash(original_texts[idx])}
|
||||
for idx, meta in enumerate(collection_result.metadatas[0])
|
||||
]
|
||||
|
||||
bm25_texts = get_enriched_texts(collection_result) if enable_enriched_texts else original_texts
|
||||
|
||||
from rank_bm25 import BM25Okapi
|
||||
|
||||
bm25_retriever = BM25Retriever(
|
||||
docs=[Document(page_content=text, metadata=meta) for text, meta in zip(bm25_texts, bm25_metadatas)],
|
||||
vectorizer=BM25Okapi([text.split() for text in bm25_texts]),
|
||||
k=k,
|
||||
)
|
||||
|
||||
vector_search_retriever = VectorSearchRetriever(
|
||||
collection_name=collection_name,
|
||||
embedding_function=embedding_function,
|
||||
top_k=k,
|
||||
)
|
||||
|
||||
# Use CHUNK_HASH_KEY for dedup so enriched BM25 texts don't defeat RRF
|
||||
if hybrid_bm25_weight <= 0:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[vector_search_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
elif hybrid_bm25_weight >= 1:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever],
|
||||
weights=[1.0],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
else:
|
||||
ensemble_retriever = EnsembleRetriever(
|
||||
retrievers=[bm25_retriever, vector_search_retriever],
|
||||
weights=[hybrid_bm25_weight, 1.0 - hybrid_bm25_weight],
|
||||
id_key=CHUNK_HASH_KEY,
|
||||
)
|
||||
|
||||
compression_retriever = ContextualCompressionRetriever(
|
||||
base_compressor=compressor, base_retriever=ensemble_retriever
|
||||
)
|
||||
compressor = RerankCompressor(
|
||||
embedding_function=embedding_function,
|
||||
top_n=k_reranker,
|
||||
reranking_function=reranking_function,
|
||||
r_score=r,
|
||||
)
|
||||
|
||||
result = await compression_retriever.ainvoke(query)
|
||||
compression_retriever = ContextualCompressionRetriever(
|
||||
base_compressor=compressor, base_retriever=ensemble_retriever
|
||||
)
|
||||
|
||||
distances = [d.metadata.get('score') for d in result]
|
||||
documents = [d.page_content for d in result]
|
||||
metadatas = [d.metadata for d in result]
|
||||
result = await compression_retriever.ainvoke(query)
|
||||
|
||||
# retrieve only min(k, k_reranker) items, sort and cut by distance if k < k_reranker
|
||||
if k < k_reranker:
|
||||
sorted_items = sorted(zip(distances, documents, metadatas), key=lambda x: x[0], reverse=True)
|
||||
sorted_items = sorted_items[:k]
|
||||
distances = [d.metadata.get('score') for d in result]
|
||||
documents = [d.page_content for d in result]
|
||||
metadatas = [d.metadata for d in result]
|
||||
|
||||
if sorted_items:
|
||||
distances, documents, metadatas = map(list, zip(*sorted_items))
|
||||
else:
|
||||
distances, documents, metadatas = [], [], []
|
||||
# retrieve only min(k, k_reranker) items, sort and cut by distance if k < k_reranker
|
||||
if k < k_reranker:
|
||||
sorted_items = sorted(zip(distances, documents, metadatas), key=lambda x: x[0], reverse=True)
|
||||
sorted_items = sorted_items[:k]
|
||||
|
||||
result = {
|
||||
'distances': [distances],
|
||||
'documents': [documents],
|
||||
'metadatas': [metadatas],
|
||||
}
|
||||
if sorted_items:
|
||||
distances, documents, metadatas = map(list, zip(*sorted_items))
|
||||
else:
|
||||
distances, documents, metadatas = [], [], []
|
||||
|
||||
log.info('query_doc_with_hybrid_search:result %s %s', result['metadatas'], result['distances'])
|
||||
return result
|
||||
except Exception as e:
|
||||
log.exception(f'Error querying doc {collection_name} with hybrid search: {e}')
|
||||
raise e
|
||||
result = {
|
||||
'distances': [distances],
|
||||
'documents': [documents],
|
||||
'metadatas': [metadatas],
|
||||
}
|
||||
|
||||
log.info('query_doc_with_hybrid_search:result %s %s', result['metadatas'], result['distances'])
|
||||
return result
|
||||
|
||||
|
||||
def merge_get_results(get_results: list[dict]) -> dict:
|
||||
|
|
@ -731,7 +733,8 @@ async def query_collection(
|
|||
log.debug('Hybrid search failed, falling back to vector search: %s', e)
|
||||
|
||||
results = []
|
||||
error = False
|
||||
last_error = None
|
||||
failed_collection_names = set()
|
||||
|
||||
def process_query_collection(collection_name, query_embedding):
|
||||
try:
|
||||
|
|
@ -742,11 +745,10 @@ async def query_collection(
|
|||
query_embedding=query_embedding,
|
||||
)
|
||||
if result is not None:
|
||||
return result.model_dump(), None
|
||||
return None, None
|
||||
return result.model_dump(), None, collection_name
|
||||
return None, None, collection_name
|
||||
except Exception as e:
|
||||
log.exception(f'Error when querying the collection: {e}')
|
||||
return None, e
|
||||
return None, e, collection_name
|
||||
|
||||
# Sanitize: filter out None/empty queries to prevent embedding crashes
|
||||
# (e.g. when get_last_user_message returns None)
|
||||
|
|
@ -767,14 +769,20 @@ async def query_collection(
|
|||
]
|
||||
)
|
||||
|
||||
for result, err in task_results:
|
||||
for result, err, collection_name in task_results:
|
||||
if err is not None:
|
||||
error = True
|
||||
last_error = err
|
||||
failed_collection_names.add(collection_name)
|
||||
elif result is not None:
|
||||
results.append(result)
|
||||
|
||||
if error and not results:
|
||||
log.warning('All collection queries failed. No results returned.')
|
||||
if failed_collection_names:
|
||||
log.error(
|
||||
'query_collection: %s collection(s) had failing queries: %s',
|
||||
len(failed_collection_names),
|
||||
', '.join(sorted(failed_collection_names)),
|
||||
exc_info=last_error,
|
||||
)
|
||||
|
||||
return merge_and_sort_query_results(results, k=k)
|
||||
|
||||
|
|
@ -791,7 +799,8 @@ async def query_collection_with_hybrid_search(
|
|||
enable_enriched_texts: bool = False,
|
||||
) -> dict:
|
||||
results = []
|
||||
error = False
|
||||
last_error = None
|
||||
failed_collection_names = set()
|
||||
|
||||
if not enable_enriched_texts:
|
||||
|
||||
|
|
@ -850,10 +859,9 @@ async def query_collection_with_hybrid_search(
|
|||
enable_enriched_texts=enable_enriched_texts,
|
||||
native_hybrid_search=False,
|
||||
)
|
||||
return result, None
|
||||
return result, None, collection_name
|
||||
except Exception as e:
|
||||
log.exception(f'Error when querying the collection with hybrid_search: {e}')
|
||||
return None, e
|
||||
return None, e, collection_name
|
||||
|
||||
# Prepare tasks for all collections and queries
|
||||
# Avoid running any tasks for collections that failed to fetch data (have assigned None)
|
||||
|
|
@ -867,13 +875,22 @@ async def query_collection_with_hybrid_search(
|
|||
# Run all queries in parallel using asyncio.gather
|
||||
task_results = await asyncio.gather(*[process_query(collection_name, query) for collection_name, query in tasks])
|
||||
|
||||
for result, err in task_results:
|
||||
for result, err, collection_name in task_results:
|
||||
if err is not None:
|
||||
error = True
|
||||
last_error = err
|
||||
failed_collection_names.add(collection_name)
|
||||
elif result is not None:
|
||||
results.append(result)
|
||||
|
||||
if error and not results:
|
||||
if failed_collection_names:
|
||||
log.error(
|
||||
'query_collection_with_hybrid_search: %s collection(s) had failing queries: %s',
|
||||
len(failed_collection_names),
|
||||
', '.join(sorted(failed_collection_names)),
|
||||
exc_info=last_error,
|
||||
)
|
||||
|
||||
if failed_collection_names and not results:
|
||||
raise Exception('Hybrid search failed for all collections. Using Non-hybrid search as fallback.')
|
||||
|
||||
return merge_and_sort_query_results(results, k=k)
|
||||
|
|
@ -1109,6 +1126,8 @@ def get_embedding_function(
|
|||
if embedding_engine == '':
|
||||
# Sentence transformers: CPU-bound sync operation
|
||||
async def async_embedding_function(query, prefix=None, user=None):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external embedding engine (openai, ollama, azure_openai).')
|
||||
# Deferred so a missing local model degrades RAG instead of crashing boot.
|
||||
if embedding_function is None:
|
||||
raise ValueError(
|
||||
|
|
@ -1116,17 +1135,16 @@ def get_embedding_function(
|
|||
'SentenceTransformer model name, or configure an external '
|
||||
'RAG_EMBEDDING_ENGINE (ollama, openai, azure_openai).'
|
||||
)
|
||||
return await asyncio.to_thread(
|
||||
(
|
||||
lambda query, prefix=None: embedding_function.encode(
|
||||
|
||||
def encode():
|
||||
with MPS_INFERENCE_LOCK:
|
||||
return embedding_function.encode(
|
||||
query,
|
||||
batch_size=int(embedding_batch_size),
|
||||
**({'prompt': prefix} if prefix else {}),
|
||||
).tolist()
|
||||
),
|
||||
query,
|
||||
prefix,
|
||||
)
|
||||
|
||||
return await asyncio.to_thread(encode)
|
||||
|
||||
return async_embedding_function
|
||||
elif embedding_engine in ['ollama', 'openai', 'azure_openai']:
|
||||
|
|
@ -1243,6 +1261,14 @@ async def generate_embeddings(
|
|||
|
||||
|
||||
def get_reranking_function(reranking_engine, reranking_model, reranking_function, reranking_batch_size=32):
|
||||
if USE_SLIM and reranking_model and reranking_engine != 'external':
|
||||
|
||||
def unavailable(query, documents, user=None):
|
||||
raise HTTPException(
|
||||
503, 'Configure an external reranker, or clear the reranking model to use cosine scoring.'
|
||||
)
|
||||
|
||||
return unavailable
|
||||
if reranking_function is None:
|
||||
return None
|
||||
if reranking_engine == 'external':
|
||||
|
|
@ -1250,9 +1276,14 @@ def get_reranking_function(reranking_engine, reranking_model, reranking_function
|
|||
[(query, doc.page_content) for doc in documents], user=user
|
||||
)
|
||||
else:
|
||||
return lambda query, documents, user=None: reranking_function.predict(
|
||||
[(query, doc.page_content) for doc in documents], batch_size=int(reranking_batch_size)
|
||||
)
|
||||
|
||||
def predict(query, documents, user=None):
|
||||
with MPS_INFERENCE_LOCK:
|
||||
return reranking_function.predict(
|
||||
[(query, doc.page_content) for doc in documents], batch_size=int(reranking_batch_size)
|
||||
)
|
||||
|
||||
return predict
|
||||
|
||||
|
||||
# UUIDs, SHA-256 digests, and prefixed variants thereof all fit [A-Za-z0-9_-].
|
||||
|
|
@ -1331,6 +1362,11 @@ async def filter_accessible_collections(
|
|||
return validated
|
||||
|
||||
|
||||
def filter_source_metadata(metadata: dict) -> dict:
|
||||
"""Keep only the chunk metadata keys the operator allowed the model to see."""
|
||||
return {key: metadata[key] for key in RAG_SOURCE_METADATA_KEYS if metadata.get(key) is not None}
|
||||
|
||||
|
||||
async def get_sources_from_items(
|
||||
request,
|
||||
items,
|
||||
|
|
@ -1683,6 +1719,8 @@ async def get_sources_from_items(
|
|||
|
||||
|
||||
def get_model_path(model: str, update_model: bool = False):
|
||||
from huggingface_hub import snapshot_download
|
||||
|
||||
# Construct huggingface_hub kwargs with local_files_only to return the snapshot path
|
||||
cache_dir = os.getenv('SENTENCE_TRANSFORMERS_HOME')
|
||||
|
||||
|
|
@ -1728,6 +1766,17 @@ from langchain_core.callbacks import Callbacks
|
|||
from langchain_core.documents import BaseDocumentCompressor, Document
|
||||
|
||||
|
||||
def cosine_similarity(query, documents) -> np.ndarray:
|
||||
"""Score one query against documents without loading a model runtime."""
|
||||
if len(documents) == 0:
|
||||
return np.array([], dtype=float)
|
||||
query = np.asarray(query, dtype=float).reshape(-1)
|
||||
documents = np.asarray(documents, dtype=float)
|
||||
query = query / max(np.linalg.norm(query), 1e-12)
|
||||
documents = documents / np.maximum(np.linalg.norm(documents, axis=1, keepdims=True), 1e-12)
|
||||
return documents @ query
|
||||
|
||||
|
||||
class RerankCompressor(BaseDocumentCompressor):
|
||||
embedding_function: Any
|
||||
top_n: int
|
||||
|
|
@ -1763,18 +1812,18 @@ class RerankCompressor(BaseDocumentCompressor):
|
|||
query: str,
|
||||
callbacks: Callbacks | None = None,
|
||||
) -> Sequence[Document]:
|
||||
if not documents:
|
||||
return []
|
||||
reranking = self.reranking_function is not None
|
||||
|
||||
scores = None
|
||||
if reranking:
|
||||
scores = await asyncio.to_thread(self.reranking_function, query, documents)
|
||||
else:
|
||||
from sentence_transformers import util as st_util
|
||||
|
||||
query_embedding = await self.embedding_function(query, RAG_EMBEDDING_QUERY_PREFIX)
|
||||
doc_texts = [doc.page_content for doc in documents]
|
||||
document_embedding = await self.embedding_function(doc_texts, RAG_EMBEDDING_CONTENT_PREFIX)
|
||||
scores = st_util.cos_sim(query_embedding, document_embedding)[0]
|
||||
scores = cosine_similarity(query_embedding, document_embedding)
|
||||
|
||||
if scores is not None:
|
||||
docs_with_scores = list(
|
||||
|
|
|
|||
|
|
@ -15,9 +15,8 @@ transparently dispatches each call to a worker thread via
|
|||
`asyncio.to_thread`. Async callers can `await ASYNC_VECTOR_DB_CLIENT.x(...)`
|
||||
in place of `VECTOR_DB_CLIENT.x(...)` and the loop stays responsive.
|
||||
|
||||
The original `VECTOR_DB_CLIENT` is unchanged, so callers already running
|
||||
inside `run_in_threadpool` (e.g. `save_docs_to_vector_db`) are not
|
||||
affected.
|
||||
Client initialization and calls run in the worker thread. Synchronous callers
|
||||
already inside `run_in_threadpool` use `get_vector_db_client()` directly.
|
||||
|
||||
Thread-safety expectations
|
||||
--------------------------
|
||||
|
|
@ -55,7 +54,7 @@ from __future__ import annotations
|
|||
import asyncio
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
from open_webui.retrieval.vector.factory import VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import get_vector_db_client
|
||||
from open_webui.retrieval.vector.main import (
|
||||
GetResult,
|
||||
SearchResult,
|
||||
|
|
@ -73,30 +72,30 @@ class AsyncVectorDBClient:
|
|||
typically swallowed by surrounding ``try/except``).
|
||||
"""
|
||||
|
||||
def __init__(self, sync_client: VectorDBBase) -> None:
|
||||
def __init__(self, sync_client: Optional[VectorDBBase] = None) -> None:
|
||||
self._sync = sync_client
|
||||
|
||||
@property
|
||||
def sync(self) -> VectorDBBase:
|
||||
"""Escape hatch for code that must call the sync client directly
|
||||
(e.g. already inside a worker thread)."""
|
||||
return self._sync
|
||||
return self._sync if self._sync is not None else get_vector_db_client()
|
||||
|
||||
@property
|
||||
def supports_hybrid_search(self) -> bool:
|
||||
return type(self._sync).hybrid_search is not VectorDBBase.hybrid_search
|
||||
return type(self.sync).hybrid_search is not VectorDBBase.hybrid_search
|
||||
|
||||
async def has_collection(self, collection_name: str) -> bool:
|
||||
return await asyncio.to_thread(self._sync.has_collection, collection_name)
|
||||
return await asyncio.to_thread(lambda: self.sync.has_collection(collection_name))
|
||||
|
||||
async def delete_collection(self, collection_name: str) -> None:
|
||||
return await asyncio.to_thread(self._sync.delete_collection, collection_name)
|
||||
return await asyncio.to_thread(lambda: self.sync.delete_collection(collection_name))
|
||||
|
||||
async def insert(self, collection_name: str, items: List[VectorItem]) -> None:
|
||||
return await asyncio.to_thread(self._sync.insert, collection_name, items)
|
||||
return await asyncio.to_thread(lambda: self.sync.insert(collection_name, items))
|
||||
|
||||
async def upsert(self, collection_name: str, items: List[VectorItem]) -> None:
|
||||
return await asyncio.to_thread(self._sync.upsert, collection_name, items)
|
||||
return await asyncio.to_thread(lambda: self.sync.upsert(collection_name, items))
|
||||
|
||||
async def search(
|
||||
self,
|
||||
|
|
@ -105,7 +104,7 @@ class AsyncVectorDBClient:
|
|||
filter: Optional[Dict] = None,
|
||||
limit: int = 10,
|
||||
) -> Optional[SearchResult]:
|
||||
return await asyncio.to_thread(self._sync.search, collection_name, vectors, filter, limit)
|
||||
return await asyncio.to_thread(lambda: self.sync.search(collection_name, vectors, filter, limit))
|
||||
|
||||
async def hybrid_search(
|
||||
self,
|
||||
|
|
@ -117,13 +116,7 @@ class AsyncVectorDBClient:
|
|||
hybrid_bm25_weight: float = 0.5,
|
||||
) -> Optional[SearchResult]:
|
||||
return await asyncio.to_thread(
|
||||
self._sync.hybrid_search,
|
||||
collection_name,
|
||||
query,
|
||||
vectors,
|
||||
filter,
|
||||
limit,
|
||||
hybrid_bm25_weight,
|
||||
lambda: self.sync.hybrid_search(collection_name, query, vectors, filter, limit, hybrid_bm25_weight)
|
||||
)
|
||||
|
||||
async def query(
|
||||
|
|
@ -132,10 +125,10 @@ class AsyncVectorDBClient:
|
|||
filter: Dict,
|
||||
limit: Optional[int] = None,
|
||||
) -> Optional[GetResult]:
|
||||
return await asyncio.to_thread(self._sync.query, collection_name, filter, limit)
|
||||
return await asyncio.to_thread(lambda: self.sync.query(collection_name, filter, limit))
|
||||
|
||||
async def get(self, collection_name: str) -> Optional[GetResult]:
|
||||
return await asyncio.to_thread(self._sync.get, collection_name)
|
||||
return await asyncio.to_thread(lambda: self.sync.get(collection_name))
|
||||
|
||||
async def delete(
|
||||
self,
|
||||
|
|
@ -143,10 +136,10 @@ class AsyncVectorDBClient:
|
|||
ids: Optional[List[str]] = None,
|
||||
filter: Optional[Dict] = None,
|
||||
) -> None:
|
||||
return await asyncio.to_thread(self._sync.delete, collection_name, ids, filter)
|
||||
return await asyncio.to_thread(lambda: self.sync.delete(collection_name, ids, filter))
|
||||
|
||||
async def reset(self) -> None:
|
||||
return await asyncio.to_thread(self._sync.reset)
|
||||
return await asyncio.to_thread(lambda: self.sync.reset())
|
||||
|
||||
|
||||
ASYNC_VECTOR_DB_CLIENT = AsyncVectorDBClient(VECTOR_DB_CLIENT)
|
||||
ASYNC_VECTOR_DB_CLIENT = AsyncVectorDBClient()
|
||||
|
|
|
|||
|
|
@ -16,6 +16,8 @@ from open_webui.config import (
|
|||
CHROMA_HTTP_SSL,
|
||||
CHROMA_TENANT,
|
||||
)
|
||||
from open_webui.env import USE_SLIM
|
||||
from fastapi import HTTPException
|
||||
from open_webui.retrieval.vector.main import (
|
||||
GetResult,
|
||||
SearchResult,
|
||||
|
|
@ -29,6 +31,8 @@ log = logging.getLogger(__name__)
|
|||
|
||||
class ChromaClient(VectorDBBase):
|
||||
def __init__(self):
|
||||
if USE_SLIM and not CHROMA_HTTP_HOST:
|
||||
raise HTTPException(503, 'Configure CHROMA_HTTP_HOST: embedded Chroma is unavailable in slim.')
|
||||
settings_dict = {
|
||||
'allow_reset': True,
|
||||
'anonymized_telemetry': False,
|
||||
|
|
@ -58,7 +62,7 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def has_collection(self, collection_name: str) -> bool:
|
||||
try:
|
||||
self.client.get_collection(name=collection_name)
|
||||
self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
return True
|
||||
except NotFoundError:
|
||||
return False
|
||||
|
|
@ -76,7 +80,7 @@ class ChromaClient(VectorDBBase):
|
|||
) -> Optional[SearchResult]:
|
||||
# Search for the nearest neighbor items based on the vectors and return 'limit' number of results.
|
||||
try:
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
result = collection.query(
|
||||
query_embeddings=vectors,
|
||||
|
|
@ -105,7 +109,7 @@ class ChromaClient(VectorDBBase):
|
|||
def query(self, collection_name: str, filter: dict, limit: Optional[int] = None) -> Optional[GetResult]:
|
||||
# Query the items from the collection based on the filter.
|
||||
try:
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
result = collection.get(
|
||||
where=filter,
|
||||
|
|
@ -125,7 +129,7 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def get(self, collection_name: str) -> Optional[GetResult]:
|
||||
# Get all the items in the collection.
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
result = collection.get()
|
||||
return GetResult(
|
||||
|
|
@ -139,7 +143,9 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def insert(self, collection_name: str, items: list[VectorItem]):
|
||||
# Insert the items into the collection, if the collection does not exist, it will be created.
|
||||
collection = self.client.get_or_create_collection(name=collection_name, metadata={'hnsw:space': 'cosine'})
|
||||
collection = self.client.get_or_create_collection(
|
||||
name=collection_name, metadata={'hnsw:space': 'cosine'}, embedding_function=None
|
||||
)
|
||||
|
||||
ids = [item['id'] for item in items]
|
||||
documents = [item['text'] for item in items]
|
||||
|
|
@ -157,7 +163,9 @@ class ChromaClient(VectorDBBase):
|
|||
|
||||
def upsert(self, collection_name: str, items: list[VectorItem]):
|
||||
# Update the items in the collection, if the items are not present, insert them. If the collection does not exist, it will be created.
|
||||
collection = self.client.get_or_create_collection(name=collection_name, metadata={'hnsw:space': 'cosine'})
|
||||
collection = self.client.get_or_create_collection(
|
||||
name=collection_name, metadata={'hnsw:space': 'cosine'}, embedding_function=None
|
||||
)
|
||||
|
||||
ids = [item['id'] for item in items]
|
||||
documents = [item['text'] for item in items]
|
||||
|
|
@ -174,7 +182,7 @@ class ChromaClient(VectorDBBase):
|
|||
):
|
||||
# Delete the items from the collection based on the ids.
|
||||
try:
|
||||
collection = self.client.get_collection(name=collection_name)
|
||||
collection = self.client.get_collection(name=collection_name, embedding_function=None)
|
||||
if collection:
|
||||
if ids:
|
||||
collection.delete(ids=ids)
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import process_metadata
|
||||
from pymilvus import DataType
|
||||
from pymilvus import MilvusClient as Client
|
||||
from pymilvus.exceptions import MilvusException
|
||||
|
|
@ -191,7 +192,7 @@ class MilvusClient(VectorDBBase):
|
|||
'id': item['id'],
|
||||
'vector': item['vector'],
|
||||
'text': text,
|
||||
'metadata': item['metadata'],
|
||||
'metadata': process_metadata(item['metadata']),
|
||||
RESOURCE_ID_FIELD: resource_id,
|
||||
}
|
||||
)
|
||||
|
|
|
|||
|
|
@ -311,6 +311,7 @@ class OpenGaussClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
@ -333,6 +334,7 @@ class OpenGaussClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions, process_metadata
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -400,7 +400,7 @@ class Oracle23aiClient(VectorDBBase):
|
|||
Returns:
|
||||
str: JSON representation of metadata
|
||||
"""
|
||||
return json.dumps(metadata, default=self._decimal_handler) if metadata else '{}'
|
||||
return json.dumps(process_metadata(metadata), default=self._decimal_handler) if metadata else '{}'
|
||||
|
||||
def _json_to_metadata(self, json_str: str) -> Dict:
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from open_webui.config import (
|
|||
PGVECTOR_HNSW_M,
|
||||
PGVECTOR_INDEX_METHOD,
|
||||
PGVECTOR_INITIALIZE_MAX_VECTOR_LENGTH,
|
||||
PGVECTOR_ITERATIVE_SCAN,
|
||||
PGVECTOR_IVFFLAT_LISTS,
|
||||
PGVECTOR_PGCRYPTO,
|
||||
PGVECTOR_PGCRYPTO_KEY,
|
||||
|
|
@ -154,6 +155,7 @@ class PgvectorClient(VectorDBBase):
|
|||
index_method, index_options = self._vector_index_configuration()
|
||||
self._ensure_vector_index(index_method, index_options)
|
||||
self._ensure_text_search_index()
|
||||
self.iterative_scan_sql = self._iterative_scan_setting(index_method)
|
||||
|
||||
self.session.execute(
|
||||
text(
|
||||
|
|
@ -223,6 +225,9 @@ class PgvectorClient(VectorDBBase):
|
|||
)
|
||||
|
||||
if not existing_index_def:
|
||||
if index_method == 'ivfflat' and not self._has_enough_ivfflat_training_rows():
|
||||
return
|
||||
|
||||
index_sql = (
|
||||
f'CREATE INDEX IF NOT EXISTS {index_name} '
|
||||
f'ON document_chunk USING {index_method} (vector {VECTOR_OPCLASS})'
|
||||
|
|
@ -237,6 +242,38 @@ class PgvectorClient(VectorDBBase):
|
|||
f' {index_options}' if index_options else '',
|
||||
)
|
||||
|
||||
def _has_enough_ivfflat_training_rows(self) -> bool:
|
||||
# ivfflat samples 50 rows per list to place its centroids, so recall stays poor until the table holds that many
|
||||
min_training_rows = 50 * PGVECTOR_IVFFLAT_LISTS
|
||||
row_count = self.session.execute(
|
||||
text('SELECT count(*) FROM (SELECT 1 FROM document_chunk LIMIT :min_training_rows) AS sample'),
|
||||
{'min_training_rows': min_training_rows},
|
||||
).scalar()
|
||||
|
||||
if row_count < min_training_rows:
|
||||
log.info(
|
||||
"Deferring vector index 'idx_document_chunk_vector' until document_chunk holds %s rows to cluster on, "
|
||||
'it has %s. Searches run as an exact scan until then.',
|
||||
min_training_rows,
|
||||
row_count,
|
||||
)
|
||||
return False
|
||||
return True
|
||||
|
||||
def _iterative_scan_setting(self, index_method: str) -> Optional[str]:
|
||||
if PGVECTOR_ITERATIVE_SCAN == 'off':
|
||||
return None
|
||||
|
||||
version = self.session.execute(text("SELECT extversion FROM pg_extension WHERE extname = 'vector'")).scalar()
|
||||
version_parts = [int(part) for part in (version or '').split('.') if part.isdigit()]
|
||||
if version_parts[:2] < [0, 8]:
|
||||
log.info('Iterative scan needs pgvector 0.8 or newer, the server has %s.', version or 'none')
|
||||
return None
|
||||
|
||||
# ivfflat only accepts relaxed_order
|
||||
mode = 'relaxed_order' if index_method == 'ivfflat' else PGVECTOR_ITERATIVE_SCAN
|
||||
return f'SET LOCAL {index_method}.iterative_scan = {mode}'
|
||||
|
||||
def _ensure_text_search_index(self) -> None:
|
||||
if PGVECTOR_PGCRYPTO:
|
||||
return
|
||||
|
|
@ -503,6 +540,9 @@ class PgvectorClient(VectorDBBase):
|
|||
.order_by(query_vectors.c.qid, subq.c.distance)
|
||||
)
|
||||
|
||||
if self.iterative_scan_sql:
|
||||
self.session.execute(text(self.iterative_scan_sql))
|
||||
|
||||
result_proxy = self.session.execute(stmt)
|
||||
results = result_proxy.all()
|
||||
|
||||
|
|
@ -512,6 +552,7 @@ class PgvectorClient(VectorDBBase):
|
|||
metadatas = [[] for _ in range(num_queries)]
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return SearchResult(
|
||||
ids=ids,
|
||||
distances=distances,
|
||||
|
|
@ -631,6 +672,7 @@ class PgvectorClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
@ -670,6 +712,7 @@ class PgvectorClient(VectorDBBase):
|
|||
results = query.all()
|
||||
|
||||
if not results:
|
||||
self.session.rollback()
|
||||
return None
|
||||
|
||||
ids = [[result.id for result in results]]
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions, process_metadata
|
||||
from qdrant_client import QdrantClient as Qclient
|
||||
from qdrant_client.http.models import PointStruct
|
||||
from qdrant_client.models import models
|
||||
|
|
@ -136,7 +136,7 @@ class QdrantClient(VectorDBBase):
|
|||
PointStruct(
|
||||
id=item['id'],
|
||||
vector=item['vector'],
|
||||
payload={'text': item['text'], 'metadata': item['metadata']},
|
||||
payload={'text': item['text'], 'metadata': process_metadata(item['metadata'])},
|
||||
)
|
||||
for item in items
|
||||
]
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ from open_webui.retrieval.vector.main import (
|
|||
VectorDBBase,
|
||||
VectorItem,
|
||||
)
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions
|
||||
from open_webui.retrieval.vector.utils import iter_filter_conditions, process_metadata
|
||||
from qdrant_client import QdrantClient as Qclient
|
||||
from qdrant_client.http.exceptions import UnexpectedResponse
|
||||
from qdrant_client.http.models import PointStruct
|
||||
|
|
@ -182,7 +182,7 @@ class QdrantClient(VectorDBBase):
|
|||
vector=item['vector'],
|
||||
payload={
|
||||
'text': item['text'],
|
||||
'metadata': item['metadata'],
|
||||
'metadata': process_metadata(item['metadata']),
|
||||
TENANT_ID_FIELD: tenant_id,
|
||||
},
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,12 @@
|
|||
from threading import Lock
|
||||
|
||||
from fastapi import HTTPException
|
||||
from open_webui.config import (
|
||||
ENABLE_MILVUS_MULTITENANCY_MODE,
|
||||
ENABLE_QDRANT_MULTITENANCY_MODE,
|
||||
VECTOR_DB,
|
||||
)
|
||||
from open_webui.env import USE_SLIM
|
||||
from open_webui.retrieval.vector.main import VectorDBBase
|
||||
from open_webui.retrieval.vector.type import VectorType
|
||||
|
||||
|
|
@ -13,6 +17,11 @@ class Vector:
|
|||
"""
|
||||
get vector db instance by vector type
|
||||
"""
|
||||
if USE_SLIM and vector_type != VectorType.PGVECTOR:
|
||||
raise HTTPException(
|
||||
503,
|
||||
'Slim requires PostgreSQL/pgvector for vector storage. Set VECTOR_DB=pgvector and PGVECTOR_DB_URL, or use the standard image.',
|
||||
)
|
||||
match vector_type:
|
||||
case VectorType.MILVUS:
|
||||
if ENABLE_MILVUS_MULTITENANCY_MODE:
|
||||
|
|
@ -88,4 +97,27 @@ class Vector:
|
|||
raise ValueError(f'Unsupported vector type: {vector_type}')
|
||||
|
||||
|
||||
VECTOR_DB_CLIENT = Vector.get_vector(VECTOR_DB)
|
||||
VECTOR_DB_CLIENT = None if USE_SLIM else Vector.get_vector(VECTOR_DB)
|
||||
_vector_client_lock = Lock()
|
||||
|
||||
|
||||
def get_vector_db_client() -> VectorDBBase:
|
||||
"""Initialize slim's remote client on first use so chat can start without it."""
|
||||
global VECTOR_DB_CLIENT
|
||||
if VECTOR_DB_CLIENT is not None:
|
||||
return VECTOR_DB_CLIENT
|
||||
with _vector_client_lock:
|
||||
if VECTOR_DB_CLIENT is None:
|
||||
from open_webui import config
|
||||
|
||||
if VECTOR_DB == VectorType.PGVECTOR and not config.PGVECTOR_DB_URL.startswith('postgres'):
|
||||
raise HTTPException(503, 'Configure PGVECTOR_DB_URL for remote vector storage.')
|
||||
try:
|
||||
VECTOR_DB_CLIENT = Vector.get_vector(VECTOR_DB)
|
||||
except HTTPException:
|
||||
raise
|
||||
except Exception as exc:
|
||||
raise HTTPException(
|
||||
503, f'Unable to connect to configured vector database ({VECTOR_DB}): {exc}'
|
||||
) from exc
|
||||
return VECTOR_DB_CLIENT
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ from __future__ import annotations
|
|||
import logging
|
||||
import urllib.request
|
||||
|
||||
from ddgs import DDGS
|
||||
from open_webui.env import USE_SLIM
|
||||
from open_webui.retrieval.web.main import SearchResult, get_filtered_results
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -26,6 +26,13 @@ def search_duckduckgo(
|
|||
Returns:
|
||||
list[SearchResult]: A list of search results
|
||||
"""
|
||||
if USE_SLIM:
|
||||
raise ValueError(
|
||||
'DDGS is unavailable in slim. Configure another web search provider in Admin Settings > Web Search.'
|
||||
)
|
||||
|
||||
from ddgs import DDGS
|
||||
|
||||
# The ddgs library (primp-based) does not auto-detect proxy env vars.
|
||||
# Resolve via stdlib getproxies() — same pattern as the other loaders.
|
||||
env_proxies = urllib.request.getproxies()
|
||||
|
|
|
|||
|
|
@ -1,6 +1,4 @@
|
|||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
from open_webui.retrieval.web.main import SearchResult
|
||||
|
|
@ -10,18 +8,12 @@ log = logging.getLogger(__name__)
|
|||
EXA_API_BASE = 'https://api.exa.ai'
|
||||
|
||||
|
||||
@dataclass
|
||||
class ExaResult:
|
||||
url: str
|
||||
title: str
|
||||
text: str
|
||||
|
||||
|
||||
def search_exa(
|
||||
api_key: str,
|
||||
query: str,
|
||||
count: int,
|
||||
filter_list: Optional[list[str]] = None,
|
||||
filter_list: list[str] | None = None,
|
||||
max_content_length: int | None = None,
|
||||
) -> list[SearchResult]:
|
||||
"""Search using Exa Search API and return the results as a list of SearchResult objects.
|
||||
|
||||
|
|
@ -29,7 +21,8 @@ def search_exa(
|
|||
api_key (str): A Exa Search API key
|
||||
query (str): The query to search for
|
||||
count (int): Number of results to return
|
||||
filter_list (Optional[list[str]]): List of domains to filter results by
|
||||
filter_list (list[str] | None): List of domains to filter results by
|
||||
max_content_length (int | None): Maximum characters per result; None leaves text unlimited.
|
||||
"""
|
||||
log.info('Searching with Exa for query: %s', query)
|
||||
|
||||
|
|
@ -39,7 +32,7 @@ def search_exa(
|
|||
'query': query,
|
||||
'numResults': count or 5,
|
||||
'includeDomains': filter_list,
|
||||
'contents': {'text': True, 'highlights': True},
|
||||
'contents': {'text': {'maxCharacters': max_content_length} if max_content_length is not None else True},
|
||||
'type': 'auto', # Use the auto search type (keyword or neural)
|
||||
}
|
||||
|
||||
|
|
@ -48,22 +41,13 @@ def search_exa(
|
|||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
results = []
|
||||
for result in data['results']:
|
||||
results.append(
|
||||
ExaResult(
|
||||
url=result['url'],
|
||||
title=result['title'],
|
||||
text=result['text'],
|
||||
)
|
||||
)
|
||||
|
||||
results = data['results']
|
||||
log.info('Found %s results', len(results))
|
||||
return [
|
||||
SearchResult(
|
||||
link=result.url,
|
||||
title=result.title,
|
||||
snippet=result.text,
|
||||
link=result['url'],
|
||||
title=result['title'],
|
||||
snippet=(result.get('text') or '')[:max_content_length],
|
||||
)
|
||||
for result in results
|
||||
]
|
||||
|
|
|
|||
|
|
@ -26,19 +26,26 @@ def search_searchapi(
|
|||
engine = engine or 'google'
|
||||
|
||||
payload = {'engine': engine, 'q': query, 'api_key': api_key}
|
||||
if engine.startswith('google'):
|
||||
payload['link'] = 'resolved'
|
||||
|
||||
url = f'{url}?{urlencode(payload)}'
|
||||
response = requests.request('GET', url)
|
||||
response = requests.request('GET', url, timeout=30)
|
||||
response.raise_for_status()
|
||||
|
||||
json_response = response.json()
|
||||
log.info('results from searchapi search: %s', json_response)
|
||||
log.debug('results from searchapi search: %s', json_response)
|
||||
|
||||
results = sorted(json_response.get('organic_results', []), key=lambda x: x.get('position', 0))
|
||||
# top_stories entries carry no position, so the merged list keeps API order
|
||||
results = [
|
||||
*json_response.get('organic_results', []),
|
||||
*json_response.get('top_stories', []),
|
||||
]
|
||||
if filter_list:
|
||||
results = get_filtered_results(results, filter_list)
|
||||
return [
|
||||
SearchResult(
|
||||
link=result['link'],
|
||||
link=result.get('link', ''),
|
||||
title=result.get('title'),
|
||||
snippet=result.get('snippet'),
|
||||
)
|
||||
|
|
|
|||
60
backend/open_webui/retrieval/web/staan.py
Normal file
60
backend/open_webui/retrieval/web/staan.py
Normal file
|
|
@ -0,0 +1,60 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import requests
|
||||
from open_webui.retrieval.web.main import SearchResult, get_filtered_results
|
||||
|
||||
|
||||
def search_staan(
|
||||
api_key: str,
|
||||
query: str,
|
||||
count: int,
|
||||
filter_list: list[str] | None = None,
|
||||
market: str | None = None,
|
||||
max_snippets: int | None = None,
|
||||
) -> list[SearchResult]:
|
||||
"""Search using Staan's Web Search API and return the results as a list of SearchResult objects.
|
||||
|
||||
Args:
|
||||
api_key (str): A Staan API key
|
||||
query (str): The query to search for
|
||||
count (int): The maximum number of results to return
|
||||
filter_list (list[str] | None): The domains to allow or block
|
||||
market (str | None): The market to search in, e.g. 'en-us'
|
||||
max_snippets (int | None): The maximum extra snippets to request per result
|
||||
|
||||
Returns:
|
||||
A list of SearchResult objects.
|
||||
"""
|
||||
url = 'https://api.staan.ai/v2/search/web'
|
||||
headers = {
|
||||
'Accept': 'application/json',
|
||||
'Authorization': f'Bearer {api_key}',
|
||||
}
|
||||
params = {'q': query, 'market': market}
|
||||
|
||||
if max_snippets:
|
||||
params['extra_snippets'] = 'true'
|
||||
params['max_snippets'] = max_snippets
|
||||
|
||||
response = requests.get(url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
|
||||
results = response.json().get('web', {}).get('results', [])
|
||||
if filter_list:
|
||||
results = get_filtered_results(results, filter_list)
|
||||
|
||||
return [
|
||||
SearchResult(
|
||||
link=result.get('url', ''),
|
||||
title=result.get('title'),
|
||||
snippet=_build_snippet(result),
|
||||
)
|
||||
for result in results[:count]
|
||||
]
|
||||
|
||||
|
||||
def _build_snippet(result: dict) -> str:
|
||||
"""Combine the snippet and the extra snippets list into a single string."""
|
||||
parts = [result.get('snippet')]
|
||||
parts.extend(extra.get('chunk') for extra in result.get('extra_snippets', []))
|
||||
return '\n\n'.join(part for part in parts if part)
|
||||
|
|
@ -8,6 +8,7 @@ import time
|
|||
import urllib.parse
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta
|
||||
from importlib import import_module
|
||||
from typing import (
|
||||
Any,
|
||||
AsyncIterator,
|
||||
|
|
@ -29,9 +30,9 @@ import urllib3.connection
|
|||
import urllib3.connectionpool
|
||||
import validators
|
||||
from requests.adapters import HTTPAdapter
|
||||
from fastapi import HTTPException
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from langchain_community.document_loaders import PlaywrightURLLoader, WebBaseLoader
|
||||
from langchain_community.document_loaders.base import BaseLoader
|
||||
from langchain_core.document_loaders import BaseLoader
|
||||
from langchain_core.documents import Document
|
||||
from open_webui.config import (
|
||||
ENABLE_LOCAL_WEB_FETCH,
|
||||
|
|
@ -58,6 +59,7 @@ from open_webui.env import (
|
|||
AIOHTTP_CLIENT_SSL_CERT_FILE,
|
||||
AIOHTTP_CLIENT_TIMEOUT,
|
||||
USER_AGENT,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.retrieval.loaders.external_web import ExternalWebLoader
|
||||
from open_webui.retrieval.loaders.microsoft_web_iq import MicrosoftWebIQLoader
|
||||
|
|
@ -128,7 +130,7 @@ def _assert_addresses_allowed(addresses: Sequence[str]) -> None:
|
|||
|
||||
def validate_url(url: Union[str, Sequence[str]]):
|
||||
if isinstance(url, str):
|
||||
if isinstance(validators.url(url), validators.ValidationError):
|
||||
if isinstance(validators.url(url, simple_host=ENABLE_LOCAL_WEB_FETCH), validators.ValidationError):
|
||||
raise ValueError(ERROR_MESSAGES.INVALID_URL)
|
||||
|
||||
# Reject parser-confusing chars: urlparse and requests/aiohttp split
|
||||
|
|
@ -302,6 +304,9 @@ _DROPPED_REQUEST_HEADERS = {'accept-encoding', 'connection', 'content-length', '
|
|||
# The clients hand us a decoded body, so the sender's framing no longer describes it.
|
||||
_DROPPED_RESPONSE_HEADERS = {'connection', 'content-encoding', 'content-length', 'transfer-encoding'}
|
||||
|
||||
# The Playwright loader only reads the page HTML, which none of these feed.
|
||||
_DROPPED_RESOURCE_TYPES = {'font', 'image', 'media'}
|
||||
|
||||
|
||||
def _forwardable_request_headers(headers: Dict[str, str]) -> Dict[str, str]:
|
||||
return {name: value for name, value in headers.items() if name.lower() not in _DROPPED_REQUEST_HEADERS}
|
||||
|
|
@ -643,7 +648,7 @@ class SafeMicrosoftWebIQLoader(BaseLoader, RateLimitMixin, URLProcessingMixin):
|
|||
raise e
|
||||
|
||||
|
||||
class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessingMixin):
|
||||
class SafePlaywrightURLLoader(BaseLoader, RateLimitMixin, URLProcessingMixin):
|
||||
"""Load HTML pages safely with Playwright, supporting SSL verification, rate limiting, and remote browser connection.
|
||||
|
||||
Attributes:
|
||||
|
|
@ -674,6 +679,13 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
playwright_timeout: Optional[int] = 10000,
|
||||
):
|
||||
"""Initialize with additional safety parameters and remote browser support."""
|
||||
if USE_SLIM:
|
||||
raise HTTPException(
|
||||
503, 'Playwright is unavailable in slim. Use basic HTTP fetching or an external web loader.'
|
||||
)
|
||||
|
||||
for package in ('playwright', 'unstructured'):
|
||||
import_module(package)
|
||||
|
||||
proxy_server = proxy.get('server') if proxy else None
|
||||
if trust_env and not proxy_server:
|
||||
|
|
@ -685,14 +697,11 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
else:
|
||||
proxy = {'server': env_proxy_server}
|
||||
|
||||
# We'll set headless to False if using playwright_ws_url since it's handled by the remote browser
|
||||
super().__init__(
|
||||
urls=web_paths,
|
||||
continue_on_failure=continue_on_failure,
|
||||
headless=headless if playwright_ws_url is None else False,
|
||||
remove_selectors=remove_selectors,
|
||||
proxy=proxy,
|
||||
)
|
||||
self.urls = web_paths
|
||||
self.continue_on_failure = continue_on_failure
|
||||
self.headless = headless if playwright_ws_url is None else False
|
||||
self.remove_selectors = remove_selectors or []
|
||||
self.proxy = proxy
|
||||
self.verify_ssl = verify_ssl
|
||||
self.requests_per_second = requests_per_second
|
||||
self.last_request_time = None
|
||||
|
|
@ -700,6 +709,12 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
self.trust_env = trust_env
|
||||
self.playwright_timeout = playwright_timeout
|
||||
|
||||
@staticmethod
|
||||
def _extract_html(html):
|
||||
from unstructured.partition.html import partition_html
|
||||
|
||||
return '\n\n'.join(str(element) for element in partition_html(text=html))
|
||||
|
||||
def _request_timeout(self) -> float:
|
||||
# per-hop budget, since page.goto's timeout cannot reach into our own fetch and 0 disables
|
||||
# it. aiohttp treats it as a total where requests only caps each read, so sync runs looser.
|
||||
|
|
@ -719,6 +734,9 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
|
||||
def _intercept_navigation_sync(self, route, session):
|
||||
req = route.request
|
||||
if req.resource_type in _DROPPED_RESOURCE_TYPES:
|
||||
route.abort()
|
||||
return
|
||||
|
||||
hop_cookies: List[Tuple[str, str]] = []
|
||||
|
||||
|
|
@ -773,6 +791,9 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
|
||||
async def _intercept_navigation(self, route, session):
|
||||
req = route.request
|
||||
if req.resource_type in _DROPPED_RESOURCE_TYPES:
|
||||
await route.abort()
|
||||
return
|
||||
|
||||
hop_cookies: List[Tuple[str, str]] = []
|
||||
|
||||
|
|
@ -847,12 +868,18 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
browser.new_page(service_workers='block') as page,
|
||||
):
|
||||
page.route('**/*', lambda route: self._intercept_navigation_sync(route, session))
|
||||
page.route_web_socket('**/*', lambda ws_route: ws_route.close())
|
||||
# sync close() hangs the dispatcher; a no-op handler still never connects to the server
|
||||
page.route_web_socket('**/*', lambda ws_route: None)
|
||||
response = page.goto(url, timeout=self.playwright_timeout)
|
||||
if response is None:
|
||||
raise ValueError(f'page.goto() returned None for url {url}')
|
||||
|
||||
text = self.evaluator.evaluate(page, browser, response)
|
||||
for selector in self.remove_selectors:
|
||||
for element in page.locator(selector).all():
|
||||
if element.is_visible():
|
||||
element.evaluate('element => element.remove()')
|
||||
text = self._extract_html(page.content())
|
||||
page.unroute_all(behavior='ignoreErrors')
|
||||
metadata = {'source': url}
|
||||
yield Document(page_content=text, metadata=metadata)
|
||||
except Exception as e:
|
||||
|
|
@ -887,7 +914,12 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
if response is None:
|
||||
raise ValueError(f'page.goto() returned None for url {url}')
|
||||
|
||||
text = await self.evaluator.evaluate_async(page, browser, response)
|
||||
for selector in self.remove_selectors:
|
||||
for element in await page.locator(selector).all():
|
||||
if await element.is_visible():
|
||||
await element.evaluate('element => element.remove()')
|
||||
text = await asyncio.to_thread(self._extract_html, await page.content())
|
||||
await page.unroute_all(behavior='ignoreErrors')
|
||||
metadata = {'source': url}
|
||||
yield Document(page_content=text, metadata=metadata)
|
||||
except Exception as e:
|
||||
|
|
@ -897,42 +929,58 @@ class SafePlaywrightURLLoader(PlaywrightURLLoader, RateLimitMixin, URLProcessing
|
|||
raise e
|
||||
|
||||
|
||||
class SafeWebBaseLoader(WebBaseLoader):
|
||||
"""WebBaseLoader with enhanced error handling for URLs."""
|
||||
class SafeWebBaseLoader(BaseLoader):
|
||||
"""Fetch pages with connect-time address checks and bounded concurrency."""
|
||||
|
||||
def __init__(self, trust_env: bool = False, *args, **kwargs):
|
||||
"""Initialize SafeWebBaseLoader
|
||||
Args:
|
||||
trust_env (bool, optional): set to True if using proxy to make web requests, for example
|
||||
using http(s)_proxy environment variables. Defaults to False.
|
||||
"""
|
||||
# lxml parses scraped pages far faster than the html.parser default
|
||||
kwargs.setdefault('default_parser', 'lxml')
|
||||
super().__init__(*args, **kwargs)
|
||||
def __init__(
|
||||
self,
|
||||
web_paths,
|
||||
verify_ssl=True,
|
||||
trust_env=False,
|
||||
requests_per_second=2,
|
||||
continue_on_failure=False,
|
||||
requests_kwargs=None,
|
||||
raise_for_status=False,
|
||||
default_parser='lxml',
|
||||
bs_kwargs=None,
|
||||
bs_get_text_kwargs=None,
|
||||
):
|
||||
self.web_paths = list(web_paths)
|
||||
self.trust_env = trust_env
|
||||
|
||||
# Propagate USER_AGENT env var so that both the sync _scrape() and
|
||||
# async _fetch() paths present a real UA instead of python-requests/2.x
|
||||
# which gets blocked by Cloudflare, Wikipedia, and similar bot-detection.
|
||||
# _fetch() forwards self.session.headers to the aiohttp session, so
|
||||
# setting it here covers both code-paths.
|
||||
if USER_AGENT:
|
||||
self.session.headers['User-Agent'] = USER_AGENT
|
||||
|
||||
# Prevent redirect-based SSRF on the synchronous _scrape() path.
|
||||
# validate_url() is called once on the originally-submitted URL, but the
|
||||
# parent WebBaseLoader's _scrape() invokes self.session.get(url, **self.requests_kwargs)
|
||||
# which by default follows redirects. Without the override below, an attacker
|
||||
# can submit a public URL that 302-redirects to an internal address (RFC1918,
|
||||
# 127.0.0.1, 169.254.169.254, etc.) and the redirected target is fetched without
|
||||
# re-validation. Matches the policy enforced on the async _fetch() path below.
|
||||
self.requests_kwargs = {
|
||||
**(self.requests_kwargs or {}),
|
||||
'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS,
|
||||
self.requests_per_second = requests_per_second
|
||||
self.continue_on_failure = continue_on_failure
|
||||
self.requests_kwargs = {**(requests_kwargs or {}), 'allow_redirects': AIOHTTP_CLIENT_ALLOW_REDIRECTS}
|
||||
self.raise_for_status = raise_for_status
|
||||
self.default_parser = default_parser
|
||||
self.bs_kwargs = bs_kwargs or {}
|
||||
self.bs_get_text_kwargs = bs_get_text_kwargs or {}
|
||||
# Preserve the synchronous loader's environment-proxy behavior.
|
||||
self.session = get_ssrf_safe_requests_session()
|
||||
self.session.verify = verify_ssl
|
||||
self.session.headers = {
|
||||
'User-Agent': USER_AGENT or 'DefaultLangchainUserAgent',
|
||||
'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
|
||||
'Accept-Language': 'en-US,en;q=0.5',
|
||||
'Referer': 'https://www.google.com/',
|
||||
'DNT': '1',
|
||||
'Connection': 'keep-alive',
|
||||
'Upgrade-Insecure-Requests': '1',
|
||||
}
|
||||
|
||||
self.session.mount('http://', _SSRFSafeAdapter())
|
||||
self.session.mount('https://', _SSRFSafeAdapter())
|
||||
async def fetch_all(self, urls):
|
||||
semaphore = asyncio.Semaphore(self.requests_per_second)
|
||||
|
||||
async def fetch(url):
|
||||
async with semaphore:
|
||||
try:
|
||||
return await self._fetch(url)
|
||||
except Exception as e:
|
||||
if not self.continue_on_failure:
|
||||
raise
|
||||
log.warning('Error fetching %s: %s', url, e)
|
||||
return ''
|
||||
|
||||
return await asyncio.gather(*(fetch(url) for url in urls))
|
||||
|
||||
async def _fetch(self, url: str, retries: int = 3, cooldown: int = 2, backoff: float = 1.5) -> str:
|
||||
connector = _SSRFSafeConnector()
|
||||
|
|
@ -948,10 +996,10 @@ class SafeWebBaseLoader(WebBaseLoader):
|
|||
else:
|
||||
kwargs['ssl'] = AIOHTTP_CLIENT_SESSION_SSL
|
||||
|
||||
async with session.get(
|
||||
url,
|
||||
**(self.requests_kwargs | kwargs),
|
||||
) as response:
|
||||
options = self.requests_kwargs | kwargs
|
||||
if isinstance(options.get('timeout'), (int, float)):
|
||||
options['timeout'] = aiohttp.ClientTimeout(total=options['timeout'])
|
||||
async with session.get(url, **options) as response:
|
||||
if self.raise_for_status:
|
||||
response.raise_for_status()
|
||||
return await response.text()
|
||||
|
|
@ -963,38 +1011,24 @@ class SafeWebBaseLoader(WebBaseLoader):
|
|||
await asyncio.sleep(cooldown * backoff**i)
|
||||
raise ValueError('retry count exceeded')
|
||||
|
||||
def _unpack_fetch_results(self, results: Any, urls: List[str], parser: Union[str, None] = None) -> List[Any]:
|
||||
"""Unpack fetch results into BeautifulSoup objects."""
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
final_results = []
|
||||
for i, result in enumerate(results):
|
||||
url = urls[i]
|
||||
url_parser = parser
|
||||
if url_parser is None:
|
||||
url_parser = 'xml' if url.endswith('.xml') else self.default_parser
|
||||
self._check_parser(url_parser)
|
||||
final_results.append(BeautifulSoup(result, url_parser, **self.bs_kwargs))
|
||||
return final_results
|
||||
|
||||
def lazy_load(self) -> Iterator[Document]:
|
||||
"""Lazy load text from the url(s) in web_path with error handling."""
|
||||
for path in self.web_paths:
|
||||
try:
|
||||
soup = self._scrape(path, bs_kwargs=self.bs_kwargs)
|
||||
text = soup.get_text(**self.bs_get_text_kwargs)
|
||||
|
||||
# Build metadata
|
||||
metadata = extract_metadata(soup, path)
|
||||
|
||||
yield Document(page_content=text, metadata=metadata)
|
||||
with self.session.get(path, **self.requests_kwargs) as response:
|
||||
if self.raise_for_status:
|
||||
response.raise_for_status()
|
||||
response.encoding = response.apparent_encoding
|
||||
yield self._document_from_html(response.text, path)
|
||||
except Exception as e:
|
||||
# Log the error and continue with the next URL
|
||||
log.exception(f'Error loading {path}: {e}')
|
||||
|
||||
def _document_from_html(self, html: str, url: str) -> Document:
|
||||
"""Build one Document."""
|
||||
soup = self._unpack_fetch_results([html], [url])[0]
|
||||
from bs4 import BeautifulSoup
|
||||
|
||||
parser = 'xml' if url.endswith('.xml') else self.default_parser
|
||||
soup = BeautifulSoup(html, parser, **self.bs_kwargs)
|
||||
return Document(
|
||||
page_content=soup.get_text(**self.bs_get_text_kwargs),
|
||||
metadata=extract_metadata(soup, url),
|
||||
|
|
|
|||
|
|
@ -46,6 +46,7 @@ from open_webui.env import (
|
|||
DEVICE_TYPE,
|
||||
ENABLE_FORWARD_USER_INFO_HEADERS,
|
||||
ENV,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.models.config import Config
|
||||
|
|
@ -58,9 +59,10 @@ from open_webui.utils.session_pool import get_session
|
|||
from pydantic import BaseModel
|
||||
|
||||
# pydub needs stdlib audioop (gone in 3.13); keep requires-python capped < 3.13
|
||||
from pydub import AudioSegment
|
||||
from pydub.silence import split_on_silence
|
||||
from pydub.utils import mediainfo
|
||||
if not USE_SLIM:
|
||||
from pydub import AudioSegment
|
||||
from pydub.silence import split_on_silence
|
||||
from pydub.utils import mediainfo
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
router = APIRouter()
|
||||
|
|
@ -213,6 +215,8 @@ def transcode_audio_to_mp3(audio_data: bytes, content_type_header: str, output_p
|
|||
|
||||
|
||||
def set_faster_whisper_model(model: str, auto_update: bool = False):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external speech-to-text engine. Local Whisper is unavailable in slim.')
|
||||
whisper_model = None
|
||||
if model:
|
||||
from faster_whisper import WhisperModel
|
||||
|
|
@ -285,14 +289,20 @@ async def get_audio_config(request: Request, user=Depends(get_admin_user)):
|
|||
|
||||
@router.post('/config/update')
|
||||
async def update_audio_config(request: Request, form_data: AudioConfigUpdateForm, user=Depends(get_admin_user)):
|
||||
if USE_SLIM:
|
||||
current = await Config.get_many('audio.stt.engine', 'audio.tts.engine')
|
||||
if form_data.stt.ENGINE == '' and current.get('audio.stt.engine') != '':
|
||||
raise HTTPException(400, 'Local Whisper is unavailable in slim. Select an external speech-to-text engine.')
|
||||
if form_data.tts.ENGINE == 'transformers' and current.get('audio.tts.engine') != 'transformers':
|
||||
raise HTTPException(400, 'Local TTS is unavailable in slim. Select an external text-to-speech engine.')
|
||||
await Config.upsert(
|
||||
{
|
||||
**config_updates(form_data.tts.model_dump(), TTS_CONFIG_KEYS),
|
||||
**config_updates(form_data.stt.model_dump(), STT_CONFIG_KEYS),
|
||||
**config_updates(form_data.tts.model_dump(exclude_unset=True), TTS_CONFIG_KEYS),
|
||||
**config_updates(form_data.stt.model_dump(exclude_unset=True), STT_CONFIG_KEYS),
|
||||
}
|
||||
)
|
||||
|
||||
if form_data.stt.ENGINE == '':
|
||||
if form_data.stt.ENGINE == '' and not USE_SLIM:
|
||||
request.app.state.faster_whisper_model = await asyncio.to_thread(
|
||||
set_faster_whisper_model, form_data.stt.WHISPER_MODEL, WHISPER_MODEL_AUTO_UPDATE
|
||||
)
|
||||
|
|
@ -314,6 +324,8 @@ async def update_audio_config(request: Request, form_data: AudioConfigUpdateForm
|
|||
|
||||
|
||||
def load_speech_pipeline(request):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external text-to-speech engine. Local TTS is unavailable in slim.')
|
||||
from datasets import load_dataset
|
||||
from transformers import pipeline
|
||||
|
||||
|
|
@ -328,7 +340,9 @@ def load_speech_pipeline(request):
|
|||
|
||||
async def _raise_tts_error(exc: Exception, r=None) -> None:
|
||||
"""Raise a standardised HTTPException from a TTS provider failure."""
|
||||
code = r.status if r is not None else 500
|
||||
if isinstance(exc, HTTPException):
|
||||
raise exc
|
||||
code = r.status if r is not None and r.status >= 400 else 500
|
||||
# LICENSE covers this Open WebUI error identifier.
|
||||
# Do not alter, remove, obscure, or replace it except as LICENSE permits:
|
||||
# https://docs.openwebui.com/license.
|
||||
|
|
@ -351,8 +365,29 @@ async def _write_tts_cache(
|
|||
audio: bytes,
|
||||
body_path: Path,
|
||||
payload: dict,
|
||||
content_type: str = 'audio/mpeg',
|
||||
) -> None:
|
||||
"""Persist audio + request metadata to the speech cache."""
|
||||
if USE_SLIM:
|
||||
mime_type = content_type.split(';')[0].strip().lower()
|
||||
if mime_type not in {
|
||||
'audio/mpeg',
|
||||
'audio/mp3',
|
||||
'audio/wav',
|
||||
'audio/x-wav',
|
||||
'audio/ogg',
|
||||
'audio/opus',
|
||||
'audio/webm',
|
||||
'audio/flac',
|
||||
'audio/aac',
|
||||
'audio/mp4',
|
||||
}:
|
||||
raise HTTPException(
|
||||
502,
|
||||
f'TTS returned unsupported format {mime_type}. Configure the provider to return MP3, WAV, Ogg, or another browser-playable audio format.',
|
||||
)
|
||||
async with aiofiles.open(file_path.with_suffix('.mime'), 'w') as f:
|
||||
await f.write(content_type)
|
||||
async with aiofiles.open(file_path, 'wb') as f:
|
||||
await f.write(audio)
|
||||
async with aiofiles.open(body_path, 'w') as f:
|
||||
|
|
@ -389,6 +424,10 @@ async def _tts_openai(request, payload, file_path, file_body_path, user):
|
|||
audio_data = await r.read()
|
||||
content_type = r.headers.get('Content-Type', 'audio/mpeg')
|
||||
|
||||
if USE_SLIM:
|
||||
await _write_tts_cache(file_path, audio_data, file_body_path, payload, content_type)
|
||||
return FileResponse(file_path, media_type=content_type)
|
||||
|
||||
if not await asyncio.to_thread(transcode_audio_to_mp3, audio_data, content_type, file_path):
|
||||
async with aiofiles.open(file_path, 'wb') as f:
|
||||
await f.write(audio_data)
|
||||
|
|
@ -430,8 +469,9 @@ async def _tts_elevenlabs(request, payload, file_path, file_body_path, user):
|
|||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload)
|
||||
return FileResponse(file_path)
|
||||
content_type = r.headers.get('Content-Type', 'audio/mpeg')
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload, content_type)
|
||||
return FileResponse(file_path, media_type=content_type if USE_SLIM else None)
|
||||
except Exception as exc:
|
||||
log.exception(exc)
|
||||
await _raise_tts_error(exc, r)
|
||||
|
|
@ -465,8 +505,9 @@ async def _tts_azure(request, payload, file_path, file_body_path, user):
|
|||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload)
|
||||
return FileResponse(file_path)
|
||||
content_type = r.headers.get('Content-Type', 'audio/mpeg')
|
||||
await _write_tts_cache(file_path, await r.read(), file_body_path, payload, content_type)
|
||||
return FileResponse(file_path, media_type=content_type if USE_SLIM else None)
|
||||
except Exception as exc:
|
||||
log.exception(exc)
|
||||
await _raise_tts_error(exc, r)
|
||||
|
|
@ -474,6 +515,8 @@ async def _tts_azure(request, payload, file_path, file_body_path, user):
|
|||
|
||||
async def _tts_transformers(request, payload, file_path, file_body_path, user):
|
||||
"""Generate speech via the local HuggingFace SpeechT5 pipeline (thread-offloaded)."""
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Configure an external text-to-speech engine. Local TTS is unavailable in slim.')
|
||||
import soundfile as sf
|
||||
import torch
|
||||
|
||||
|
|
@ -558,6 +601,8 @@ _TTS_ENGINES = {
|
|||
@router.post('/speech')
|
||||
async def speech(request: Request, user=Depends(get_verified_user)):
|
||||
engine = await Config.get('audio.tts.engine')
|
||||
if USE_SLIM and engine in ('', 'transformers'):
|
||||
raise HTTPException(503, 'Configure an external text-to-speech engine.')
|
||||
if engine == '':
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
|
|
@ -572,7 +617,10 @@ async def speech(request: Request, user=Depends(get_verified_user)):
|
|||
|
||||
body = await request.body()
|
||||
name = hashlib.sha256(
|
||||
body + str(engine).encode('utf-8') + str(await Config.get('audio.tts.model')).encode('utf-8')
|
||||
body
|
||||
+ str(engine).encode('utf-8')
|
||||
+ str(await Config.get('audio.tts.model')).encode('utf-8')
|
||||
+ (b':slim' if USE_SLIM else b'')
|
||||
).hexdigest()
|
||||
|
||||
file_path = SPEECH_CACHE_DIR.joinpath(f'{name}.mp3')
|
||||
|
|
@ -587,7 +635,11 @@ async def speech(request: Request, user=Depends(get_verified_user)):
|
|||
subject_id=name,
|
||||
data={'engine': engine, 'cached': True},
|
||||
)
|
||||
return FileResponse(file_path)
|
||||
content_type = None
|
||||
if USE_SLIM:
|
||||
async with aiofiles.open(file_path.with_suffix('.mime')) as f:
|
||||
content_type = await f.read()
|
||||
return FileResponse(file_path, media_type=content_type)
|
||||
|
||||
try:
|
||||
payload = JSONCodec.loads(body)
|
||||
|
|
@ -960,7 +1012,11 @@ async def _transcribe_mistral(request, file_path, filename, metadata, file_dir,
|
|||
session = await get_session()
|
||||
if use_chat_completions:
|
||||
audio_file_to_use = file_path
|
||||
if is_audio_conversion_required(file_path):
|
||||
if USE_SLIM and Path(filename).suffix.lower() not in ('.mp3', '.wav'):
|
||||
raise HTTPException(
|
||||
400, 'Mistral chat transcription requires MP3 or WAV in slim; local conversion is unavailable.'
|
||||
)
|
||||
if not BYPASS_PYDUB_PREPROCESSING and is_audio_conversion_required(file_path):
|
||||
log.debug('Converting audio to mp3 for chat completions API')
|
||||
converted_path = await asyncio.to_thread(convert_audio_to_mp3, file_path)
|
||||
if converted_path:
|
||||
|
|
|
|||
|
|
@ -75,10 +75,10 @@ from open_webui.utils.auth import (
|
|||
verify_password,
|
||||
)
|
||||
from open_webui.utils.groups import apply_default_group_assignment
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.misc import parse_duration, validate_email_format
|
||||
from open_webui.utils.rate_limit import RateLimiter
|
||||
from open_webui.utils.redis import get_redis_client
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, StrictStr, field_validator
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
|
|
@ -88,12 +88,11 @@ log = logging.getLogger(__name__)
|
|||
|
||||
# Forgive us our failed attempts, as we forgive those
|
||||
# who exceed their allotted rate against this gate.
|
||||
signin_rate_limiter = RateLimiter(redis_client=get_redis_client(), limit=5 * 3, window=60 * 3)
|
||||
signin_rate_limiter = RateLimiter(limit=5 * 3, window=60 * 3)
|
||||
# Best-effort throttle only: there is no caller identity before the provider answers,
|
||||
# and deployments may derive request.client from proxy headers.
|
||||
token_exchange_rate_limiter = (
|
||||
RateLimiter(
|
||||
redis_client=get_redis_client(),
|
||||
limit=OAUTH_TOKEN_EXCHANGE_RATE_LIMIT,
|
||||
window=OAUTH_TOKEN_EXCHANGE_RATE_LIMIT_WINDOW,
|
||||
)
|
||||
|
|
@ -106,6 +105,7 @@ ADMIN_CONFIG_KEYS = {
|
|||
'SHOW_ADMIN_DETAILS': 'auth.admin.show',
|
||||
'ADMIN_EMAIL': 'auth.admin.email',
|
||||
'WEBUI_URL': 'webui.url',
|
||||
'ENABLE_LOGIN_FORM': 'ui.enable_login_form',
|
||||
'ENABLE_SIGNUP': 'ui.enable_signup',
|
||||
'ENABLE_API_KEYS': 'auth.enable_api_keys',
|
||||
'ENABLE_API_KEYS_ENDPOINT_RESTRICTIONS': 'auth.api_key.endpoint_restrictions',
|
||||
|
|
@ -113,6 +113,7 @@ ADMIN_CONFIG_KEYS = {
|
|||
'DEFAULT_USER_ROLE': 'ui.default_user_role',
|
||||
'DEFAULT_GROUP_ID': 'ui.default_group_id',
|
||||
'DEFAULT_INTERFACE_SETTINGS': 'ui.default_interface_settings',
|
||||
'I18N': 'ui.i18n',
|
||||
'JWT_EXPIRES_IN': 'auth.jwt_expiry',
|
||||
'ENABLE_COMMUNITY_SHARING': 'ui.enable_community_sharing',
|
||||
'ENABLE_MESSAGE_RATING': 'ui.enable_message_rating',
|
||||
|
|
@ -814,7 +815,7 @@ async def signin(
|
|||
db=db,
|
||||
)
|
||||
else:
|
||||
if signin_rate_limiter.is_limited(form_data.email.lower()):
|
||||
if await signin_rate_limiter.is_limited(request.app.state.redis, form_data.email.lower()):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
detail=ERROR_MESSAGES.RATE_LIMIT_EXCEEDED,
|
||||
|
|
@ -1209,6 +1210,7 @@ class AdminConfig(BaseModel):
|
|||
SHOW_ADMIN_DETAILS: bool
|
||||
ADMIN_EMAIL: str | None = None
|
||||
WEBUI_URL: str
|
||||
ENABLE_LOGIN_FORM: bool = True
|
||||
ENABLE_SIGNUP: bool
|
||||
ENABLE_API_KEYS: bool
|
||||
ENABLE_API_KEYS_ENDPOINT_RESTRICTIONS: bool
|
||||
|
|
@ -1216,6 +1218,7 @@ class AdminConfig(BaseModel):
|
|||
DEFAULT_USER_ROLE: str
|
||||
DEFAULT_GROUP_ID: str
|
||||
DEFAULT_INTERFACE_SETTINGS: dict | None = None
|
||||
I18N: dict[str, dict[str, StrictStr]] | None = None
|
||||
JWT_EXPIRES_IN: str
|
||||
ENABLE_COMMUNITY_SHARING: bool
|
||||
ENABLE_MESSAGE_RATING: bool
|
||||
|
|
@ -1236,10 +1239,40 @@ class AdminConfig(BaseModel):
|
|||
PENDING_USER_OVERLAY_CONTENT: str | None = None
|
||||
RESPONSE_WATERMARK: str | None = None
|
||||
|
||||
@field_validator('I18N')
|
||||
@classmethod
|
||||
def validate_i18n(cls, value):
|
||||
if value is None:
|
||||
raise ValueError('I18N must be a dictionary')
|
||||
unsafe_keys = {'__proto__', 'prototype', 'constructor'}
|
||||
|
||||
def placeholders(text):
|
||||
return {match.strip() for match in re.findall(r'\{\{\s*-?\s*([^},]+)(?:,[^}]+)?\s*\}\}', text)}
|
||||
|
||||
cleaned = {}
|
||||
for locale, entries in value.items():
|
||||
if not locale.strip() or locale in unsafe_keys:
|
||||
raise ValueError(f'Invalid language: {locale}')
|
||||
translations = {}
|
||||
for key, text in entries.items():
|
||||
if not key.strip() or key in unsafe_keys:
|
||||
raise ValueError(f'Invalid translation key: {key}')
|
||||
if text.strip():
|
||||
if placeholders(key) != placeholders(text):
|
||||
raise ValueError(f'Interpolation placeholders do not match: {locale}: {key}')
|
||||
translations[key] = text
|
||||
if translations:
|
||||
cleaned[locale] = translations
|
||||
return cleaned
|
||||
|
||||
|
||||
@router.post('/admin/config')
|
||||
async def update_admin_config(request: Request, form_data: AdminConfig, user=Depends(get_admin_user)):
|
||||
updates = config_updates(form_data.model_dump(), ADMIN_CONFIG_KEYS)
|
||||
if 'ENABLE_LOGIN_FORM' not in form_data.model_fields_set:
|
||||
updates.pop('ui.enable_login_form', None)
|
||||
if 'I18N' not in form_data.model_fields_set:
|
||||
updates.pop('ui.i18n', None)
|
||||
updates['ui.default_interface_settings'] = form_data.DEFAULT_INTERFACE_SETTINGS or {}
|
||||
updates['folders.max_file_count'] = int(form_data.FOLDER_MAX_FILE_COUNT) if form_data.FOLDER_MAX_FILE_COUNT else ''
|
||||
updates['automations.max_count'] = int(form_data.AUTOMATION_MAX_COUNT) if form_data.AUTOMATION_MAX_COUNT else ''
|
||||
|
|
@ -1431,6 +1464,9 @@ OAUTH_CONFIG_KEYS = {
|
|||
|
||||
|
||||
def _format_oauth_form_value(field: str, value):
|
||||
if field == 'OAUTH_BLOCKED_GROUPS' and isinstance(value, list):
|
||||
# Preserve commas in group names and regex patterns when the form is saved.
|
||||
return JSONCodec.dumps(value)
|
||||
if field in OAUTH_COMMA_LIST_FIELDS and isinstance(value, list):
|
||||
return ','.join(str(item) for item in value)
|
||||
return value
|
||||
|
|
@ -1603,8 +1639,8 @@ async def token_exchange(
|
|||
detail='Token exchange is disabled',
|
||||
)
|
||||
|
||||
if token_exchange_rate_limiter and token_exchange_rate_limiter.is_limited(
|
||||
request.client.host if request.client else 'unknown'
|
||||
if token_exchange_rate_limiter and await token_exchange_rate_limiter.is_limited(
|
||||
request.app.state.redis, request.client.host if request.client else 'unknown'
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_429_TOO_MANY_REQUESTS,
|
||||
|
|
@ -1720,6 +1756,7 @@ async def token_exchange(
|
|||
user=user,
|
||||
user_data=user_data,
|
||||
provider=provider,
|
||||
access_token=form_data.token,
|
||||
db=db,
|
||||
)
|
||||
if await Config.get('oauth.enable_group_mapping'):
|
||||
|
|
|
|||
|
|
@ -87,7 +87,7 @@ async def check_automation_limits(request, user, rrule_str: str, db, is_create:
|
|||
if min_interval:
|
||||
min_interval = int(min_interval)
|
||||
if min_interval > 0:
|
||||
interval = rrule_interval_seconds(rrule_str)
|
||||
interval = await rrule_interval_seconds(rrule_str)
|
||||
if interval is not None and interval < min_interval:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
|
|
@ -150,7 +150,7 @@ async def enrich_automation(automation: AutomationModel, db: AsyncSession, tz: s
|
|||
return AutomationResponse(
|
||||
**automation.model_dump(),
|
||||
last_run=last_run,
|
||||
next_runs=next_n_runs_ns(automation.data['rrule'], tz=tz),
|
||||
next_runs=await next_n_runs_ns(automation.data['rrule'], tz=tz),
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -216,7 +216,7 @@ async def create_new_automation(
|
|||
await check_automation_folder_access(form_data.folder_id, user, db)
|
||||
await check_automation_channel_access(form_data, user, db)
|
||||
try:
|
||||
validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
await validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
|
|
@ -226,7 +226,7 @@ async def create_new_automation(
|
|||
await check_automation_limits(request, user, form_data.data.rrule, db, is_create=True)
|
||||
|
||||
tz = user.timezone
|
||||
automation = await Automations.insert(user.id, form_data, next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
automation = await Automations.insert(user.id, form_data, await next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
response = await enrich_automation(automation, db, tz=tz)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -276,7 +276,7 @@ async def update_automation_by_id(
|
|||
await check_automation_channel_access(form_data, user, db)
|
||||
|
||||
try:
|
||||
validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
await validate_rrule(form_data.data.rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
|
|
@ -286,7 +286,7 @@ async def update_automation_by_id(
|
|||
await check_automation_limits(request, user, form_data.data.rrule, db, is_create=False)
|
||||
|
||||
tz = user.timezone
|
||||
updated = await Automations.update_by_id(id, form_data, next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
updated = await Automations.update_by_id(id, form_data, await next_run_ns(form_data.data.rrule, tz=tz), db=db)
|
||||
response = await enrich_automation(updated, db, tz=tz)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -313,7 +313,7 @@ async def toggle_automation_by_id(
|
|||
await check_automations_permission(request, user)
|
||||
automation = await Automations.get_by_id(id, db=db)
|
||||
check_automation_access(automation, user)
|
||||
toggled = await Automations.toggle(id, next_run_ns(automation.data['rrule'], tz=user.timezone), db=db)
|
||||
toggled = await Automations.toggle(id, await next_run_ns(automation.data['rrule'], tz=user.timezone), db=db)
|
||||
response = await enrich_automation(toggled, db, tz=user.timezone)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
|
|||
|
|
@ -273,7 +273,10 @@ async def get_events(
|
|||
async def create_event(request: Request, form_data: CalendarEventForm, user: UserModel = Depends(get_verified_user)):
|
||||
await check_calendar_permission(request, user)
|
||||
await _check_calendar_access(form_data.calendar_id, user, 'write')
|
||||
event = await CalendarEvents.insert_new_event(user.id, form_data)
|
||||
try:
|
||||
event = await CalendarEvents.insert_new_event(user.id, form_data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=422, detail=str(e)) from e
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.CALENDAR_EVENT_CREATED,
|
||||
|
|
@ -325,7 +328,10 @@ async def update_event(
|
|||
if form_data.calendar_id is not None and form_data.calendar_id != event.calendar_id:
|
||||
await _check_calendar_access(form_data.calendar_id, user, 'write')
|
||||
|
||||
updated = await CalendarEvents.update_event_by_id(event_id, form_data)
|
||||
try:
|
||||
updated = await CalendarEvents.update_event_by_id(event_id, form_data)
|
||||
except ValueError as e:
|
||||
raise HTTPException(status_code=422, detail=str(e)) from e
|
||||
if not updated:
|
||||
raise HTTPException(status_code=500, detail='Failed to update')
|
||||
await publish_event(
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@ from fastapi.responses import FileResponse, Response, StreamingResponse
|
|||
from open_webui.config import ENABLE_ADMIN_CHAT_ACCESS, ENABLE_ADMIN_EXPORT
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.env import STATIC_DIR
|
||||
from open_webui.env import ENABLE_PROFILE_IMAGE_URL_FORWARDING, STATIC_DIR
|
||||
from open_webui.internal.db import get_async_session
|
||||
from open_webui.models.access_grants import AccessGrants, has_public_read_access_grant, has_public_write_access_grant
|
||||
from open_webui.models.config import Config
|
||||
|
|
@ -1102,13 +1102,12 @@ async def model_response_handler(request, channel, message, user, db=None):
|
|||
# Resolve model config (same path automations use)
|
||||
from open_webui.utils.automations import _resolve_model_defaults
|
||||
|
||||
tool_ids, features, filter_ids, _ = await _resolve_model_defaults(request.app, model_id)
|
||||
|
||||
# Build full form_data — same shape as frontend POST.
|
||||
# The channel: prefix routes pipeline events to the
|
||||
# channel emitter in socket/main.py instead of the
|
||||
# default chat emitter.
|
||||
form_data = {
|
||||
**await _resolve_model_defaults(request.app, model_id),
|
||||
'model': model_id,
|
||||
'messages': [
|
||||
system_message,
|
||||
|
|
@ -1122,12 +1121,6 @@ async def model_response_handler(request, channel, message, user, db=None):
|
|||
}
|
||||
if files:
|
||||
form_data['files'] = files
|
||||
if tool_ids:
|
||||
form_data['tool_ids'] = tool_ids
|
||||
if features:
|
||||
form_data['features'] = features
|
||||
if filter_ids:
|
||||
form_data['filter_ids'] = filter_ids
|
||||
|
||||
# Call the full chat completion pipeline — streaming,
|
||||
# tools, filters, RAG — everything. The pipeline runs as
|
||||
|
|
@ -1823,9 +1816,15 @@ async def delete_message_by_id(
|
|||
|
||||
|
||||
@router.get('/webhooks/{webhook_id}/profile/image')
|
||||
async def get_webhook_profile_image(webhook_id: str, user=Depends(get_verified_user)):
|
||||
async def get_webhook_profile_image(
|
||||
request: Request,
|
||||
webhook_id: str,
|
||||
user=Depends(get_verified_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
"""Get webhook profile image by webhook ID."""
|
||||
webhook = await Channels.get_webhook_by_id(webhook_id)
|
||||
await check_channels_access(request, user)
|
||||
webhook = await Channels.get_webhook_by_id(webhook_id, db=db)
|
||||
if not webhook:
|
||||
# Return default favicon if webhook not found
|
||||
# LICENSE covers this Open WebUI fallback logo.
|
||||
|
|
@ -1833,13 +1832,26 @@ async def get_webhook_profile_image(webhook_id: str, user=Depends(get_verified_u
|
|||
# https://docs.openwebui.com/license.
|
||||
return FileResponse(f'{STATIC_DIR}/favicon.png')
|
||||
|
||||
channel = await Channels.get_channel_by_id(webhook.channel_id, db=db)
|
||||
if not channel:
|
||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=ERROR_MESSAGES.NOT_FOUND)
|
||||
|
||||
if channel.type in ['group', 'dm']:
|
||||
if not await Channels.is_user_channel_member(channel.id, user.id, db=db):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.DEFAULT())
|
||||
else:
|
||||
if user.role != 'admin' and not await channel_has_access(user.id, channel, permission='read', db=db):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.DEFAULT())
|
||||
|
||||
if webhook.profile_image_url:
|
||||
# Check if it's url or base64
|
||||
if webhook.profile_image_url.startswith('http'):
|
||||
return Response(
|
||||
status_code=status.HTTP_302_FOUND,
|
||||
headers={'Location': webhook.profile_image_url},
|
||||
)
|
||||
if ENABLE_PROFILE_IMAGE_URL_FORWARDING:
|
||||
return Response(
|
||||
status_code=status.HTTP_302_FOUND,
|
||||
headers={'Location': webhook.profile_image_url},
|
||||
)
|
||||
# When forwarding is disabled, fall through to the default image to prevent client-side IP/UA/Referer leaks.
|
||||
elif webhook.profile_image_url.startswith('data:image'):
|
||||
try:
|
||||
header, base64_data = webhook.profile_image_url.split(',', 1)
|
||||
|
|
|
|||
|
|
@ -888,6 +888,7 @@ async def search_user_chats(
|
|||
created_at=chat.created_at,
|
||||
last_read_at=chat.last_read_at,
|
||||
snippet=chat_search_snippet(chat.chat, search_text),
|
||||
archived=chat.archived,
|
||||
)
|
||||
)
|
||||
|
||||
|
|
@ -1592,7 +1593,7 @@ async def delete_chat_by_id(
|
|||
# Cancel any in-flight LLM tasks (streaming, title/tags generation) before
|
||||
# deleting the chat to prevent orphaned requests.
|
||||
await stop_item_tasks(request.app.state.redis, id)
|
||||
await Chats.delete_orphan_tags_for_user(chat.meta.get('tags', []), user.id, threshold=1, db=db)
|
||||
await Chats.delete_orphan_tags_for_user(chat.meta.get('tags', []), chat.user_id, threshold=1, db=db)
|
||||
|
||||
# Cascade to internal child chats spawned from this one.
|
||||
for child_id in await Chats.get_internal_chat_ids_by_parent_id(id, chat.user_id):
|
||||
|
|
@ -1690,15 +1691,6 @@ async def fork_chat_by_id(
|
|||
|
||||
history = (chat.chat or {}).get('history') or {}
|
||||
messages_map = await Chats.get_messages_map_by_chat_id(id) or history.get('messages') or {}
|
||||
if any(
|
||||
message.get('role') == 'assistant' and message.get('done') is False
|
||||
for message in messages_map.values()
|
||||
if isinstance(message, dict)
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_409_CONFLICT,
|
||||
detail='Wait for the current response to finish before forking.',
|
||||
)
|
||||
|
||||
source_message_id = (
|
||||
(form_data.message_id if form_data else None) or chat.current_message_id or history.get('currentId')
|
||||
|
|
@ -1715,6 +1707,22 @@ async def fork_chat_by_id(
|
|||
detail=detail,
|
||||
) from exc
|
||||
|
||||
# An unfinished message is stale unless it is awaiting tool approval
|
||||
for message in fork_history['messages'].values():
|
||||
if message.get('role') != 'assistant' or message.get('done') is not False:
|
||||
continue
|
||||
|
||||
output = message.get('output')
|
||||
if isinstance(output, list) and any(
|
||||
isinstance(item, dict)
|
||||
and item.get('type') == 'function_call'
|
||||
and item.get('status') in {'pending', 'queued', 'requires_approval'}
|
||||
for item in output
|
||||
):
|
||||
continue
|
||||
|
||||
message['done'] = True
|
||||
|
||||
updated_chat = {**(chat.chat or {})}
|
||||
updated_chat.pop('currentId', None)
|
||||
updated_chat.update(
|
||||
|
|
@ -1732,10 +1740,15 @@ async def fork_chat_by_id(
|
|||
'forked_from_message_id': source_message_id,
|
||||
}
|
||||
|
||||
# The source chat's folder may no longer be writable by the caller.
|
||||
folder_id = chat.folder_id
|
||||
if folder_id is not None and not await has_folder_write_access(user.id, folder_id, db=db):
|
||||
folder_id = None
|
||||
|
||||
fork = await Chats.insert_new_chat(
|
||||
str(uuid4()),
|
||||
user.id,
|
||||
ChatForm(chat=updated_chat, folder_id=chat.folder_id),
|
||||
ChatForm(chat=updated_chat, folder_id=folder_id),
|
||||
db=db,
|
||||
internal_meta=meta,
|
||||
)
|
||||
|
|
|
|||
|
|
@ -1,8 +1,7 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
from typing import Optional
|
||||
from typing import Any, Optional
|
||||
|
||||
import aiohttp
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request
|
||||
|
|
@ -40,6 +39,7 @@ log = logging.getLogger(__name__)
|
|||
|
||||
CONNECTIONS_CONFIG_KEYS = {
|
||||
'ENABLE_DIRECT_CONNECTIONS': 'direct.enable',
|
||||
'ENABLE_DIRECT_INTEGRATIONS': 'direct.integrations.enable',
|
||||
'ENABLE_BASE_MODELS_CACHE': 'models.base_models_cache',
|
||||
}
|
||||
CODE_EXECUTION_CONFIG_KEYS = {
|
||||
|
|
@ -132,6 +132,7 @@ async def get_config_namespace(namespace: str, user=Depends(get_admin_user)):
|
|||
|
||||
class ConnectionsConfigForm(BaseModel):
|
||||
ENABLE_DIRECT_CONNECTIONS: bool
|
||||
ENABLE_DIRECT_INTEGRATIONS: bool = False
|
||||
ENABLE_BASE_MODELS_CACHE: bool
|
||||
|
||||
|
||||
|
|
@ -146,7 +147,7 @@ async def set_connections_config(
|
|||
form_data: ConnectionsConfigForm,
|
||||
user=Depends(get_admin_user),
|
||||
):
|
||||
await Config.upsert(config_updates(form_data.model_dump(), CONNECTIONS_CONFIG_KEYS))
|
||||
await Config.upsert(config_updates(form_data.model_dump(exclude_unset=True), CONNECTIONS_CONFIG_KEYS))
|
||||
values = await get_config_values(CONNECTIONS_CONFIG_KEYS)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -218,6 +219,7 @@ class ToolServerConnection(BaseModel):
|
|||
path: str
|
||||
type: str | None = 'openapi' # openapi, mcp
|
||||
auth_type: str | None
|
||||
forward_cookies: bool = False
|
||||
headers: dict | str | None = None
|
||||
key: str | None
|
||||
config: dict | None
|
||||
|
|
@ -307,6 +309,7 @@ class TerminalServerConnection(BaseModel):
|
|||
|
||||
key: str | None = ''
|
||||
auth_type: str | None = 'bearer'
|
||||
forward_cookies: bool = False
|
||||
|
||||
config: dict | None = None
|
||||
|
||||
|
|
@ -807,18 +810,25 @@ class PromptSuggestion(BaseModel):
|
|||
|
||||
|
||||
class SetDefaultSuggestionsForm(BaseModel):
|
||||
suggestions: list[PromptSuggestion]
|
||||
suggestions: list[PromptSuggestion] | None
|
||||
i18n: dict[str, Any] | None = None
|
||||
|
||||
|
||||
@router.post('/suggestions', response_model=list[PromptSuggestion])
|
||||
@router.post('/suggestions', response_model=dict)
|
||||
async def set_default_suggestions(
|
||||
request: Request,
|
||||
form_data: SetDefaultSuggestionsForm,
|
||||
user=Depends(get_admin_user),
|
||||
):
|
||||
data = form_data.model_dump()
|
||||
await Config.upsert({'ui.prompt_suggestions': data['suggestions']})
|
||||
await Config.upsert(
|
||||
{
|
||||
'ui.prompt_suggestions': data['suggestions'],
|
||||
'ui.prompt_suggestions_i18n': data.get('i18n') or {},
|
||||
}
|
||||
)
|
||||
suggestions = await Config.get('ui.prompt_suggestions')
|
||||
suggestions_i18n = await Config.get('ui.prompt_suggestions_i18n')
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.CONFIG_SUGGESTIONS_UPDATED,
|
||||
|
|
@ -827,7 +837,7 @@ async def set_default_suggestions(
|
|||
subject_type='config',
|
||||
data={'count': len(suggestions or [])},
|
||||
)
|
||||
return suggestions
|
||||
return {'suggestions': suggestions, 'i18n': suggestions_i18n}
|
||||
|
||||
|
||||
############################
|
||||
|
|
|
|||
|
|
@ -4,7 +4,10 @@ from typing import Optional
|
|||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from fastapi.concurrency import run_in_threadpool
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.env import MPS_INFERENCE_LOCK
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.env import USE_SLIM
|
||||
from open_webui.retrieval.utils import cosine_similarity
|
||||
from open_webui.internal.db import get_async_session
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.models.feedbacks import (
|
||||
|
|
@ -68,6 +71,8 @@ _embedding_model = None
|
|||
|
||||
def _get_embedding_model():
|
||||
global _embedding_model
|
||||
if USE_SLIM:
|
||||
return None
|
||||
if _embedding_model is None:
|
||||
try:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
|
@ -179,8 +184,9 @@ def _compute_similarities(feedbacks: list[LeaderboardFeedbackData], query: str)
|
|||
return {}
|
||||
|
||||
try:
|
||||
tag_embeddings = embedding_model.encode(all_tags)
|
||||
query_embedding = embedding_model.encode([query])[0]
|
||||
with MPS_INFERENCE_LOCK:
|
||||
tag_embeddings = embedding_model.encode(all_tags)
|
||||
query_embedding = embedding_model.encode([query])[0]
|
||||
except Exception as e:
|
||||
log.error(f'Embedding error: {e}')
|
||||
return {}
|
||||
|
|
@ -215,6 +221,7 @@ class LeaderboardResponse(BaseModel):
|
|||
|
||||
@router.get('/leaderboard', response_model=LeaderboardResponse)
|
||||
async def get_leaderboard(
|
||||
request: Request,
|
||||
query: Optional[str] = None,
|
||||
user=Depends(get_admin_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
|
|
@ -224,7 +231,17 @@ async def get_leaderboard(
|
|||
|
||||
similarities = None
|
||||
if query and query.strip():
|
||||
similarities = await run_in_threadpool(_compute_similarities, feedbacks, query.strip())
|
||||
if USE_SLIM:
|
||||
tags = list({tag for feedback in feedbacks for tag in (feedback.data or {}).get('tags', [])})
|
||||
embeddings = await request.app.state.EMBEDDING_FUNCTION([query.strip(), *tags], user=user)
|
||||
scores = cosine_similarity(embeddings[0], embeddings[1:])
|
||||
tag_scores = dict(zip(tags, scores.tolist()))
|
||||
similarities = {
|
||||
feedback.id: max((tag_scores.get(tag, 0) for tag in (feedback.data or {}).get('tags', [])), default=0)
|
||||
for feedback in feedbacks
|
||||
}
|
||||
else:
|
||||
similarities = await run_in_threadpool(_compute_similarities, feedbacks, query.strip())
|
||||
|
||||
elo_stats = _calculate_elo(feedbacks, similarities)
|
||||
tags_by_model = _get_top_tags(feedbacks)
|
||||
|
|
|
|||
|
|
@ -224,6 +224,15 @@ async def process_uploaded_file(
|
|||
f'{knowledge_id}: user {user.id} lacks write access'
|
||||
)
|
||||
else:
|
||||
directory_id = file_metadata.get('directory_id') or None
|
||||
if directory_id:
|
||||
directory = await Knowledges.get_directory_by_id(directory_id, db=db_session)
|
||||
if not directory or directory.knowledge_id != knowledge_id:
|
||||
log.warning(
|
||||
'Ignoring directory %s: not a directory of knowledge %s', directory_id, knowledge_id
|
||||
)
|
||||
directory_id = None
|
||||
|
||||
# Keep the generic file status stream open until the
|
||||
# KB-specific vector write and durable link both finish.
|
||||
await Files.update_file_data_by_id(file_item.id, {'status': 'processing'}, db=db_session)
|
||||
|
|
@ -237,7 +246,7 @@ async def process_uploaded_file(
|
|||
knowledge_id=knowledge_id,
|
||||
file_id=file_item.id,
|
||||
user_id=user.id,
|
||||
directory_id=file_metadata.get('directory_id'),
|
||||
directory_id=directory_id,
|
||||
db=db_session,
|
||||
)
|
||||
if not knowledge_file:
|
||||
|
|
@ -461,7 +470,11 @@ async def upload_file_handler(
|
|||
log.exception(e)
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=ERROR_MESSAGES.DEFAULT('Error uploading file'),
|
||||
detail=(
|
||||
ERROR_MESSAGES.EMPTY_CONTENT
|
||||
if isinstance(e, ValueError) and e.args == (ERROR_MESSAGES.EMPTY_CONTENT,)
|
||||
else ERROR_MESSAGES.DEFAULT('Error uploading file')
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -287,7 +287,12 @@ async def get_shared_folders(
|
|||
############################
|
||||
|
||||
|
||||
@router.get('/{id}', response_model=None)
|
||||
class FolderResponse(FolderModel):
|
||||
access_grants: list[dict] = []
|
||||
write_access: bool = False
|
||||
|
||||
|
||||
@router.get('/{id}', response_model=FolderResponse)
|
||||
async def get_folder_by_id(
|
||||
request: Request, id: str, user=Depends(get_verified_user), db: AsyncSession = Depends(get_async_session)
|
||||
):
|
||||
|
|
@ -295,13 +300,21 @@ async def get_folder_by_id(
|
|||
folder = await Folders.get_folder_by_id_and_user_id(id, user.id, db=db)
|
||||
if folder:
|
||||
grants = await AccessGrants.get_grants_by_resource('folder', id, db=db)
|
||||
return {**folder.model_dump(), 'access_grants': [g.model_dump() for g in grants]}
|
||||
return FolderResponse(
|
||||
**folder.model_dump(),
|
||||
access_grants=[g.model_dump() for g in grants],
|
||||
write_access=True,
|
||||
)
|
||||
|
||||
# Check shared access
|
||||
folder = await Folders.get_folder_by_id(id, db=db)
|
||||
if folder and (user.role == 'admin' or await _has_folder_access(user.id, folder, 'read', db)):
|
||||
grants = await AccessGrants.get_grants_by_resource('folder', id, db=db)
|
||||
return {**folder.model_dump(), 'access_grants': [g.model_dump() for g in grants]}
|
||||
return FolderResponse(
|
||||
**folder.model_dump(),
|
||||
access_grants=[g.model_dump() for g in grants],
|
||||
write_access=user.role == 'admin' or await _has_folder_access(user.id, folder, 'write', db),
|
||||
)
|
||||
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
|
|
@ -401,7 +414,7 @@ async def update_folder_parent_id_by_id(
|
|||
form_data.parent_id, user.id, folder.name, db=db
|
||||
)
|
||||
|
||||
if existing_folder:
|
||||
if existing_folder and existing_folder.id != id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_400_BAD_REQUEST,
|
||||
detail=ERROR_MESSAGES.DEFAULT('Folder already exists'),
|
||||
|
|
@ -683,7 +696,7 @@ async def delete_folder_by_id(
|
|||
folder_owner_id = folder.user_id
|
||||
|
||||
folder_ids = await Folders.get_folder_ids_by_id_and_user_id_in_subtree(id, folder_owner_id, db=db)
|
||||
if await Chats.count_chats_by_folder_ids_and_user_id(folder_ids, folder_owner_id, db=db):
|
||||
if delete_contents and await Chats.count_chats_by_folder_ids_and_user_id(folder_ids, folder_owner_id, db=db):
|
||||
chat_delete_permission = await has_permission(
|
||||
user.id, 'chat.delete', await Config.get('user.permissions'), db=db
|
||||
)
|
||||
|
|
@ -704,8 +717,8 @@ async def delete_folder_by_id(
|
|||
for folder_id in folder_ids:
|
||||
if delete_contents:
|
||||
await Chats.delete_chats_by_user_id_and_folder_id(folder_owner_id, folder_id, db=db)
|
||||
else:
|
||||
await Chats.move_chats_by_user_id_and_folder_id(folder_owner_id, folder_id, None, db=db)
|
||||
|
||||
await Chats.move_chats_by_folder_id(folder_id, None, db=db)
|
||||
|
||||
# Clean up access grants for this folder
|
||||
await AccessGrants.revoke_all_access('folder', folder_id, db=db)
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ from open_webui.models.functions import (
|
|||
)
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.plugin import (
|
||||
get_function_contents_cache,
|
||||
get_functions_cache,
|
||||
get_function_module_from_cache,
|
||||
load_function_module_by_id,
|
||||
|
|
@ -440,6 +441,8 @@ async def delete_function_by_id(
|
|||
if result:
|
||||
FUNCTIONS = get_functions_cache(request)
|
||||
FUNCTIONS.pop(id, None)
|
||||
FUNCTION_CONTENTS = get_function_contents_cache(request)
|
||||
FUNCTION_CONTENTS.pop(id, None)
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.FUNCTION_DELETED,
|
||||
|
|
|
|||
|
|
@ -329,38 +329,34 @@ def get_automatic1111_api_auth(image_config):
|
|||
return f'Basic {auth1111_base64_encoded_string}'
|
||||
|
||||
|
||||
@router.get('/config/url/verify')
|
||||
async def verify_url(request: Request, user=Depends(get_admin_user)):
|
||||
image_config = await get_image_config()
|
||||
if image_config.IMAGE_GENERATION_ENGINE == 'automatic1111':
|
||||
try:
|
||||
session = await get_session()
|
||||
async with session.get(
|
||||
url=f'{image_config.AUTOMATIC1111_BASE_URL}/sdapi/v1/options',
|
||||
headers={'authorization': get_automatic1111_api_auth(image_config)},
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
return True
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail=ERROR_MESSAGES.INVALID_URL)
|
||||
elif image_config.IMAGE_GENERATION_ENGINE == 'comfyui':
|
||||
headers = None
|
||||
if image_config.COMFYUI_API_KEY:
|
||||
headers = {'Authorization': f'Bearer {image_config.COMFYUI_API_KEY}'}
|
||||
try:
|
||||
session = await get_session()
|
||||
async with session.get(
|
||||
url=f'{image_config.COMFYUI_BASE_URL}/object_info',
|
||||
headers=headers,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as r:
|
||||
r.raise_for_status()
|
||||
return True
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail=ERROR_MESSAGES.INVALID_URL)
|
||||
class ConnectionVerificationForm(BaseModel):
|
||||
engine: str
|
||||
url: str
|
||||
key: str | None = None
|
||||
|
||||
|
||||
@router.post('/verify')
|
||||
async def verify_connection(form_data: ConnectionVerificationForm, user=Depends(get_admin_user)):
|
||||
url = form_data.url.rstrip('/')
|
||||
headers = {}
|
||||
if form_data.engine == 'automatic1111':
|
||||
url = f'{url}/sdapi/v1/options'
|
||||
if form_data.key is not None:
|
||||
headers['Authorization'] = f'Basic {base64.b64encode(form_data.key.encode("utf-8")).decode("utf-8")}'
|
||||
elif form_data.engine == 'comfyui':
|
||||
url = f'{url}/object_info'
|
||||
if form_data.key:
|
||||
headers['Authorization'] = f'Bearer {form_data.key}'
|
||||
else:
|
||||
return True
|
||||
raise HTTPException(status_code=400, detail='Unsupported image engine')
|
||||
|
||||
try:
|
||||
session = await get_session()
|
||||
async with session.get(url=url, headers=headers, ssl=AIOHTTP_CLIENT_SESSION_SSL) as r:
|
||||
r.raise_for_status()
|
||||
return True
|
||||
except Exception:
|
||||
raise HTTPException(status_code=400, detail=ERROR_MESSAGES.INVALID_URL)
|
||||
|
||||
|
||||
@router.get('/models')
|
||||
|
|
@ -436,7 +432,10 @@ async def get_models(request: Request, user=Depends(get_verified_user)):
|
|||
)
|
||||
)
|
||||
except Exception as e:
|
||||
log.exception(f'Failed to list image generation models: {e}')
|
||||
log.error(
|
||||
f'Failed to list image generation models: {str(e) or type(e).__name__}',
|
||||
exc_info=not isinstance(e, (aiohttp.ClientConnectionError, TimeoutError)),
|
||||
)
|
||||
raise HTTPException(
|
||||
status_code=400,
|
||||
detail=ERROR_MESSAGES.DEFAULT(e, 'Failed to retrieve image generation models'),
|
||||
|
|
@ -668,7 +667,9 @@ async def image_generations(
|
|||
if image_url := image.get('url', None):
|
||||
image_data, content_type = await get_image_data(
|
||||
image_url,
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'},
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'}
|
||||
if _is_same_origin(image_url, image_config.IMAGES_OPENAI_API_BASE_URL)
|
||||
else None,
|
||||
)
|
||||
else:
|
||||
image_data, content_type = await get_image_data(image['b64_json'])
|
||||
|
|
@ -919,11 +920,8 @@ async def image_edits(
|
|||
|
||||
if data.startswith('http://') or data.startswith('https://'):
|
||||
parsed = urlparse(data)
|
||||
if (
|
||||
parsed.netloc == urlparse(str(request.base_url)).netloc
|
||||
and parsed.path.startswith('/api/v1/files/')
|
||||
and '/content' in parsed.path
|
||||
):
|
||||
# Fetching /api/v1/files/{id}/content over the network would be unauthenticated.
|
||||
if parsed.path.startswith('/api/v1/files/') and '/content' in parsed.path:
|
||||
return await load_url_image(parsed.path)
|
||||
|
||||
# Validate URL to prevent SSRF attacks against local/private networks.
|
||||
|
|
@ -1047,7 +1045,9 @@ async def image_edits(
|
|||
if image_url := image.get('url', None):
|
||||
image_data, content_type = await get_image_data(
|
||||
image_url,
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'},
|
||||
{k: v for k, v in headers.items() if k != 'Content-Type'}
|
||||
if _is_same_origin(image_url, image_config.IMAGES_EDIT_OPENAI_API_BASE_URL)
|
||||
else None,
|
||||
)
|
||||
else:
|
||||
image_data, content_type = await get_image_data(image['b64_json'])
|
||||
|
|
|
|||
|
|
@ -151,6 +151,24 @@ def external_knowledge_error():
|
|||
)
|
||||
|
||||
|
||||
async def _verify_directory_in_knowledge(
|
||||
id: str,
|
||||
directory_id: str | None,
|
||||
db: AsyncSession,
|
||||
detail: str = ERROR_MESSAGES.NOT_FOUND,
|
||||
):
|
||||
"""Verify a caller-supplied directory belongs to the knowledge base in the URL. Unset means the root level."""
|
||||
if not directory_id:
|
||||
return None
|
||||
|
||||
directory = await Knowledges.get_directory_by_id(directory_id, db=db)
|
||||
if not directory or directory.knowledge_id != id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=detail,
|
||||
)
|
||||
|
||||
|
||||
@router.get('/', response_model=KnowledgeAccessListResponse)
|
||||
async def get_knowledge_bases(
|
||||
page: int | None = 1,
|
||||
|
|
@ -1437,6 +1455,8 @@ async def add_file_to_knowledge_by_id(
|
|||
detail=ERROR_MESSAGES.ACCESS_PROHIBITED,
|
||||
)
|
||||
|
||||
await _verify_directory_in_knowledge(id, form_data.directory_id, db, detail='Target directory not found.')
|
||||
|
||||
file = await Files.get_file_by_id(form_data.file_id, db=db)
|
||||
if not file:
|
||||
raise HTTPException(
|
||||
|
|
@ -1642,13 +1662,9 @@ async def remove_file_from_knowledge_by_id(
|
|||
|
||||
# Remove content from the vector database
|
||||
try:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(
|
||||
collection_name=knowledge.id, filter={'file_id': form_data.file_id}
|
||||
) # Remove by file_id first
|
||||
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(
|
||||
collection_name=knowledge.id, filter={'hash': file.hash}
|
||||
) # Remove by hash as well in case of duplicates
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=knowledge.id, filter={'file_id': form_data.file_id})
|
||||
if file.hash:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=knowledge.id, filter={'hash': file.hash})
|
||||
except Exception as e:
|
||||
log.debug('This was most likely caused by bypassing embedding processing')
|
||||
log.debug(e)
|
||||
|
|
@ -1988,7 +2004,8 @@ async def sync_knowledge_cleanup(
|
|||
|
||||
try:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'file_id': file_id})
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'hash': file.hash})
|
||||
if file.hash:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'hash': file.hash})
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
|
|
@ -2052,6 +2069,9 @@ async def add_files_to_knowledge_batch(
|
|||
detail=ERROR_MESSAGES.ACCESS_PROHIBITED,
|
||||
)
|
||||
|
||||
for directory_id in {form.directory_id for form in form_data if form.directory_id}:
|
||||
await _verify_directory_in_knowledge(id, directory_id, db, detail='Target directory not found.')
|
||||
|
||||
# Batch-fetch all files to avoid N+1 queries
|
||||
log.info('files/batch/add - %s files', len(form_data))
|
||||
file_ids = [form.file_id for form in form_data]
|
||||
|
|
@ -2240,6 +2260,8 @@ async def create_knowledge_directory(
|
|||
):
|
||||
await _verify_knowledge_write_access(id, user, db)
|
||||
|
||||
await _verify_directory_in_knowledge(id, form_data.parent_id, db, detail='Parent directory not found.')
|
||||
|
||||
directory = await Knowledges.create_directory(
|
||||
knowledge_id=id,
|
||||
name=form_data.name,
|
||||
|
|
@ -2272,14 +2294,11 @@ async def update_knowledge_directory(
|
|||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
await _verify_knowledge_write_access(id, user, db)
|
||||
await _verify_directory_in_knowledge(id, dir_id, db)
|
||||
|
||||
# Verify directory belongs to this knowledge base
|
||||
directory = await Knowledges.get_directory_by_id(dir_id, db=db)
|
||||
if not directory or directory.knowledge_id != id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=ERROR_MESSAGES.NOT_FOUND,
|
||||
)
|
||||
# '__unset__' leaves the parent alone, None moves the directory to the root
|
||||
if form_data.parent_id not in (None, '__unset__'):
|
||||
await _verify_directory_in_knowledge(id, form_data.parent_id, db, detail='Parent directory not found.')
|
||||
|
||||
result = await Knowledges.update_directory(
|
||||
directory_id=dir_id,
|
||||
|
|
@ -2312,14 +2331,10 @@ async def delete_knowledge_directory(
|
|||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
await _verify_knowledge_write_access(id, user, db)
|
||||
await _verify_directory_in_knowledge(id, dir_id, db)
|
||||
|
||||
# Verify directory belongs to this knowledge base
|
||||
directory = await Knowledges.get_directory_by_id(dir_id, db=db)
|
||||
if not directory or directory.knowledge_id != id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail=ERROR_MESSAGES.NOT_FOUND,
|
||||
)
|
||||
# Collect before delete_directory drops the KnowledgeFile rows
|
||||
files = [] if move_files else await Knowledges.get_files_by_id_and_directory_id(id, dir_id, db=db)
|
||||
|
||||
success = await Knowledges.delete_directory(
|
||||
directory_id=dir_id,
|
||||
|
|
@ -2331,6 +2346,23 @@ async def delete_knowledge_directory(
|
|||
status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
|
||||
detail='Failed to delete directory.',
|
||||
)
|
||||
|
||||
for file in files:
|
||||
try:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'file_id': file.id})
|
||||
if file.hash:
|
||||
await ASYNC_VECTOR_DB_CLIENT.delete(collection_name=id, filter={'hash': file.hash})
|
||||
except Exception as e:
|
||||
log.debug('This was most likely caused by bypassing embedding processing')
|
||||
log.debug(e)
|
||||
|
||||
if (
|
||||
not ENABLE_KNOWLEDGE_FILE_RETENTION
|
||||
and not await Knowledges.get_knowledges_by_file_id(file.id, db=db)
|
||||
and (file.user_id == user.id or user.role == 'admin')
|
||||
):
|
||||
await delete_file_resource(file, db)
|
||||
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.KNOWLEDGE_DIRECTORY_DELETED,
|
||||
|
|
@ -2358,14 +2390,7 @@ async def move_file_in_knowledge(
|
|||
detail=ERROR_MESSAGES.NOT_FOUND,
|
||||
)
|
||||
|
||||
# If target directory is set, verify it belongs to this knowledge base
|
||||
if form_data.directory_id:
|
||||
directory = await Knowledges.get_directory_by_id(form_data.directory_id, db=db)
|
||||
if not directory or directory.knowledge_id != id:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_404_NOT_FOUND,
|
||||
detail='Target directory not found.',
|
||||
)
|
||||
await _verify_directory_in_knowledge(id, form_data.directory_id, db, detail='Target directory not found.')
|
||||
|
||||
success = await Knowledges.move_file_to_directory(
|
||||
knowledge_id=id,
|
||||
|
|
|
|||
|
|
@ -241,6 +241,7 @@ async def update_memories(
|
|||
|
||||
operations = validate_memory_operations(form_data)
|
||||
metadata = getattr(request.state, 'metadata', {}) or {}
|
||||
model = metadata.get('model')
|
||||
source = form_data.source or 'tool'
|
||||
for operation in operations:
|
||||
if operation.get('action') in {'add', 'replace', 'move'}:
|
||||
|
|
@ -248,7 +249,7 @@ async def update_memories(
|
|||
'created_by': source,
|
||||
'chat_id': metadata.get('chat_id'),
|
||||
'message_id': metadata.get('message_id'),
|
||||
'model': metadata.get('model'),
|
||||
'model': model.get('id') if isinstance(model, dict) else None,
|
||||
}
|
||||
|
||||
try:
|
||||
|
|
@ -263,7 +264,7 @@ async def update_memories(
|
|||
for result in results:
|
||||
memory = result.get('memory')
|
||||
if isinstance(memory, MemoryModel):
|
||||
result = {**result, 'memory': memory.model_dump()}
|
||||
result = {**result, 'memory': memory.model_dump(exclude={'meta'})}
|
||||
if result.get('status') in {'created', 'updated'}:
|
||||
vector = await request.app.state.EMBEDDING_FUNCTION(
|
||||
memory_vector_text(memory.content, memory.path),
|
||||
|
|
@ -438,7 +439,7 @@ async def read_memory_path(
|
|||
)
|
||||
return {
|
||||
**result,
|
||||
'memories': [memory.model_dump() for memory in result['memories']],
|
||||
'memories': [memory.model_dump(exclude={'meta'}) for memory in result['memories']],
|
||||
}
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@ from fastapi import (
|
|||
APIRouter,
|
||||
Depends,
|
||||
HTTPException,
|
||||
Query,
|
||||
Request,
|
||||
Response,
|
||||
status,
|
||||
|
|
@ -19,11 +20,16 @@ from fastapi import (
|
|||
from fastapi.responses import RedirectResponse, StreamingResponse
|
||||
from open_webui.config import BYPASS_ADMIN_ACCESS_CONTROL
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.env import ENABLE_PROFILE_IMAGE_URL_FORWARDING, PROFILE_IMAGE_ALLOWED_MIME_TYPES
|
||||
from open_webui.env import (
|
||||
BYPASS_MODEL_ACCESS_CONTROL,
|
||||
ENABLE_PROFILE_IMAGE_URL_FORWARDING,
|
||||
PROFILE_IMAGE_ALLOWED_MIME_TYPES,
|
||||
)
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.internal.db import get_async_session
|
||||
from open_webui.models.access_grants import AccessGrants
|
||||
from open_webui.models.access_grants import AccessGrants, normalize_access_grants
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.models.files import Files
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.models import (
|
||||
ModelAccessListResponse,
|
||||
|
|
@ -36,12 +42,14 @@ from open_webui.models.models import (
|
|||
ModelResponse,
|
||||
Models,
|
||||
)
|
||||
from open_webui.utils.access_control import filter_allowed_access_grants, has_permission
|
||||
from open_webui.storage.provider import Storage
|
||||
from open_webui.utils.access_control import filter_allowed_access_grants, has_access, has_permission
|
||||
from open_webui.utils.access_control.files import has_access_to_file
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.chat_variables import get_chat_variables_schema
|
||||
from open_webui.utils.models import get_all_models
|
||||
from pydantic import BaseModel
|
||||
from open_webui.utils.validate import BACKGROUND_IMAGE_MAX_BYTES, validate_background_image
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -54,6 +62,8 @@ def add_chat_variables_schema(model_dict: dict) -> dict:
|
|||
schema = get_chat_variables_schema(system)
|
||||
if schema:
|
||||
model_dict.setdefault('meta', {})['chat_variables_schema'] = schema
|
||||
elif isinstance(model_dict.get('meta'), dict):
|
||||
model_dict['meta'].pop('chat_variables_schema', None)
|
||||
return model_dict
|
||||
|
||||
|
||||
|
|
@ -89,7 +99,28 @@ def _safe_static_redirect_path(url: str) -> str | None:
|
|||
|
||||
|
||||
def is_valid_model_id(model_id: str) -> bool:
|
||||
return model_id and len(model_id) <= 256
|
||||
return model_id and len(model_id) <= 256 and not any(char.isspace() for char in model_id)
|
||||
|
||||
|
||||
async def _verify_background_image(url: str | None, user, db, previous_url: str | None = None) -> None:
|
||||
if not url or url == previous_url:
|
||||
return
|
||||
file_id = url.split('/')[-2]
|
||||
file = await Files.get_file_by_id(file_id, db=db)
|
||||
if not file or not (
|
||||
user.role == 'admin' or file.user_id == user.id or await has_access_to_file(file_id, 'read', user, db=db)
|
||||
):
|
||||
raise HTTPException(status_code=403, detail='Background image is not accessible.')
|
||||
try:
|
||||
path = await asyncio.to_thread(Storage.get_file, file.path)
|
||||
with open(path, 'rb') as image:
|
||||
data = await asyncio.to_thread(image.read, BACKGROUND_IMAGE_MAX_BYTES + 1)
|
||||
content_type = await asyncio.to_thread(validate_background_image, data)
|
||||
except (ValueError, OSError) as error:
|
||||
raise HTTPException(status_code=400, detail=str(error)) from error
|
||||
if (file.meta or {}).get('content_type') != content_type:
|
||||
if not await Files.update_file_metadata_by_id(file_id, {'content_type': content_type}, db=db):
|
||||
raise HTTPException(status_code=500, detail='Could not validate background image.')
|
||||
|
||||
|
||||
async def _verify_knowledge_file_access(
|
||||
|
|
@ -211,6 +242,11 @@ async def get_models(
|
|||
###########################
|
||||
|
||||
|
||||
@router.get('/all', response_model=list[ModelResponse])
|
||||
async def get_all_model_records(user=Depends(get_admin_user), db: AsyncSession = Depends(get_async_session)):
|
||||
return await Models.get_all_models(db=db)
|
||||
|
||||
|
||||
@router.get('/base/tags', response_model=list[str])
|
||||
async def get_base_model_tags(user=Depends(get_admin_user), db: AsyncSession = Depends(get_async_session)):
|
||||
tags = await Models.get_all_tags(user_id=user.id, is_admin=True, is_base_model=True, db=db)
|
||||
|
|
@ -306,6 +342,8 @@ async def create_new_model(
|
|||
db,
|
||||
)
|
||||
|
||||
await _verify_background_image(form_data.meta.background_image_url, user, db)
|
||||
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
|
|
@ -336,9 +374,14 @@ async def create_new_model(
|
|||
############################
|
||||
|
||||
|
||||
@router.get('/export', response_model=list[ModelModel])
|
||||
class ModelExportResponse(ModelModel):
|
||||
background_image_data: str | None = None
|
||||
|
||||
|
||||
@router.get('/export', response_model=list[ModelExportResponse])
|
||||
async def export_models(
|
||||
request: Request,
|
||||
ids: list[str] | None = Query(None),
|
||||
user=Depends(get_verified_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
|
|
@ -354,9 +397,36 @@ async def export_models(
|
|||
)
|
||||
|
||||
if user.role == 'admin' and BYPASS_ADMIN_ACCESS_CONTROL:
|
||||
return await Models.get_models(db=db)
|
||||
models = await Models.get_models(db=db, ids=ids)
|
||||
else:
|
||||
return await Models.get_models(writable_by_user_id=user.id, db=db)
|
||||
models = await Models.get_models(writable_by_user_id=user.id, db=db, ids=ids)
|
||||
if ids is not None:
|
||||
requested = set(ids)
|
||||
if requested != {model.id for model in models}:
|
||||
raise HTTPException(status_code=403, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
exported = []
|
||||
for model in models:
|
||||
data = model.model_dump()
|
||||
url = model.meta.background_image_url
|
||||
if url:
|
||||
try:
|
||||
file = await Files.get_file_by_id(url.split('/')[-2], db=db)
|
||||
if not file:
|
||||
raise ValueError('Image file is missing')
|
||||
path = await asyncio.to_thread(Storage.get_file, file.path)
|
||||
with open(path, 'rb') as image:
|
||||
image_data = await asyncio.to_thread(image.read, BACKGROUND_IMAGE_MAX_BYTES + 1)
|
||||
content_type = await asyncio.to_thread(validate_background_image, image_data)
|
||||
data['background_image_data'] = f'data:{content_type};base64,' + base64.b64encode(image_data).decode(
|
||||
'ascii'
|
||||
)
|
||||
data['meta']['background_image_url'] = None
|
||||
except Exception as error:
|
||||
raise HTTPException(
|
||||
status_code=400, detail=f'Could not export background for model {model.id}.'
|
||||
) from error
|
||||
exported.append(data)
|
||||
return exported
|
||||
|
||||
|
||||
############################
|
||||
|
|
@ -486,7 +556,7 @@ async def import_models(
|
|||
updated_model.access_grants,
|
||||
'sharing.public_models',
|
||||
)
|
||||
await Models.update_model_by_id(model_id, updated_model, db=db)
|
||||
imported_model = updated_model
|
||||
else:
|
||||
# Insert new model
|
||||
model_data['meta'] = model_data.get('meta', {})
|
||||
|
|
@ -530,7 +600,58 @@ async def import_models(
|
|||
new_model.access_grants,
|
||||
'sharing.public_models',
|
||||
)
|
||||
await Models.insert_new_model(user_id=user.id, form_data=new_model, db=db)
|
||||
imported_model = new_model
|
||||
|
||||
uploaded = None
|
||||
try:
|
||||
encoded = model_data.pop('background_image_data', None)
|
||||
if encoded is not None:
|
||||
if (
|
||||
not isinstance(encoded, str)
|
||||
or len(encoded) > 4 * ((BACKGROUND_IMAGE_MAX_BYTES + 2) // 3) + 64
|
||||
):
|
||||
raise ValueError('Background image must be at most 5 MiB.')
|
||||
header, payload = encoded.split(',', 1)
|
||||
image_data = base64.b64decode(payload, validate=True)
|
||||
content_type = await asyncio.to_thread(validate_background_image, image_data)
|
||||
if header != f'data:{content_type};base64':
|
||||
raise ValueError('Invalid background image data URI.')
|
||||
from fastapi import UploadFile
|
||||
from open_webui.routers.files import upload_file_handler
|
||||
|
||||
uploaded = await upload_file_handler(
|
||||
request,
|
||||
file=UploadFile(
|
||||
file=io.BytesIO(image_data),
|
||||
filename='background.' + content_type.split('/')[1],
|
||||
),
|
||||
metadata=None,
|
||||
process=False,
|
||||
user=user,
|
||||
db=db,
|
||||
)
|
||||
imported_model.meta.background_image_url = f'/api/v1/files/{uploaded.id}/content'
|
||||
await _verify_background_image(
|
||||
imported_model.meta.background_image_url,
|
||||
user,
|
||||
db,
|
||||
existing_model.meta.background_image_url if existing_model else None,
|
||||
)
|
||||
saved = (
|
||||
await Models.update_model_by_id(model_id, imported_model, db=db)
|
||||
if existing_model
|
||||
else await Models.insert_new_model(user_id=user.id, form_data=imported_model, db=db)
|
||||
)
|
||||
if not saved:
|
||||
raise HTTPException(status_code=500, detail=f'Could not import model {model_id}.')
|
||||
except Exception:
|
||||
if uploaded:
|
||||
try:
|
||||
await Files.delete_file_by_id(uploaded.id, db=db)
|
||||
await asyncio.to_thread(Storage.delete_file, uploaded.path)
|
||||
except Exception:
|
||||
log.exception('Could not clean up failed model background upload')
|
||||
raise
|
||||
|
||||
imported_ids.append(model_id)
|
||||
await publish_event(
|
||||
|
|
@ -543,6 +664,10 @@ async def import_models(
|
|||
return True
|
||||
else:
|
||||
raise HTTPException(status_code=400, detail='Invalid JSON format')
|
||||
except HTTPException:
|
||||
raise
|
||||
except ValueError as error:
|
||||
raise HTTPException(status_code=400, detail=str(error)) from error
|
||||
except Exception as e:
|
||||
log.exception(e)
|
||||
raise HTTPException(status_code=500, detail=str(e))
|
||||
|
|
@ -564,6 +689,14 @@ async def sync_models(
|
|||
user=Depends(get_admin_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
existing = {model.id: model for model in await Models.get_models_by_ids([m.id for m in form_data.models], db=db)}
|
||||
for model in form_data.models:
|
||||
previous = existing.get(model.id)
|
||||
if previous and 'background_image_url' not in model.meta.model_fields_set:
|
||||
model.meta.background_image_url = previous.meta.background_image_url
|
||||
await _verify_background_image(
|
||||
model.meta.background_image_url, user, db, previous.meta.background_image_url if previous else None
|
||||
)
|
||||
models = await Models.sync_models(user.id, form_data.models, db=db)
|
||||
await publish_event(
|
||||
request,
|
||||
|
|
@ -649,18 +782,37 @@ async def get_model_profile_image(
|
|||
profile_image_url = None
|
||||
updated_at = None
|
||||
|
||||
bypass_access_control = BYPASS_MODEL_ACCESS_CONTROL or (user.role == 'admin' and BYPASS_ADMIN_ACCESS_CONTROL)
|
||||
|
||||
# First, check the database for regular models
|
||||
model_meta = await Models.get_model_meta_by_id(id, db=db)
|
||||
if model_meta:
|
||||
meta, updated_at = model_meta
|
||||
profile_image_url = (meta or {}).get('profile_image_url')
|
||||
meta, model_user_id, model_updated_at = model_meta
|
||||
# Denied callers get the default image rather than an error, so model ids stay unprobeable.
|
||||
if (
|
||||
bypass_access_control
|
||||
or user.id == model_user_id
|
||||
or await AccessGrants.has_access(
|
||||
user_id=user.id,
|
||||
resource_type='model',
|
||||
resource_id=id,
|
||||
permission='read',
|
||||
db=db,
|
||||
)
|
||||
):
|
||||
profile_image_url = (meta or {}).get('profile_image_url')
|
||||
updated_at = model_updated_at
|
||||
|
||||
# Fallback: check arena models stored in config (not in the DB)
|
||||
if not profile_image_url:
|
||||
arena_models = await Config.get('evaluation.arena.models', []) or []
|
||||
for arena_model in arena_models:
|
||||
if arena_model.get('id') == id:
|
||||
profile_image_url = arena_model.get('meta', {}).get('profile_image_url')
|
||||
arena_meta = arena_model.get('meta', {})
|
||||
if bypass_access_control or await has_access(
|
||||
user.id, permission='read', access_grants=arena_meta.get('access_grants', []), db=db
|
||||
):
|
||||
profile_image_url = arena_meta.get('profile_image_url')
|
||||
break
|
||||
|
||||
if profile_image_url:
|
||||
|
|
@ -829,13 +981,33 @@ async def update_model_by_id(
|
|||
if 'profile_image_url' not in form_data.meta.model_fields_set:
|
||||
form_data.meta.profile_image_url = model.meta.profile_image_url
|
||||
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
user.role,
|
||||
form_data.access_grants,
|
||||
'sharing.public_models',
|
||||
)
|
||||
if 'background_image_url' not in form_data.meta.model_fields_set:
|
||||
form_data.meta.background_image_url = model.meta.background_image_url
|
||||
await _verify_background_image(form_data.meta.background_image_url, user, db, model.meta.background_image_url)
|
||||
|
||||
if form_data.access_grants is not None:
|
||||
# The editor resends every stored grant, so re-checking them would strip sharing this user cannot re-create.
|
||||
existing_access_grants = {
|
||||
(grant.principal_type, grant.principal_id, grant.permission) for grant in model.access_grants
|
||||
}
|
||||
submitted_access_grants_map = {
|
||||
(grant['principal_type'], grant['principal_id'], grant['permission']): grant
|
||||
for grant in normalize_access_grants(form_data.access_grants)
|
||||
}
|
||||
preserved_access_grants = [
|
||||
grant for key, grant in submitted_access_grants_map.items() if key in existing_access_grants
|
||||
]
|
||||
new_access_grants = [
|
||||
grant for key, grant in submitted_access_grants_map.items() if key not in existing_access_grants
|
||||
]
|
||||
|
||||
form_data.access_grants = preserved_access_grants + await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
user.role,
|
||||
new_access_grants,
|
||||
'sharing.public_models',
|
||||
)
|
||||
|
||||
model = await Models.update_model_by_id(form_data.id, ModelForm(**form_data.model_dump()), db=db)
|
||||
if model:
|
||||
|
|
@ -855,7 +1027,7 @@ async def update_model_by_id(
|
|||
|
||||
|
||||
class ModelAccessGrantsForm(BaseModel):
|
||||
id: str
|
||||
id: str = Field(pattern=r'^\S+$')
|
||||
name: str | None = None
|
||||
access_grants: list[dict]
|
||||
|
||||
|
|
|
|||
|
|
@ -554,14 +554,15 @@ async def update_note_by_id(
|
|||
):
|
||||
raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.DEFAULT())
|
||||
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
user.role,
|
||||
form_data.access_grants,
|
||||
'sharing.public_notes',
|
||||
db=db,
|
||||
)
|
||||
if form_data.access_grants is not None:
|
||||
form_data.access_grants = await filter_allowed_access_grants(
|
||||
await Config.get('user.permissions'),
|
||||
user.id,
|
||||
user.role,
|
||||
form_data.access_grants,
|
||||
'sharing.public_notes',
|
||||
db=db,
|
||||
)
|
||||
|
||||
try:
|
||||
note = await Notes.update_note_by_id(id, form_data, db=db)
|
||||
|
|
|
|||
|
|
@ -55,14 +55,15 @@ log = logging.getLogger(__name__)
|
|||
# response body. Forwarding them verbatim causes desktop / programmatic
|
||||
# clients to attempt decompression of an already-decoded payload, resulting
|
||||
# in ZlibError. See https://github.com/aio-libs/aiohttp/issues/4462.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'Content-Encoding', 'Content-Length', 'Transfer-Encoding'})
|
||||
# Also drop server and date: uvicorn adds its own and forwarding both duplicates them.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'content-encoding', 'content-length', 'transfer-encoding', 'server', 'date'})
|
||||
_MODEL_LIST_TIMEOUT = aiohttp.ClientTimeout(total=AIOHTTP_CLIENT_TIMEOUT_MODEL_LIST)
|
||||
BASE_MODELS_CACHE_KEY = f'{REDIS_KEY_PREFIX}:models:base'
|
||||
|
||||
|
||||
def _clean_proxy_headers(raw_headers) -> dict:
|
||||
"""Return a copy of *raw_headers* with stale encoding headers removed."""
|
||||
return {k: v for k, v in raw_headers.items() if k not in _STRIP_PROXY_HEADERS}
|
||||
"""Return a copy of *raw_headers* without the encoding, server and date headers."""
|
||||
return {k: v for k, v in raw_headers.items() if k.lower() not in _STRIP_PROXY_HEADERS}
|
||||
|
||||
|
||||
async def send_get_request(
|
||||
|
|
@ -119,7 +120,7 @@ async def send_request(
|
|||
}
|
||||
|
||||
if ENABLE_FORWARD_USER_INFO_HEADERS and user:
|
||||
headers = include_user_info_headers(headers, user)
|
||||
headers = include_user_info_headers(headers, user, request=request)
|
||||
if metadata and metadata.get('chat_id'):
|
||||
headers[FORWARD_SESSION_INFO_HEADER_CHAT_ID] = metadata.get('chat_id')
|
||||
|
||||
|
|
@ -478,13 +479,16 @@ async def get_filtered_models(models, user, db=None):
|
|||
|
||||
|
||||
@router.get('/api/tags')
|
||||
@router.get('/api/tags/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/api/tags/{url_idx}')
|
||||
async def get_ollama_tags(
|
||||
request: Request,
|
||||
url_idx: int | None = None,
|
||||
user=Depends(get_verified_user),
|
||||
):
|
||||
"""List Ollama model tags, optionally from a specific backend."""
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if not await Config.get('ollama.enable'):
|
||||
raise HTTPException(status_code=503, detail=ERROR_MESSAGES.OLLAMA_API_DISABLED)
|
||||
|
||||
|
|
@ -541,13 +545,16 @@ async def get_ollama_loaded_models(
|
|||
|
||||
|
||||
@router.get('/api/version')
|
||||
@router.get('/api/version/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/api/version/{url_idx}')
|
||||
async def get_ollama_versions(
|
||||
request: Request,
|
||||
user=Depends(get_verified_user),
|
||||
url_idx: int | None = None,
|
||||
):
|
||||
"""Return the lowest Ollama version across all configured backends."""
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if not await Config.get('ollama.enable'):
|
||||
return {'version': False}
|
||||
|
||||
|
|
@ -1479,7 +1486,7 @@ async def generate_responses(
|
|||
|
||||
|
||||
@router.get('/v1/models')
|
||||
@router.get('/v1/models/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/v1/models/{url_idx}')
|
||||
async def get_openai_models(
|
||||
request: Request,
|
||||
url_idx: int | None = None,
|
||||
|
|
@ -1487,6 +1494,9 @@ async def get_openai_models(
|
|||
db: AsyncSession = Depends(get_async_session),
|
||||
) -> dict:
|
||||
"""List models in the OpenAI-compatible format."""
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if url_idx is None:
|
||||
model_list = await get_all_models(request, user=user)
|
||||
raw_models = model_list['models']
|
||||
|
|
|
|||
|
|
@ -74,15 +74,16 @@ log = logging.getLogger(__name__)
|
|||
# response body. Forwarding them verbatim causes desktop / programmatic
|
||||
# clients to attempt decompression of an already-decoded payload, resulting
|
||||
# in ZlibError. See https://github.com/aio-libs/aiohttp/issues/4462.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'Content-Encoding', 'Content-Length', 'Transfer-Encoding'})
|
||||
# Also drop server and date: uvicorn adds its own and forwarding both duplicates them.
|
||||
_STRIP_PROXY_HEADERS = frozenset({'content-encoding', 'content-length', 'transfer-encoding', 'server', 'date'})
|
||||
_MODEL_LIST_TIMEOUT = aiohttp.ClientTimeout(total=AIOHTTP_CLIENT_TIMEOUT_MODEL_LIST)
|
||||
_UNSUPPORTED_OPENAI_MODEL_KEYWORDS = ('babbage', 'dall-e', 'davinci', 'embedding', 'tts', 'whisper')
|
||||
BASE_MODELS_CACHE_KEY = f'{REDIS_KEY_PREFIX}:models:base'
|
||||
|
||||
|
||||
def _clean_proxy_headers(raw_headers) -> dict:
|
||||
"""Return a copy of *raw_headers* with stale encoding headers removed."""
|
||||
return {k: v for k, v in raw_headers.items() if k not in _STRIP_PROXY_HEADERS}
|
||||
"""Return a copy of *raw_headers* without the encoding, server and date headers."""
|
||||
return {k: v for k, v in raw_headers.items() if k.lower() not in _STRIP_PROXY_HEADERS}
|
||||
|
||||
|
||||
async def send_get_request(
|
||||
|
|
@ -103,7 +104,7 @@ async def send_get_request(
|
|||
cookies = None
|
||||
|
||||
if ENABLE_FORWARD_USER_INFO_HEADERS and user:
|
||||
headers = include_user_info_headers(headers, user)
|
||||
headers = include_user_info_headers(headers, user, request=request)
|
||||
|
||||
async with session.get(
|
||||
url,
|
||||
|
|
@ -159,7 +160,7 @@ async def get_headers_and_cookies(
|
|||
metadata: dict | None = None,
|
||||
user: UserModel = None,
|
||||
):
|
||||
cookies = {}
|
||||
cookies = getattr(request, 'cookies', {}) if config.get('forward_cookies', False) else {}
|
||||
headers = {
|
||||
'Content-Type': 'application/json',
|
||||
**(
|
||||
|
|
@ -176,7 +177,7 @@ async def get_headers_and_cookies(
|
|||
}
|
||||
|
||||
if ENABLE_FORWARD_USER_INFO_HEADERS and user:
|
||||
headers = include_user_info_headers(headers, user)
|
||||
headers = include_user_info_headers(headers, user, request=request)
|
||||
if metadata and metadata.get('chat_id'):
|
||||
headers[FORWARD_SESSION_INFO_HEADER_CHAT_ID] = metadata.get('chat_id')
|
||||
|
||||
|
|
@ -189,11 +190,8 @@ async def get_headers_and_cookies(
|
|||
elif auth_type == 'none':
|
||||
token = None
|
||||
elif auth_type == 'session':
|
||||
cookies = request.cookies
|
||||
token = request.state.token.credentials
|
||||
elif auth_type == 'system_oauth':
|
||||
cookies = request.cookies
|
||||
|
||||
oauth_token = None
|
||||
try:
|
||||
if request.cookies.get('oauth_session_id', None):
|
||||
|
|
@ -864,8 +862,11 @@ async def get_all_models(request: Request, user: UserModel) -> dict[str, list]:
|
|||
|
||||
|
||||
@router.get('/models')
|
||||
@router.get('/models/{url_idx}', dependencies=[Depends(get_admin_user)])
|
||||
@router.get('/models/{url_idx}')
|
||||
async def get_models(request: Request, url_idx: int | None = None, user=Depends(get_verified_user)):
|
||||
if url_idx is not None and user.role != 'admin':
|
||||
raise HTTPException(status_code=401, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
if not await Config.get('openai.enable'):
|
||||
raise HTTPException(status_code=503, detail='OpenAI API is disabled')
|
||||
|
||||
|
|
@ -1416,14 +1417,19 @@ def convert_to_responses_payload(payload: dict) -> dict:
|
|||
converted_tool['description'] = func['description']
|
||||
if 'parameters' in func:
|
||||
converted_tool['parameters'] = func['parameters']
|
||||
if 'strict' in func:
|
||||
converted_tool['strict'] = func['strict']
|
||||
# Responses defaults strict to true, Chat Completions to false
|
||||
converted_tool['strict'] = func.get('strict', False)
|
||||
converted_tools.append(converted_tool)
|
||||
else:
|
||||
# Already in correct format or unknown format, pass through
|
||||
converted_tools.append(tool)
|
||||
responses_payload['tools'] = converted_tools
|
||||
|
||||
# Responses API expects a forced function choice as {"type": "function", "name": ...}
|
||||
tool_choice = responses_payload.get('tool_choice')
|
||||
if isinstance(tool_choice, dict) and isinstance(tool_choice.get('function'), dict):
|
||||
responses_payload['tool_choice'] = {'type': 'function', 'name': tool_choice['function'].get('name', '')}
|
||||
|
||||
return responses_payload
|
||||
|
||||
|
||||
|
|
@ -1431,17 +1437,32 @@ def convert_responses_result(response: dict) -> dict:
|
|||
"""
|
||||
Convert non-streaming Responses API result to Chat Completions format.
|
||||
|
||||
Extracts text from message output items so all downstream consumers
|
||||
Extracts text and function calls from output items so all downstream consumers
|
||||
(frontend tasks, get_content_from_response) work without modification.
|
||||
"""
|
||||
output_items = response.get('output', [])
|
||||
|
||||
content = ''
|
||||
tool_calls = []
|
||||
for item in output_items:
|
||||
if item.get('type') == 'message':
|
||||
for part in item.get('content', []):
|
||||
if part.get('type') == 'output_text':
|
||||
content += part.get('text', '')
|
||||
elif item.get('type') == 'function_call':
|
||||
arguments = item.get('arguments', '{}')
|
||||
if not isinstance(arguments, str):
|
||||
arguments = JSONCodec.dumps(arguments)
|
||||
tool_calls.append(
|
||||
{
|
||||
'id': item.get('call_id', ''),
|
||||
'type': 'function',
|
||||
'function': {
|
||||
'name': item.get('name', ''),
|
||||
'arguments': arguments,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
'id': response.get('id', ''),
|
||||
|
|
@ -1453,8 +1474,9 @@ def convert_responses_result(response: dict) -> dict:
|
|||
'message': {
|
||||
'role': 'assistant',
|
||||
'content': content,
|
||||
**({'tool_calls': tool_calls} if tool_calls else {}),
|
||||
},
|
||||
'finish_reason': 'stop',
|
||||
'finish_reason': 'tool_calls' if tool_calls else 'stop',
|
||||
}
|
||||
],
|
||||
'usage': response.get('usage', {}),
|
||||
|
|
@ -1566,6 +1588,20 @@ async def generate_chat_completion(
|
|||
|
||||
is_responses = api_config.get('api_type') == 'responses'
|
||||
|
||||
# Explicit continuation keeps llama.cpp from echoing the prefill in streamed replies.
|
||||
if (
|
||||
api_config.get('provider') == 'llama.cpp'
|
||||
# These flags apply to Chat Completions, not the Responses API.
|
||||
and not is_responses
|
||||
# The frontend sends this ID when the user clicks Continue.
|
||||
and (metadata or {}).get('assistant_message_id')
|
||||
# Tool follow-ups retain the metadata but must start a new assistant turn.
|
||||
and payload.get('messages')
|
||||
and payload['messages'][-1].get('role') == 'assistant'
|
||||
):
|
||||
payload['continue_final_message'] = True
|
||||
payload['add_generation_prompt'] = False
|
||||
|
||||
if api_config.get('azure') or api_config.get('provider') == 'azure':
|
||||
# Only set api-key header if not using Azure Entra ID authentication
|
||||
auth_type = api_config.get('auth_type', 'bearer')
|
||||
|
|
|
|||
|
|
@ -58,6 +58,7 @@ from open_webui.env import (
|
|||
SENTENCE_TRANSFORMERS_CROSS_ENCODER_MODEL_KWARGS,
|
||||
SENTENCE_TRANSFORMERS_CROSS_ENCODER_SIGMOID_ACTIVATION_FUNCTION,
|
||||
SENTENCE_TRANSFORMERS_MODEL_KWARGS,
|
||||
USE_SLIM,
|
||||
)
|
||||
from open_webui.events import EVENTS, publish_event
|
||||
from open_webui.internal.db import get_async_db, get_async_session
|
||||
|
|
@ -82,7 +83,7 @@ from open_webui.retrieval.utils import (
|
|||
query_doc_with_hybrid_search,
|
||||
)
|
||||
from open_webui.retrieval.vector.async_client import ASYNC_VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import VECTOR_DB_CLIENT
|
||||
from open_webui.retrieval.vector.factory import get_vector_db_client
|
||||
from open_webui.retrieval.vector.utils import filter_metadata
|
||||
from open_webui.retrieval.web.azure import search_azure
|
||||
from open_webui.retrieval.web.bing import search_bing
|
||||
|
|
@ -114,6 +115,7 @@ from open_webui.retrieval.web.serphouse import search_serphouse
|
|||
from open_webui.retrieval.web.serply import search_serply
|
||||
from open_webui.retrieval.web.serpstack import search_serpstack
|
||||
from open_webui.retrieval.web.sougou import search_sougou
|
||||
from open_webui.retrieval.web.staan import search_staan
|
||||
from open_webui.retrieval.web.tavily import search_tavily
|
||||
from open_webui.retrieval.web.utils import get_web_loader
|
||||
from open_webui.retrieval.web.yacy import search_yacy
|
||||
|
|
@ -128,7 +130,7 @@ from open_webui.utils.misc import (
|
|||
calculate_sha256_string,
|
||||
sanitize_text_for_db,
|
||||
)
|
||||
from pydantic import BaseModel
|
||||
from pydantic import BaseModel, Field
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -150,7 +152,7 @@ def get_ef(
|
|||
auto_update: bool = RAG_EMBEDDING_MODEL_AUTO_UPDATE,
|
||||
):
|
||||
ef = None
|
||||
if embedding_model and engine == '':
|
||||
if embedding_model and engine == '' and not USE_SLIM:
|
||||
from sentence_transformers import SentenceTransformer
|
||||
|
||||
try:
|
||||
|
|
@ -178,6 +180,17 @@ def get_rf(
|
|||
rf = None
|
||||
# Convert timeout string to int or None (system default)
|
||||
timeout_value = int(external_reranker_timeout) if external_reranker_timeout else None
|
||||
if reranking_model and engine == 'external':
|
||||
from open_webui.retrieval.models.external import ExternalReranker
|
||||
|
||||
return ExternalReranker(
|
||||
url=external_reranker_url,
|
||||
api_key=external_reranker_api_key,
|
||||
model=reranking_model,
|
||||
timeout=timeout_value,
|
||||
)
|
||||
if USE_SLIM:
|
||||
return None
|
||||
if reranking_model:
|
||||
if any(model in reranking_model for model in ['jinaai/jina-colbert-v2']):
|
||||
try:
|
||||
|
|
@ -192,55 +205,39 @@ def get_rf(
|
|||
log.error(f'ColBERT: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'Error loading reranking model'))
|
||||
else:
|
||||
if engine == 'external':
|
||||
try:
|
||||
from open_webui.retrieval.models.external import ExternalReranker
|
||||
import sentence_transformers
|
||||
import torch
|
||||
|
||||
rf = ExternalReranker(
|
||||
url=external_reranker_url,
|
||||
api_key=external_reranker_api_key,
|
||||
model=reranking_model,
|
||||
timeout=timeout_value,
|
||||
)
|
||||
except Exception as e:
|
||||
log.error(f'ExternalReranking: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'Error loading reranking model'))
|
||||
else:
|
||||
import sentence_transformers
|
||||
import torch
|
||||
try:
|
||||
rf = sentence_transformers.CrossEncoder(
|
||||
get_model_path(reranking_model, auto_update),
|
||||
device=DEVICE_TYPE,
|
||||
trust_remote_code=RAG_RERANKING_MODEL_TRUST_REMOTE_CODE,
|
||||
backend=SENTENCE_TRANSFORMERS_CROSS_ENCODER_BACKEND,
|
||||
model_kwargs=SENTENCE_TRANSFORMERS_CROSS_ENCODER_MODEL_KWARGS,
|
||||
activation_fn=(
|
||||
torch.nn.Sigmoid() if SENTENCE_TRANSFORMERS_CROSS_ENCODER_SIGMOID_ACTIVATION_FUNCTION else None
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
log.error(f'CrossEncoder: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'CrossEncoder error'))
|
||||
|
||||
try:
|
||||
rf = sentence_transformers.CrossEncoder(
|
||||
get_model_path(reranking_model, auto_update),
|
||||
device=DEVICE_TYPE,
|
||||
trust_remote_code=RAG_RERANKING_MODEL_TRUST_REMOTE_CODE,
|
||||
backend=SENTENCE_TRANSFORMERS_CROSS_ENCODER_BACKEND,
|
||||
model_kwargs=SENTENCE_TRANSFORMERS_CROSS_ENCODER_MODEL_KWARGS,
|
||||
activation_fn=(
|
||||
torch.nn.Sigmoid()
|
||||
if SENTENCE_TRANSFORMERS_CROSS_ENCODER_SIGMOID_ACTIVATION_FUNCTION
|
||||
else None
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
log.error(f'CrossEncoder: {e}')
|
||||
raise Exception(ERROR_MESSAGES.DEFAULT(e, 'CrossEncoder error'))
|
||||
|
||||
# Safely adjust pad_token_id if missing as some models do not have this in config
|
||||
try:
|
||||
model_cfg = getattr(rf, 'model', None)
|
||||
if model_cfg and hasattr(model_cfg, 'config'):
|
||||
cfg = model_cfg.config
|
||||
if getattr(cfg, 'pad_token_id', None) is None:
|
||||
# Fallback to eos_token_id when available
|
||||
eos = getattr(cfg, 'eos_token_id', None)
|
||||
if eos is not None:
|
||||
cfg.pad_token_id = eos
|
||||
log.debug('Missing pad_token_id detected; set to eos_token_id=%s', eos)
|
||||
else:
|
||||
log.warning('Neither pad_token_id nor eos_token_id present in model config')
|
||||
except Exception as e2:
|
||||
log.warning(f'Failed to adjust pad_token_id on CrossEncoder: {e2}')
|
||||
# Safely adjust pad_token_id if missing as some models do not have this in config
|
||||
try:
|
||||
model_cfg = getattr(rf, 'model', None)
|
||||
if model_cfg and hasattr(model_cfg, 'config'):
|
||||
cfg = model_cfg.config
|
||||
if getattr(cfg, 'pad_token_id', None) is None:
|
||||
# Fallback to eos_token_id when available
|
||||
eos = getattr(cfg, 'eos_token_id', None)
|
||||
if eos is not None:
|
||||
cfg.pad_token_id = eos
|
||||
log.debug('Missing pad_token_id detected; set to eos_token_id=%s', eos)
|
||||
else:
|
||||
log.warning('Neither pad_token_id nor eos_token_id present in model config')
|
||||
except Exception as e2:
|
||||
log.warning(f'Failed to adjust pad_token_id on CrossEncoder: {e2}')
|
||||
|
||||
return rf
|
||||
|
||||
|
|
@ -301,6 +298,7 @@ RETRIEVAL_CONFIG_KEYS = {
|
|||
'ENABLE_WEB_SEARCH_CONFIRMATION': 'web.search.confirmation.enable',
|
||||
'WEB_SEARCH_CONFIRMATION_CONTENT': 'web.search.confirmation.content',
|
||||
'EXA_API_KEY': 'web.search.exa_api_key',
|
||||
'EXA_MAX_CONTENT_LENGTH': 'web.search.exa_max_content_length',
|
||||
'EXTERNAL_DOCUMENT_LOADER_API_KEY': 'rag.external_document_loader_api_key',
|
||||
'EXTERNAL_DOCUMENT_LOADER_HEADERS': 'rag.external_document_loader_headers',
|
||||
'EXTERNAL_DOCUMENT_LOADER_URL': 'rag.external_document_loader_url',
|
||||
|
|
@ -383,6 +381,9 @@ RETRIEVAL_CONFIG_KEYS = {
|
|||
'SERPSTACK_HTTPS': 'web.search.serpstack_https',
|
||||
'SOUGOU_API_SID': 'web.search.sougou_api_sid',
|
||||
'SOUGOU_API_SK': 'web.search.sougou_api_sk',
|
||||
'STAAN_API_KEY': 'web.search.staan_api_key',
|
||||
'STAAN_MARKET': 'web.search.staan_market',
|
||||
'STAAN_MAX_SNIPPETS': 'web.search.staan_max_snippets',
|
||||
'TAVILY_API_KEY': 'web.search.tavily_api_key',
|
||||
'TAVILY_EXTRACT_DEPTH': 'web.search.tavily_extract_depth',
|
||||
'TEXT_SPLITTER': 'rag.text_splitter',
|
||||
|
|
@ -535,6 +536,8 @@ async def unload_embedding_model(request: Request):
|
|||
|
||||
@router.post('/embedding/update')
|
||||
async def update_embedding_config(request: Request, form_data: EmbeddingModelUpdateForm, user=Depends(get_admin_user)):
|
||||
if USE_SLIM and form_data.RAG_EMBEDDING_ENGINE == '':
|
||||
raise HTTPException(400, 'Slim requires an external embedding engine (openai, ollama, azure_openai).')
|
||||
config = await get_retrieval_config()
|
||||
log.info('Updating embedding model: %s to %s', config.RAG_EMBEDDING_MODEL, form_data.RAG_EMBEDDING_MODEL)
|
||||
await unload_embedding_model(request)
|
||||
|
|
@ -738,6 +741,9 @@ async def get_rag_config(request: Request, user=Depends(get_admin_user)):
|
|||
'SERPLY_API_KEY': config.SERPLY_API_KEY,
|
||||
'DDGS_BACKEND': config.DDGS_BACKEND,
|
||||
'TAVILY_API_KEY': config.TAVILY_API_KEY,
|
||||
'STAAN_API_KEY': config.STAAN_API_KEY,
|
||||
'STAAN_MARKET': config.STAAN_MARKET,
|
||||
'STAAN_MAX_SNIPPETS': config.STAAN_MAX_SNIPPETS,
|
||||
'SEARCHAPI_API_KEY': config.SEARCHAPI_API_KEY,
|
||||
'SEARCHAPI_ENGINE': config.SEARCHAPI_ENGINE,
|
||||
'SERPAPI_API_KEY': config.SERPAPI_API_KEY,
|
||||
|
|
@ -747,6 +753,7 @@ async def get_rag_config(request: Request, user=Depends(get_admin_user)):
|
|||
'BING_SEARCH_V7_ENDPOINT': config.BING_SEARCH_V7_ENDPOINT,
|
||||
'BING_SEARCH_V7_SUBSCRIPTION_KEY': config.BING_SEARCH_V7_SUBSCRIPTION_KEY,
|
||||
'EXA_API_KEY': config.EXA_API_KEY,
|
||||
'EXA_MAX_CONTENT_LENGTH': config.EXA_MAX_CONTENT_LENGTH,
|
||||
'PERPLEXITY_API_KEY': config.PERPLEXITY_API_KEY,
|
||||
'PERPLEXITY_MODEL': config.PERPLEXITY_MODEL,
|
||||
'PERPLEXITY_SEARCH_CONTEXT_USAGE': config.PERPLEXITY_SEARCH_CONTEXT_USAGE,
|
||||
|
|
@ -817,6 +824,9 @@ class WebConfig(BaseModel):
|
|||
SERPLY_API_KEY: str | None = None
|
||||
DDGS_BACKEND: str | None = None
|
||||
TAVILY_API_KEY: str | None = None
|
||||
STAAN_API_KEY: str | None = None
|
||||
STAAN_MARKET: str | None = None
|
||||
STAAN_MAX_SNIPPETS: int | None = None
|
||||
SEARCHAPI_API_KEY: str | None = None
|
||||
SEARCHAPI_ENGINE: str | None = None
|
||||
SERPAPI_API_KEY: str | None = None
|
||||
|
|
@ -826,6 +836,7 @@ class WebConfig(BaseModel):
|
|||
BING_SEARCH_V7_ENDPOINT: str | None = None
|
||||
BING_SEARCH_V7_SUBSCRIPTION_KEY: str | None = None
|
||||
EXA_API_KEY: str | None = None
|
||||
EXA_MAX_CONTENT_LENGTH: int | None = Field(default=None, gt=0, strict=True)
|
||||
PERPLEXITY_API_KEY: str | None = None
|
||||
PERPLEXITY_MODEL: str | None = None
|
||||
PERPLEXITY_SEARCH_CONTEXT_USAGE: str | None = None
|
||||
|
|
@ -952,6 +963,44 @@ class ConfigForm(BaseModel):
|
|||
async def update_rag_config(request: Request, form_data: ConfigForm, user=Depends(get_admin_user)):
|
||||
# RAG settings
|
||||
config = await get_retrieval_config()
|
||||
if USE_SLIM:
|
||||
if (
|
||||
form_data.web
|
||||
and form_data.web.WEB_SEARCH_ENGINE == 'duckduckgo'
|
||||
and config.WEB_SEARCH_ENGINE != 'duckduckgo'
|
||||
):
|
||||
raise HTTPException(
|
||||
400,
|
||||
'DDGS is unavailable in slim. Configure another web search provider in Admin Settings > Web Search.',
|
||||
)
|
||||
if (
|
||||
form_data.web
|
||||
and form_data.web.WEB_LOADER_ENGINE == 'playwright'
|
||||
and config.WEB_LOADER_ENGINE != 'playwright'
|
||||
):
|
||||
raise HTTPException(
|
||||
400, 'Playwright is unavailable in slim. Use basic HTTP fetching or an external web loader.'
|
||||
)
|
||||
if form_data.TEXT_SPLITTER == 'token_transformers' and config.TEXT_SPLITTER != 'token_transformers':
|
||||
raise HTTPException(
|
||||
400, 'Transformers tokenization is unavailable in slim. Use character or token splitting.'
|
||||
)
|
||||
reranker_engine = (
|
||||
form_data.RAG_RERANKING_ENGINE
|
||||
if form_data.RAG_RERANKING_ENGINE is not None
|
||||
else config.RAG_RERANKING_ENGINE
|
||||
)
|
||||
reranker_model = (
|
||||
form_data.RAG_RERANKING_MODEL if form_data.RAG_RERANKING_MODEL is not None else config.RAG_RERANKING_MODEL
|
||||
)
|
||||
if (
|
||||
reranker_engine != 'external'
|
||||
and reranker_model
|
||||
and (reranker_engine != config.RAG_RERANKING_ENGINE or reranker_model != config.RAG_RERANKING_MODEL)
|
||||
):
|
||||
raise HTTPException(
|
||||
400, 'Slim requires an external reranker, or an empty reranking model for cosine scoring.'
|
||||
)
|
||||
config.RAG_TEMPLATE = form_data.RAG_TEMPLATE if form_data.RAG_TEMPLATE is not None else config.RAG_TEMPLATE
|
||||
config.TOP_K = form_data.TOP_K if form_data.TOP_K is not None else config.TOP_K
|
||||
config.BYPASS_EMBEDDING_AND_RETRIEVAL = (
|
||||
|
|
@ -1176,7 +1225,8 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
else config.RAG_RERANKING_BATCH_SIZE
|
||||
)
|
||||
|
||||
log.info('Updating reranking model: %s to %s', config.RAG_RERANKING_MODEL, form_data.RAG_RERANKING_MODEL)
|
||||
if form_data.RAG_RERANKING_MODEL is not None:
|
||||
log.info('Updating reranking model: %s to %s', config.RAG_RERANKING_MODEL, form_data.RAG_RERANKING_MODEL)
|
||||
try:
|
||||
config.RAG_RERANKING_MODEL = (
|
||||
form_data.RAG_RERANKING_MODEL if form_data.RAG_RERANKING_MODEL is not None else config.RAG_RERANKING_MODEL
|
||||
|
|
@ -1297,6 +1347,9 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
config.SERPLY_API_KEY = form_data.web.SERPLY_API_KEY
|
||||
config.DDGS_BACKEND = form_data.web.DDGS_BACKEND
|
||||
config.TAVILY_API_KEY = form_data.web.TAVILY_API_KEY
|
||||
config.STAAN_API_KEY = form_data.web.STAAN_API_KEY
|
||||
config.STAAN_MARKET = form_data.web.STAAN_MARKET
|
||||
config.STAAN_MAX_SNIPPETS = form_data.web.STAAN_MAX_SNIPPETS
|
||||
config.SEARCHAPI_API_KEY = form_data.web.SEARCHAPI_API_KEY
|
||||
config.SEARCHAPI_ENGINE = form_data.web.SEARCHAPI_ENGINE
|
||||
config.SERPAPI_API_KEY = form_data.web.SERPAPI_API_KEY
|
||||
|
|
@ -1306,6 +1359,7 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
config.BING_SEARCH_V7_ENDPOINT = form_data.web.BING_SEARCH_V7_ENDPOINT
|
||||
config.BING_SEARCH_V7_SUBSCRIPTION_KEY = form_data.web.BING_SEARCH_V7_SUBSCRIPTION_KEY
|
||||
config.EXA_API_KEY = form_data.web.EXA_API_KEY
|
||||
config.EXA_MAX_CONTENT_LENGTH = form_data.web.EXA_MAX_CONTENT_LENGTH
|
||||
config.PERPLEXITY_API_KEY = form_data.web.PERPLEXITY_API_KEY
|
||||
config.PERPLEXITY_MODEL = form_data.web.PERPLEXITY_MODEL
|
||||
config.PERPLEXITY_SEARCH_CONTEXT_USAGE = form_data.web.PERPLEXITY_SEARCH_CONTEXT_USAGE
|
||||
|
|
@ -1449,6 +1503,9 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
'SERPHOUSE_DOMAIN': config.SERPHOUSE_DOMAIN,
|
||||
'SERPLY_API_KEY': config.SERPLY_API_KEY,
|
||||
'TAVILY_API_KEY': config.TAVILY_API_KEY,
|
||||
'STAAN_API_KEY': config.STAAN_API_KEY,
|
||||
'STAAN_MARKET': config.STAAN_MARKET,
|
||||
'STAAN_MAX_SNIPPETS': config.STAAN_MAX_SNIPPETS,
|
||||
'SEARCHAPI_API_KEY': config.SEARCHAPI_API_KEY,
|
||||
'SEARCHAPI_ENGINE': config.SEARCHAPI_ENGINE,
|
||||
'SERPAPI_API_KEY': config.SERPAPI_API_KEY,
|
||||
|
|
@ -1458,6 +1515,7 @@ async def update_rag_config(request: Request, form_data: ConfigForm, user=Depend
|
|||
'BING_SEARCH_V7_ENDPOINT': config.BING_SEARCH_V7_ENDPOINT,
|
||||
'BING_SEARCH_V7_SUBSCRIPTION_KEY': config.BING_SEARCH_V7_SUBSCRIPTION_KEY,
|
||||
'EXA_API_KEY': config.EXA_API_KEY,
|
||||
'EXA_MAX_CONTENT_LENGTH': config.EXA_MAX_CONTENT_LENGTH,
|
||||
'PERPLEXITY_API_KEY': config.PERPLEXITY_API_KEY,
|
||||
'PERPLEXITY_MODEL': config.PERPLEXITY_MODEL,
|
||||
'PERPLEXITY_SEARCH_CONTEXT_USAGE': config.PERPLEXITY_SEARCH_CONTEXT_USAGE,
|
||||
|
|
@ -1589,6 +1647,8 @@ def merge_docs_to_target_size(
|
|||
|
||||
|
||||
def get_transformers_tokenizer(request: Request, config: RetrievalConfig):
|
||||
if USE_SLIM:
|
||||
raise HTTPException(503, 'Transformers tokenization is unavailable in slim. Use character or token splitting.')
|
||||
if config.RAG_TOKENIZER_MODEL:
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
|
|
@ -1634,6 +1694,14 @@ def get_splitter_length_function(
|
|||
return len
|
||||
|
||||
|
||||
def filter_file_metadata(metadata: dict | None) -> dict:
|
||||
metadata = dict(metadata or {})
|
||||
data = metadata.pop('data', None)
|
||||
if isinstance(data, dict):
|
||||
metadata = {**filter_metadata(data), **metadata}
|
||||
return filter_metadata(metadata)
|
||||
|
||||
|
||||
def save_docs_to_vector_db(
|
||||
request: Request,
|
||||
docs,
|
||||
|
|
@ -1665,7 +1733,7 @@ def save_docs_to_vector_db(
|
|||
|
||||
# Check if entries with the same hash (metadata.hash) already exist
|
||||
if metadata and 'hash' in metadata:
|
||||
result = VECTOR_DB_CLIENT.query(
|
||||
result = get_vector_db_client().query(
|
||||
collection_name=collection_name,
|
||||
filter={'hash': metadata['hash']},
|
||||
)
|
||||
|
|
@ -1765,11 +1833,11 @@ def save_docs_to_vector_db(
|
|||
]
|
||||
|
||||
try:
|
||||
if VECTOR_DB_CLIENT.has_collection(collection_name=collection_name):
|
||||
if get_vector_db_client().has_collection(collection_name=collection_name):
|
||||
log.info('collection %s already exists', collection_name)
|
||||
|
||||
if overwrite:
|
||||
VECTOR_DB_CLIENT.delete_collection(collection_name=collection_name)
|
||||
get_vector_db_client().delete_collection(collection_name=collection_name)
|
||||
log.info('deleting existing collection %s', collection_name)
|
||||
elif add is False:
|
||||
log.info('collection %s already exists, overwrite is False and add is False', collection_name)
|
||||
|
|
@ -1832,7 +1900,7 @@ def save_docs_to_vector_db(
|
|||
]
|
||||
|
||||
log.info('adding to collection %s', collection_name)
|
||||
VECTOR_DB_CLIENT.insert(
|
||||
get_vector_db_client().insert(
|
||||
collection_name=collection_name,
|
||||
items=items,
|
||||
)
|
||||
|
|
@ -1898,7 +1966,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=form_data.content.replace('<br/>', '\n'),
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
@ -1934,7 +2002,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=stored_content,
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
@ -1967,6 +2035,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=doc.page_content,
|
||||
metadata={
|
||||
**filter_file_metadata(file.meta),
|
||||
**filter_metadata(doc.metadata),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
|
|
@ -1981,7 +2050,7 @@ async def process_file(
|
|||
Document(
|
||||
page_content=file.data.get('content', ''),
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
@ -2636,6 +2705,19 @@ async def search_web(request: Request, engine: str, query: str, user=None) -> li
|
|||
)
|
||||
else:
|
||||
raise Exception('No TAVILY_API_KEY found in environment variables')
|
||||
elif engine == 'staan':
|
||||
if config.STAAN_API_KEY:
|
||||
return await asyncio.to_thread(
|
||||
search_staan,
|
||||
config.STAAN_API_KEY,
|
||||
query,
|
||||
config.WEB_SEARCH_RESULT_COUNT,
|
||||
config.WEB_SEARCH_DOMAIN_FILTER_LIST,
|
||||
market=config.STAAN_MARKET,
|
||||
max_snippets=config.STAAN_MAX_SNIPPETS,
|
||||
)
|
||||
else:
|
||||
raise Exception('No STAAN_API_KEY found in environment variables')
|
||||
elif engine == 'exa':
|
||||
if config.EXA_API_KEY:
|
||||
return await asyncio.to_thread(
|
||||
|
|
@ -2644,6 +2726,7 @@ async def search_web(request: Request, engine: str, query: str, user=None) -> li
|
|||
query,
|
||||
config.WEB_SEARCH_RESULT_COUNT,
|
||||
config.WEB_SEARCH_DOMAIN_FILTER_LIST,
|
||||
max_content_length=config.EXA_MAX_CONTENT_LENGTH,
|
||||
)
|
||||
else:
|
||||
raise Exception('No EXA_API_KEY found in environment variables')
|
||||
|
|
@ -3026,7 +3109,7 @@ async def query_doc_handler(
|
|||
query_embedding = await request.app.state.EMBEDDING_FUNCTION(
|
||||
form_data.query, prefix=RAG_EMBEDDING_QUERY_PREFIX, user=user
|
||||
)
|
||||
# query_doc wraps a blocking VECTOR_DB_CLIENT.search call;
|
||||
# query_doc wraps a blocking get_vector_db_client().search call;
|
||||
# offload so the request's event loop stays responsive.
|
||||
return await asyncio.to_thread(
|
||||
query_doc,
|
||||
|
|
@ -3314,7 +3397,7 @@ async def process_files_batch(
|
|||
Document(
|
||||
page_content=text_content.replace('<br/>', '\n'),
|
||||
metadata={
|
||||
**file.meta,
|
||||
**filter_file_metadata(file.meta),
|
||||
'name': file.filename,
|
||||
'created_by': file.user_id,
|
||||
'file_id': file.id,
|
||||
|
|
|
|||
|
|
@ -753,31 +753,60 @@ async def patch_user(
|
|||
)
|
||||
|
||||
update_data = {}
|
||||
fields = {
|
||||
'userName': 'email',
|
||||
'displayName': 'name',
|
||||
'emails[primary eq true].value': 'email',
|
||||
'name.formatted': 'name',
|
||||
}
|
||||
|
||||
for operation in patch_data.Operations:
|
||||
op = operation.op.lower()
|
||||
path = operation.path
|
||||
value = operation.value
|
||||
|
||||
if op == 'replace':
|
||||
if op not in ('add', 'replace', 'remove'):
|
||||
return scim_error(400, f'Unsupported PATCH operation: {operation.op}')
|
||||
if op == 'remove':
|
||||
if not path:
|
||||
return scim_error(400, 'Remove requires a path', 'noTarget')
|
||||
if path != 'externalId':
|
||||
return scim_error(400, f'Removing {path} is not supported', 'mutability')
|
||||
values = {path: None}
|
||||
elif path is None:
|
||||
if not isinstance(operation.value, dict) or not operation.value:
|
||||
return scim_error(400, 'A pathless operation requires an attribute object', 'invalidValue')
|
||||
values = operation.value
|
||||
else:
|
||||
values = {path: operation.value}
|
||||
|
||||
for path, value in values.items():
|
||||
if path == 'active':
|
||||
if not isinstance(value, bool):
|
||||
return scim_error(400, 'active must be a boolean', 'invalidValue')
|
||||
# Same guard as update_user: never demote an existing admin via SCIM.
|
||||
if user.role != 'admin':
|
||||
update_data['role'] = 'user' if value else 'pending'
|
||||
elif path == 'userName':
|
||||
update_data['email'] = value
|
||||
elif path == 'displayName':
|
||||
update_data['name'] = value
|
||||
elif path == 'emails[primary eq true].value':
|
||||
update_data['email'] = value
|
||||
elif path == 'name.formatted':
|
||||
update_data['name'] = value
|
||||
elif path in fields:
|
||||
if not isinstance(value, str):
|
||||
return scim_error(400, f'{path} must be a string', 'invalidValue')
|
||||
update_data[fields[path]] = value
|
||||
elif path == 'externalId':
|
||||
if value is not None and not isinstance(value, str):
|
||||
return scim_error(400, 'externalId must be a string or null', 'invalidValue')
|
||||
provider = get_scim_provider()
|
||||
await Users.update_user_scim_by_id(user_id, provider, value, db=db)
|
||||
scim = dict(update_data.get('scim', user.scim) or {})
|
||||
scim[provider] = {'external_id': value}
|
||||
update_data['scim'] = scim
|
||||
else:
|
||||
return scim_error(400, f'Unsupported PATCH path: {path}', 'invalidPath')
|
||||
|
||||
# Validate all operations before persisting once, and leave identical writes unchanged.
|
||||
update_data = {key: value for key, value in update_data.items() if value != getattr(user, key)}
|
||||
user_updated_fields = ['externalId' if field == 'scim' else field for field in update_data if field != 'role']
|
||||
|
||||
# Update user
|
||||
if update_data:
|
||||
update_data['updated_at'] = int(time.time())
|
||||
updated_user = await Users.update_user_by_id(user_id, update_data, db=db)
|
||||
if not updated_user:
|
||||
raise HTTPException(
|
||||
|
|
@ -788,7 +817,6 @@ async def patch_user(
|
|||
updated_user = user
|
||||
|
||||
role_changed = updated_user.role != user.role
|
||||
user_updated_fields = [field for field in update_data.keys() if field != 'role']
|
||||
|
||||
if user_updated_fields:
|
||||
await publish_event(
|
||||
|
|
|
|||
|
|
@ -5,6 +5,7 @@ Routes:
|
|||
* /{server_id}/{path:path} — proxy request to terminal server
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import posixpath
|
||||
from urllib.parse import unquote
|
||||
|
|
@ -18,28 +19,33 @@ from open_webui.events import EVENTS, publish_event
|
|||
from open_webui.models.config import Config
|
||||
from open_webui.models.groups import Groups
|
||||
from open_webui.utils.access_control import has_connection_access
|
||||
from open_webui.utils.auth import get_verified_user
|
||||
from open_webui.utils.auth import get_verified_user, get_verified_user_by_token
|
||||
from open_webui.utils.headers import bearer_auth_header, normalize_bearer_token
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.terminals import (
|
||||
TERMINAL_CONTEXT_HEADER,
|
||||
get_terminal_server_url,
|
||||
is_terminal_orchestrator,
|
||||
terminal_chat_uploads,
|
||||
terminal_context_available,
|
||||
terminal_context_config,
|
||||
terminal_context_id,
|
||||
terminal_chat_uploads,
|
||||
terminal_contexts,
|
||||
)
|
||||
from starlette.background import BackgroundTask
|
||||
from starlette.requests import ClientDisconnect
|
||||
from yarl import URL
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
STREAMING_CONTENT_TYPES = ('application/octet-stream', 'image/', 'application/pdf')
|
||||
STRIPPED_RESPONSE_HEADERS = frozenset(('transfer-encoding', 'connection', 'content-encoding', 'content-length'))
|
||||
ADMIN_API_PATHS = ('api/v1/policies', 'api/v1/status', 'api/v1/terminals')
|
||||
# Drop the upstream's server and date: uvicorn adds its own and forwarding both duplicates them.
|
||||
STRIPPED_RESPONSE_HEADERS = frozenset(
|
||||
('transfer-encoding', 'connection', 'content-encoding', 'content-length', 'server', 'date')
|
||||
)
|
||||
|
||||
|
||||
def _sanitize_proxy_path(path: str) -> str | None:
|
||||
|
|
@ -62,7 +68,8 @@ def _sanitize_proxy_path(path: str) -> str | None:
|
|||
return None
|
||||
# posixpath splits on '/' only, so 'a/..\..\b' survives normpath as one component.
|
||||
# Upstreams that treat '\' as a separator would resolve it, so reject outright.
|
||||
if '\\' in decoded:
|
||||
# URL parsers also remove tabs/newlines, which can turn '.\t.' into '..'.
|
||||
if any(char in decoded for char in '\\\t\r\n'):
|
||||
return None
|
||||
had_trailing_slash = decoded.endswith('/')
|
||||
normalized = posixpath.normpath(decoded)
|
||||
|
|
@ -130,6 +137,15 @@ async def proxy_terminal(
|
|||
|
||||
target_url = f'{base_url}/{safe_path}'
|
||||
|
||||
# Check the path aiohttp will send, relative to the configured server root.
|
||||
base_path = URL(str(connection.get('url') or '')).path.rstrip('/')
|
||||
target_path = URL(target_url).path
|
||||
if any(
|
||||
target_path == f'{base_path}/{prefix}' or target_path.startswith(f'{base_path}/{prefix}/')
|
||||
for prefix in ADMIN_API_PATHS
|
||||
):
|
||||
return JSONResponse({'error': 'Path not allowed'}, status_code=403)
|
||||
|
||||
if request.query_params:
|
||||
target_url += f'?{request.query_params}'
|
||||
|
||||
|
|
@ -145,16 +161,14 @@ async def proxy_terminal(
|
|||
return JSONResponse({'error': 'A saved chat is required for this terminal'}, status_code=409)
|
||||
if context_id:
|
||||
headers[TERMINAL_CONTEXT_HEADER] = context_id
|
||||
cookies = {}
|
||||
cookies = getattr(request, 'cookies', {}) if connection.get('forward_cookies', False) else {}
|
||||
auth_type = connection.get('auth_type', 'bearer')
|
||||
|
||||
if auth_type == 'bearer':
|
||||
headers.update(bearer_auth_header(connection.get('key', '')))
|
||||
elif auth_type == 'session':
|
||||
cookies = request.cookies
|
||||
headers.update(bearer_auth_header(request.state.token.credentials))
|
||||
elif auth_type == 'system_oauth':
|
||||
cookies = request.cookies
|
||||
# Resolve the token server-side from the caller's OAuth session; never trust a client header.
|
||||
oauth_token = None
|
||||
try:
|
||||
|
|
@ -188,6 +202,7 @@ async def proxy_terminal(
|
|||
cookies=cookies,
|
||||
data=body or None,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
allow_redirects=False,
|
||||
)
|
||||
|
||||
upstream_content_type = upstream_response.headers.get('content-type', '')
|
||||
|
|
@ -248,10 +263,6 @@ async def _resolve_authenticated_connection(ws: WebSocket, server_id: str):
|
|||
Returns ``(user, connection, chat_id, token)`` on success, or ``None`` after
|
||||
closing *ws* with an appropriate error code.
|
||||
"""
|
||||
import asyncio
|
||||
|
||||
from open_webui.utils.auth import get_verified_user_by_token
|
||||
|
||||
# First-message authentication
|
||||
try:
|
||||
raw = await asyncio.wait_for(ws.receive_text(), timeout=10.0)
|
||||
|
|
@ -260,13 +271,28 @@ async def _resolve_authenticated_connection(ws: WebSocket, server_id: str):
|
|||
await ws.close(code=4001, reason='Expected auth message')
|
||||
return None
|
||||
token = payload.get('token', '')
|
||||
except (TimeoutError, JSONCodec.JSONDecodeError):
|
||||
await ws.close(code=4001, reason='Auth timeout or invalid payload')
|
||||
return None
|
||||
except Exception:
|
||||
await ws.close(code=4001, reason='Invalid token')
|
||||
return None
|
||||
|
||||
result = await _resolve_terminal_access(ws, server_id, token)
|
||||
if result is None:
|
||||
return None
|
||||
user, connection = result
|
||||
chat_id = payload.get('chat_id', '')
|
||||
return user, connection, chat_id if isinstance(chat_id, str) else '', token
|
||||
|
||||
|
||||
async def _resolve_terminal_access(ws: WebSocket, server_id: str, token: str):
|
||||
"""Resolve current access for both the handshake and an open terminal session."""
|
||||
try:
|
||||
user = await get_verified_user_by_token(token, getattr(ws.app.state, 'redis', None))
|
||||
if user is None:
|
||||
await ws.close(code=4001, reason='Invalid token')
|
||||
return None
|
||||
except (asyncio.TimeoutError, JSONCodec.JSONDecodeError):
|
||||
await ws.close(code=4001, reason='Auth timeout or invalid payload')
|
||||
return None
|
||||
except Exception:
|
||||
await ws.close(code=4001, reason='Invalid token')
|
||||
return None
|
||||
|
|
@ -283,16 +309,14 @@ async def _resolve_authenticated_connection(ws: WebSocket, server_id: str):
|
|||
await ws.close(code=4003, reason='Terminal server disabled')
|
||||
return None
|
||||
|
||||
user_group_ids = {group.id for group in await Groups.get_groups_by_member_id(user.id)}
|
||||
if not await has_connection_access(user, connection, user_group_ids):
|
||||
if not await has_connection_access(user, connection):
|
||||
await ws.close(code=4003, reason='Access denied')
|
||||
return None
|
||||
|
||||
chat_id = payload.get('chat_id', '')
|
||||
if not terminal_context_available(connection, 'chat'):
|
||||
await ws.close(code=4003, reason='Terminal server is not available in chats')
|
||||
return None
|
||||
return user, connection, chat_id if isinstance(chat_id, str) else '', token
|
||||
return user, connection
|
||||
|
||||
|
||||
@router.websocket('/{server_id}/api/terminals/{session_id}')
|
||||
|
|
@ -326,7 +350,7 @@ async def ws_terminal(
|
|||
# For orchestrator-backed servers, pass user_id
|
||||
upstream_params['user_id'] = user.id
|
||||
context_id = terminal_context_id(connection, {'chat_id': chat_id}, 'chat')
|
||||
upstream_headers = {}
|
||||
upstream_headers = {'X-User-Id': user.id, 'X-Session-Id': chat_id}
|
||||
if terminal_context_config(connection, 'chat').get('context_id') == 'chat_id' and not context_id:
|
||||
await ws.close(code=4003, reason='A saved chat is required for this terminal')
|
||||
return
|
||||
|
|
@ -352,7 +376,6 @@ async def ws_terminal(
|
|||
headers=upstream_headers,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
) as upstream:
|
||||
import asyncio
|
||||
import json as _json
|
||||
|
||||
# First-message auth to upstream terminal server
|
||||
|
|
@ -362,6 +385,8 @@ async def ws_terminal(
|
|||
await upstream.send_str(_json.dumps({'type': 'auth', 'token': key}))
|
||||
elif auth_type == 'session' and is_terminal_orchestrator(connection):
|
||||
await upstream.send_str(_json.dumps({'type': 'auth', 'token': token}))
|
||||
else:
|
||||
await upstream.send_str(_json.dumps({'type': 'auth', 'token': ''}))
|
||||
|
||||
await publish_event(
|
||||
app,
|
||||
|
|
@ -403,20 +428,30 @@ async def ws_terminal(
|
|||
except Exception:
|
||||
pass
|
||||
|
||||
# End the proxy as soon as either direction finishes (e.g. a
|
||||
# graceful upstream CLOSE) and cancel the sibling, which would
|
||||
async def _watch_access():
|
||||
try:
|
||||
while True:
|
||||
# Poll current state so revocation also works across workers.
|
||||
await asyncio.sleep(10)
|
||||
if await _resolve_terminal_access(ws, server_id, token) is None:
|
||||
return
|
||||
except Exception:
|
||||
log.exception('Terminal access recheck failed')
|
||||
|
||||
# End the proxy as soon as any task finishes (e.g. a
|
||||
# graceful upstream CLOSE) and cancel the rest, which would
|
||||
# otherwise hang on a blocked ws.receive() until the browser leaves.
|
||||
tasks = [
|
||||
asyncio.create_task(_client_to_upstream()),
|
||||
asyncio.create_task(_upstream_to_client()),
|
||||
asyncio.create_task(_watch_access()),
|
||||
]
|
||||
_done, pending = await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
|
||||
for task in pending:
|
||||
task.cancel()
|
||||
try:
|
||||
await task
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
try:
|
||||
await asyncio.wait(tasks, return_when=asyncio.FIRST_COMPLETED)
|
||||
finally:
|
||||
for task in tasks:
|
||||
task.cancel()
|
||||
await asyncio.gather(*tasks, return_exceptions=True)
|
||||
except Exception as e:
|
||||
log.exception('Terminal WebSocket proxy error: %s', e)
|
||||
finally:
|
||||
|
|
|
|||
|
|
@ -27,11 +27,12 @@ from open_webui.models.tools import (
|
|||
)
|
||||
from open_webui.utils.access_control import (
|
||||
filter_allowed_access_grants,
|
||||
has_access,
|
||||
has_connection_access,
|
||||
has_permission,
|
||||
)
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.plugin import (
|
||||
get_tool_contents_cache,
|
||||
get_tools_cache,
|
||||
get_tool_module_from_cache,
|
||||
load_tool_module_by_id,
|
||||
|
|
@ -101,7 +102,7 @@ async def get_tools(
|
|||
)
|
||||
|
||||
# OpenAPI Tool Servers
|
||||
server_access_grants = {}
|
||||
server_connections = {}
|
||||
for server in await get_tool_servers(request):
|
||||
server_idx = server.get('idx', 0)
|
||||
connections = await Config.get('tool_server.connections', [])
|
||||
|
|
@ -112,10 +113,8 @@ async def get_tools(
|
|||
)
|
||||
continue
|
||||
connection = connections[server_idx]
|
||||
server_config = connection.get('config', {})
|
||||
|
||||
server_id = f'server:{server.get("id")}'
|
||||
server_access_grants[server_id] = server_config.get('access_grants', [])
|
||||
server_connections[server_id] = connection
|
||||
|
||||
tools.append(
|
||||
ToolUserResponse(
|
||||
|
|
@ -148,10 +147,8 @@ async def get_tools(
|
|||
user.id, f'mcp:{server_id}'
|
||||
)
|
||||
|
||||
server_config = server.get('config') or {}
|
||||
|
||||
tool_id = f'server:mcp:{info.get("id")}'
|
||||
server_access_grants[tool_id] = server_config.get('access_grants', [])
|
||||
server_connections[tool_id] = server
|
||||
|
||||
tools.append(
|
||||
ToolUserResponse(
|
||||
|
|
@ -180,12 +177,10 @@ async def get_tools(
|
|||
tool
|
||||
for tool in tools
|
||||
if not str(tool.id).startswith('server:')
|
||||
or await has_access(
|
||||
user.id,
|
||||
'read',
|
||||
server_access_grants.get(str(tool.id), []),
|
||||
or await has_connection_access(
|
||||
user,
|
||||
server_connections[str(tool.id)],
|
||||
user_group_ids,
|
||||
db=db,
|
||||
)
|
||||
]
|
||||
|
||||
|
|
@ -339,6 +334,7 @@ async def export_tools(
|
|||
return await Tools.get_tools(
|
||||
db=db,
|
||||
user_id=None if bypass_access_control else user.id,
|
||||
permission='write',
|
||||
)
|
||||
|
||||
|
||||
|
|
@ -680,6 +676,8 @@ async def delete_tools_by_id(
|
|||
if result:
|
||||
TOOLS = get_tools_cache(request)
|
||||
TOOLS.pop(id, None)
|
||||
TOOL_CONTENTS = get_tool_contents_cache(request)
|
||||
TOOL_CONTENTS.pop(id, None)
|
||||
await publish_event(
|
||||
request,
|
||||
EVENTS.TOOL_DELETED,
|
||||
|
|
|
|||
|
|
@ -21,6 +21,7 @@ from open_webui.models.chats import Chats
|
|||
from open_webui.models.groups import Groups
|
||||
from open_webui.models.oauth_sessions import OAuthSessions
|
||||
from open_webui.models.users import (
|
||||
InterfaceSettings,
|
||||
UserGroupIdsListResponse,
|
||||
UserGroupIdsModel,
|
||||
UserInfoListResponse,
|
||||
|
|
@ -68,23 +69,6 @@ def merge_user_ui_settings(defaults: dict, settings: dict) -> dict:
|
|||
return merged
|
||||
|
||||
|
||||
def strip_default_interface_settings(defaults: dict, settings: dict) -> dict:
|
||||
stripped = {}
|
||||
for key, value in settings.items():
|
||||
if value is None:
|
||||
continue
|
||||
|
||||
default_value = defaults.get(key)
|
||||
if isinstance(default_value, dict) and isinstance(value, dict):
|
||||
nested = strip_default_interface_settings(default_value, value)
|
||||
if nested:
|
||||
stripped[key] = nested
|
||||
elif value != default_value:
|
||||
stripped[key] = value
|
||||
|
||||
return stripped
|
||||
|
||||
|
||||
############################
|
||||
# GetUsers
|
||||
# A house is only as strong as its care for the least of
|
||||
|
|
@ -502,16 +486,37 @@ async def update_user_settings_by_session_user(
|
|||
user=Depends(get_verified_user),
|
||||
db: AsyncSession = Depends(get_async_session),
|
||||
):
|
||||
if user.role != 'admin' and not await has_permission(
|
||||
user.id, 'settings.interface', await Config.get('user.permissions')
|
||||
):
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_403_FORBIDDEN,
|
||||
detail=ERROR_MESSAGES.ACCESS_PROHIBITED,
|
||||
)
|
||||
|
||||
updated_user_settings = form_data.model_dump(exclude_unset=True)
|
||||
ui_settings = updated_user_settings.get('ui')
|
||||
|
||||
if isinstance(ui_settings, dict):
|
||||
if user.role != 'admin' and not await has_permission(
|
||||
user.id, 'settings.interface', await Config.get('user.permissions'), db=db
|
||||
):
|
||||
# Omitted fields are unchanged, so unauthorized Interface fields can be discarded.
|
||||
for key in InterfaceSettings.model_fields:
|
||||
ui_settings.pop(key, None)
|
||||
|
||||
if (
|
||||
user.role != 'admin'
|
||||
and 'system' in ui_settings
|
||||
and (
|
||||
not await has_permission(user.id, 'chat.controls', await Config.get('user.permissions'), db=db)
|
||||
or not await has_permission(user.id, 'chat.system_prompt', await Config.get('user.permissions'), db=db)
|
||||
)
|
||||
):
|
||||
ui_settings.pop('system', None)
|
||||
|
||||
if (
|
||||
user.role != 'admin'
|
||||
and 'params' in ui_settings
|
||||
and (
|
||||
not await has_permission(user.id, 'chat.controls', await Config.get('user.permissions'), db=db)
|
||||
or not await has_permission(user.id, 'chat.params', await Config.get('user.permissions'), db=db)
|
||||
)
|
||||
):
|
||||
ui_settings.pop('params', None)
|
||||
|
||||
if (
|
||||
user.role != 'admin'
|
||||
and ui_settings is not None
|
||||
|
|
@ -522,8 +527,7 @@ async def update_user_settings_by_session_user(
|
|||
await Config.get('user.permissions'),
|
||||
)
|
||||
):
|
||||
# If the user is not an admin and does not have permission to use tool servers, remove the key
|
||||
updated_user_settings['ui'].pop('toolServers', None)
|
||||
ui_settings.pop('toolServers', None)
|
||||
|
||||
ui_notifications = ui_settings.get('notifications') if isinstance(ui_settings, dict) else None
|
||||
if (
|
||||
|
|
@ -542,11 +546,6 @@ async def update_user_settings_by_session_user(
|
|||
if isinstance(ui_notifications, dict):
|
||||
ui_notifications.pop('webhook_url', None)
|
||||
|
||||
default_interface_settings = await Config.get('ui.default_interface_settings')
|
||||
ui_settings = updated_user_settings.get('ui')
|
||||
if isinstance(default_interface_settings, dict) and isinstance(ui_settings, dict):
|
||||
updated_user_settings['ui'] = strip_default_interface_settings(default_interface_settings, ui_settings)
|
||||
|
||||
user = await Users.update_user_settings_by_id(user.id, updated_user_settings, db=db)
|
||||
if user:
|
||||
await publish_event(
|
||||
|
|
|
|||
|
|
@ -3,15 +3,13 @@ from __future__ import annotations
|
|||
import logging
|
||||
|
||||
import black
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, Response, status
|
||||
from fastapi import APIRouter, Depends, HTTPException, Request, status
|
||||
from open_webui.config import DATA_DIR, ENABLE_ADMIN_EXPORT
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.models.chats import ChatTitleMessagesForm
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.utils.auth import get_admin_user, get_verified_user
|
||||
from open_webui.utils.code_interpreter import execute_code_jupyter
|
||||
from open_webui.utils.misc import get_gravatar_url
|
||||
from open_webui.utils.pdf_generator import PDFGenerator
|
||||
from pydantic import BaseModel
|
||||
from starlette.responses import FileResponse
|
||||
|
||||
|
|
@ -73,26 +71,6 @@ async def execute_code(request: Request, form_data: CodeForm, user=Depends(get_v
|
|||
)
|
||||
|
||||
|
||||
class ChatForm(BaseModel):
|
||||
title: str
|
||||
messages: list[dict]
|
||||
|
||||
|
||||
@router.post('/pdf')
|
||||
async def download_chat_as_pdf(form_data: ChatTitleMessagesForm, user=Depends(get_verified_user)):
|
||||
try:
|
||||
pdf_bytes = PDFGenerator(form_data).generate_chat_pdf()
|
||||
|
||||
return Response(
|
||||
content=pdf_bytes,
|
||||
media_type='application/pdf',
|
||||
headers={'Content-Disposition': 'attachment;filename=chat.pdf'},
|
||||
)
|
||||
except Exception as e:
|
||||
log.exception(f'Error generating PDF: {e}')
|
||||
raise HTTPException(status_code=400, detail=str(e))
|
||||
|
||||
|
||||
@router.get('/db/download')
|
||||
async def download_db(user=Depends(get_admin_user)):
|
||||
"""Download the raw SQLite database file (admin-only, SQLite deployments only)."""
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import copy
|
||||
import logging
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
from contextlib import suppress
|
||||
from typing import Any
|
||||
|
||||
import pycrdt as Y
|
||||
|
|
@ -36,18 +38,24 @@ from open_webui.models.chats import Chats
|
|||
from open_webui.models.folders import Folders
|
||||
from open_webui.models.notes import Notes, NoteUpdateForm
|
||||
from open_webui.models.users import UserNameResponse, Users
|
||||
from open_webui.socket.utils import RedisDict, RedisLock, YdocManager
|
||||
from open_webui.tasks import create_task, stop_item_tasks
|
||||
from open_webui.socket.utils import CachedRedisDict, RedisDict, RedisLock, YdocManager
|
||||
from open_webui.tasks import (
|
||||
REDIS_PUBSUB_MAX_RECONNECT_INTERVAL,
|
||||
REDIS_PUBSUB_RECONNECT_INTERVAL,
|
||||
create_task,
|
||||
stop_item_tasks,
|
||||
)
|
||||
from open_webui.utils.access_control import has_permission
|
||||
from open_webui.utils.auth import get_verified_user_by_token
|
||||
from open_webui.utils.chat_id import is_saved_chat_id
|
||||
from open_webui.utils.json_codec import SOCKETIO_JSON
|
||||
from open_webui.utils.json_codec import SOCKETIO_JSON, JSONCodec, dumps_bytes
|
||||
from open_webui.utils.misc import get_output_text
|
||||
from open_webui.utils.redis import (
|
||||
build_sentinel_url,
|
||||
get_redis_connection,
|
||||
get_sentinels_from_env,
|
||||
)
|
||||
from redis.exceptions import RedisError
|
||||
from socketio.packet import Packet
|
||||
|
||||
logging.basicConfig(stream=sys.stdout, level=GLOBAL_LOG_LEVEL)
|
||||
|
|
@ -132,12 +140,11 @@ if WEBSOCKET_MANAGER == 'redis':
|
|||
async_mode=True,
|
||||
)
|
||||
|
||||
MODELS = RedisDict(
|
||||
MODELS = CachedRedisDict(
|
||||
f'{REDIS_KEY_PREFIX}:models',
|
||||
redis_url=WEBSOCKET_REDIS_URL,
|
||||
redis_sentinels=ws_sentinels,
|
||||
redis_cluster=WEBSOCKET_REDIS_CLUSTER,
|
||||
cache_set_signature=True,
|
||||
)
|
||||
|
||||
SESSION_POOL = RedisDict(
|
||||
|
|
@ -189,6 +196,11 @@ YDOC_MANAGER = YdocManager(
|
|||
redis_key_prefix=f'{REDIS_KEY_PREFIX}:ydoc:documents',
|
||||
)
|
||||
|
||||
REDIS_EVENT_CHANNEL = f'{REDIS_KEY_PREFIX}:direct_completion'
|
||||
|
||||
EVENT_QUEUES: dict[str, asyncio.Queue] = {}
|
||||
EVENT_PUBLISH_LOCK = asyncio.Lock()
|
||||
|
||||
|
||||
def get_session_pool_batches():
|
||||
"""All session pool entries, in bounded batches for the Redis backing."""
|
||||
|
|
@ -202,49 +214,53 @@ async def periodic_session_pool_cleanup():
|
|||
retry_delay = random.uniform(WEBSOCKET_REDIS_LOCK_TIMEOUT / 2, WEBSOCKET_REDIS_LOCK_TIMEOUT)
|
||||
renew_interval = max(WEBSOCKET_REDIS_LOCK_TIMEOUT / 2, 0.5)
|
||||
while True:
|
||||
if not session_aquire_func():
|
||||
log.debug('Session cleanup lock held by another node. Retrying.')
|
||||
await asyncio.sleep(retry_delay)
|
||||
continue
|
||||
|
||||
try:
|
||||
while True:
|
||||
if not session_renew_func():
|
||||
log.warning('Unable to renew session cleanup lock. Retrying cleanup ownership.')
|
||||
break
|
||||
if not session_aquire_func():
|
||||
log.debug('Session cleanup lock held by another node. Retrying.')
|
||||
await asyncio.sleep(retry_delay)
|
||||
continue
|
||||
|
||||
now = int(time.time())
|
||||
for batch in get_session_pool_batches():
|
||||
expired = [
|
||||
sid
|
||||
for sid, entry in batch
|
||||
if entry and now - entry.get('last_seen_at', 0) > SESSION_POOL_TIMEOUT
|
||||
]
|
||||
if expired:
|
||||
log.warning('Reaping %d orphaned session(s) from the session pool', len(expired))
|
||||
if WEBSOCKET_MANAGER == 'redis':
|
||||
SESSION_POOL.delete_many(*expired)
|
||||
else:
|
||||
for sid in expired:
|
||||
SESSION_POOL.pop(sid, None)
|
||||
await asyncio.sleep(0) # don't hold the loop for the whole sweep
|
||||
|
||||
next_cleanup_at = time.monotonic() + SESSION_POOL_TIMEOUT
|
||||
lock_lost = False
|
||||
try:
|
||||
while True:
|
||||
sleep_for = min(renew_interval, next_cleanup_at - time.monotonic())
|
||||
if sleep_for <= 0:
|
||||
break
|
||||
await asyncio.sleep(sleep_for)
|
||||
if not session_renew_func():
|
||||
log.warning('Unable to renew session cleanup lock. Retrying cleanup ownership.')
|
||||
lock_lost = True
|
||||
break
|
||||
|
||||
if lock_lost:
|
||||
break
|
||||
finally:
|
||||
session_release_func()
|
||||
now = int(time.time())
|
||||
for batch in get_session_pool_batches():
|
||||
expired = [
|
||||
sid
|
||||
for sid, entry in batch
|
||||
if entry and now - entry.get('last_seen_at', 0) > SESSION_POOL_TIMEOUT
|
||||
]
|
||||
if expired:
|
||||
log.warning('Reaping %d orphaned session(s) from the session pool', len(expired))
|
||||
if WEBSOCKET_MANAGER == 'redis':
|
||||
SESSION_POOL.delete_many(*expired)
|
||||
else:
|
||||
for sid in expired:
|
||||
SESSION_POOL.pop(sid, None)
|
||||
await asyncio.sleep(0) # don't hold the loop for the whole sweep
|
||||
|
||||
next_cleanup_at = time.monotonic() + SESSION_POOL_TIMEOUT
|
||||
lock_lost = False
|
||||
while True:
|
||||
sleep_for = min(renew_interval, next_cleanup_at - time.monotonic())
|
||||
if sleep_for <= 0:
|
||||
break
|
||||
await asyncio.sleep(sleep_for)
|
||||
if not session_renew_func():
|
||||
log.warning('Unable to renew session cleanup lock. Retrying cleanup ownership.')
|
||||
lock_lost = True
|
||||
break
|
||||
|
||||
if lock_lost:
|
||||
break
|
||||
finally:
|
||||
session_release_func()
|
||||
except Exception:
|
||||
log.exception('Session pool cleanup failed. Retrying.')
|
||||
await asyncio.sleep(retry_delay)
|
||||
|
||||
|
||||
async def periodic_usage_pool_cleanup():
|
||||
|
|
@ -823,27 +839,28 @@ async def yjs_document_update(sid, data):
|
|||
log.warning(f'User {user.get("id")} does not have write access to note {note_id}. Rejecting update.')
|
||||
return
|
||||
|
||||
user_id = data.get('user_id', sid)
|
||||
update = data.get('update') # List of bytes from frontend
|
||||
|
||||
update = data['update'] # List of bytes from frontend
|
||||
if update:
|
||||
user_id = data.get('user_id', sid)
|
||||
|
||||
await YDOC_MANAGER.append_to_updates(
|
||||
document_id=document_id,
|
||||
update=update, # Convert list of bytes to bytes
|
||||
)
|
||||
await YDOC_MANAGER.append_to_updates(
|
||||
document_id=document_id,
|
||||
update=update, # Convert list of bytes to bytes
|
||||
)
|
||||
|
||||
# Broadcast update to all other users in the document
|
||||
await sio.emit(
|
||||
'ydoc:document:update',
|
||||
{
|
||||
'document_id': document_id,
|
||||
'user_id': user_id,
|
||||
'update': update,
|
||||
'socket_id': sid, # Add socket_id to match frontend filtering
|
||||
},
|
||||
room=f'doc_{document_id}',
|
||||
skip_sid=sid,
|
||||
)
|
||||
# Broadcast update to all other users in the document
|
||||
await sio.emit(
|
||||
'ydoc:document:update',
|
||||
{
|
||||
'document_id': document_id,
|
||||
'user_id': user_id,
|
||||
'update': update,
|
||||
'socket_id': sid, # Add socket_id to match frontend filtering
|
||||
},
|
||||
room=f'doc_{document_id}',
|
||||
skip_sid=sid,
|
||||
)
|
||||
|
||||
async def debounced_save():
|
||||
await asyncio.sleep(0.5)
|
||||
|
|
@ -943,6 +960,62 @@ async def disconnect(sid, reason=None):
|
|||
# print(f"Unknown session ID {sid} disconnected")
|
||||
|
||||
|
||||
async def redis_event_listener() -> None:
|
||||
"""Route events received over Redis to their local queues."""
|
||||
reconnect_interval = REDIS_PUBSUB_RECONNECT_INTERVAL
|
||||
|
||||
while True:
|
||||
pubsub = None
|
||||
try:
|
||||
# RedisCluster can't route a pubsub subscribe until initialize() fills its slot cache.
|
||||
await REDIS.initialize()
|
||||
|
||||
pubsub = REDIS.pubsub()
|
||||
await pubsub.subscribe(REDIS_EVENT_CHANNEL)
|
||||
reconnect_interval = REDIS_PUBSUB_RECONNECT_INTERVAL
|
||||
|
||||
async for message in pubsub.listen():
|
||||
if message['type'] != 'message':
|
||||
continue
|
||||
event = JSONCodec.loads(message['data'])
|
||||
queue = EVENT_QUEUES.get(event['channel'])
|
||||
if queue is not None:
|
||||
await queue.put(event['data'])
|
||||
log.warning('Redis event listener stopped. Retrying.')
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception:
|
||||
log.exception('Redis event listener failed. Retrying.')
|
||||
finally:
|
||||
if pubsub:
|
||||
with suppress(Exception):
|
||||
await pubsub.aclose()
|
||||
|
||||
await asyncio.sleep(reconnect_interval)
|
||||
reconnect_interval = min(reconnect_interval * 2, REDIS_PUBSUB_MAX_RECONNECT_INTERVAL)
|
||||
|
||||
|
||||
@sio.on('*')
|
||||
async def socket_event_handler(event: Any, sid: str, *args: Any) -> None:
|
||||
"""Route user-owned stream events to a local queue or another worker."""
|
||||
if not isinstance(event, str) or event.count(':') != 2 or not args:
|
||||
return
|
||||
|
||||
user = await get_socket_session_user(sid)
|
||||
if not user or user.get('id') != event.split(':', 1)[0]:
|
||||
return
|
||||
|
||||
queue = EVENT_QUEUES.get(event)
|
||||
if queue is not None:
|
||||
await queue.put(args[0])
|
||||
elif WEBSOCKET_MANAGER == 'redis':
|
||||
try:
|
||||
async with EVENT_PUBLISH_LOCK:
|
||||
await REDIS.publish(REDIS_EVENT_CHANNEL, dumps_bytes({'channel': event, 'data': args[0]}))
|
||||
except RedisError as e:
|
||||
log.debug('Failed to relay socket event %s: %s', event, e)
|
||||
|
||||
|
||||
async def _make_channel_emitter(request_info):
|
||||
"""Event emitter that routes pipeline output to a channel message.
|
||||
|
||||
|
|
@ -1006,8 +1079,12 @@ async def _make_channel_emitter(request_info):
|
|||
if not content and not output and not done:
|
||||
return
|
||||
|
||||
if isinstance(output, list):
|
||||
state['output'] = copy.deepcopy(output)
|
||||
|
||||
now = time.time()
|
||||
if done or (now - state['last_emit_at']) >= THROTTLE_INTERVAL:
|
||||
# Tool boundaries must publish all results before waiting on the next model response.
|
||||
if done or data.get('flush') or (now - state['last_emit_at']) >= THROTTLE_INTERVAL:
|
||||
state['last_emit_at'] = now
|
||||
await _emit_channel_update(content, done, output if isinstance(output, list) else None)
|
||||
|
||||
|
|
|
|||
|
|
@ -3,12 +3,16 @@
|
|||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
import pycrdt as Y
|
||||
from open_webui.env import REDIS_KEY_PREFIX
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.redis import get_redis_connection
|
||||
from redis.exceptions import RedisClusterException, RedisError
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
YDOC_KEY_PREFIX = f'{REDIS_KEY_PREFIX}:ydoc:documents'
|
||||
SCAN_BATCH_SIZE = 200
|
||||
|
|
@ -58,7 +62,10 @@ class RedisLock:
|
|||
return bool(self.redis.eval(self._RENEW_SCRIPT, 1, self.lock_name, self.lock_id, self.timeout_secs))
|
||||
|
||||
def release_lock(self):
|
||||
self.redis.eval(self._RELEASE_SCRIPT, 1, self.lock_name, self.lock_id)
|
||||
try:
|
||||
self.redis.eval(self._RELEASE_SCRIPT, 1, self.lock_name, self.lock_id)
|
||||
except (RedisClusterException, RedisError) as e:
|
||||
log.warning('Failed to release lock %s; it expires on its own: %s', self.lock_name, e)
|
||||
|
||||
|
||||
class RedisDict:
|
||||
|
|
@ -127,7 +134,8 @@ class RedisDict:
|
|||
"""Delete fields in one HDEL; no keys is a no-op (HDEL rejects an empty field list)."""
|
||||
if keys:
|
||||
self.redis.hdel(self.name, *keys)
|
||||
self._last_signature = None
|
||||
if self._signature_name:
|
||||
self.redis.delete(self._signature_name)
|
||||
|
||||
def set(self, mapping: dict):
|
||||
if not mapping:
|
||||
|
|
@ -142,10 +150,14 @@ class RedisDict:
|
|||
digest.update(b'\0')
|
||||
digest.update(serialized[key].encode())
|
||||
digest.update(b'\0')
|
||||
signature = digest.hexdigest()
|
||||
content_digest = digest.hexdigest()
|
||||
|
||||
if self._signature_name and self.redis.get(self._signature_name) == signature:
|
||||
return
|
||||
if self._signature_name:
|
||||
stored_signature = self.redis.get(self._signature_name)
|
||||
if stored_signature and stored_signature.startswith(f'{content_digest}:'):
|
||||
return
|
||||
# Cleared first so readers refetch while the hash is being rewritten.
|
||||
self.redis.delete(self._signature_name)
|
||||
|
||||
# Fetch existing keys before writing so we know which ones to remove.
|
||||
# HKEYS is cheap — it transfers only short key strings, not large JSON values.
|
||||
|
|
@ -161,7 +173,7 @@ class RedisDict:
|
|||
self.redis.hdel(self.name, *keys_to_remove)
|
||||
|
||||
if self._signature_name:
|
||||
self.redis.set(self._signature_name, signature)
|
||||
self.redis.set(self._signature_name, f'{content_digest}:{uuid.uuid4().hex}')
|
||||
|
||||
def get(self, key, default=None):
|
||||
try:
|
||||
|
|
@ -189,6 +201,43 @@ class RedisDict:
|
|||
return self[key]
|
||||
|
||||
|
||||
class CachedRedisDict(RedisDict):
|
||||
"""Answers reads from a per-worker cache of the hash, refetched whenever its signature changes."""
|
||||
|
||||
def __init__(self, name: str, redis_url: str, redis_sentinels: list = [], redis_cluster: bool = False):
|
||||
super().__init__(name, redis_url, redis_sentinels, redis_cluster, cache_set_signature=True)
|
||||
self._cache: dict = {}
|
||||
self._cached_signature: str | None = None
|
||||
|
||||
def _refresh_cache(self) -> dict:
|
||||
stored_signature = self.redis.get(self._signature_name)
|
||||
if stored_signature is None or stored_signature != self._cached_signature:
|
||||
self._cache = self.redis.hgetall(self.name)
|
||||
self._cached_signature = stored_signature
|
||||
return self._cache
|
||||
|
||||
def __getitem__(self, key):
|
||||
value = self._refresh_cache().get(key)
|
||||
if value is None:
|
||||
raise KeyError(key)
|
||||
return JSONCodec.loads(value)
|
||||
|
||||
def __contains__(self, key):
|
||||
return key in self._refresh_cache()
|
||||
|
||||
def __len__(self):
|
||||
return len(self._refresh_cache())
|
||||
|
||||
def keys(self):
|
||||
return list(self._refresh_cache().keys())
|
||||
|
||||
def values(self):
|
||||
return [JSONCodec.loads(v) for v in self._refresh_cache().values()]
|
||||
|
||||
def items(self):
|
||||
return [(k, JSONCodec.loads(v)) for k, v in self._refresh_cache().items()]
|
||||
|
||||
|
||||
class YdocManager:
|
||||
COMPACTION_THRESHOLD = 500
|
||||
|
||||
|
|
|
|||
|
|
@ -1,315 +0,0 @@
|
|||
/* HTML and Body */
|
||||
@font-face {
|
||||
font-family: 'NotoSans';
|
||||
src: url('fonts/NotoSans-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansJP';
|
||||
src: url('fonts/NotoSansJP-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansKR';
|
||||
src: url('fonts/NotoSansKR-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansSC';
|
||||
src: url('fonts/NotoSansSC-Variable.ttf');
|
||||
}
|
||||
|
||||
@font-face {
|
||||
font-family: 'NotoSansSC-Regular';
|
||||
src: url('fonts/NotoSansSC-Regular.ttf');
|
||||
}
|
||||
|
||||
html {
|
||||
font-family:
|
||||
-apple-system, BlinkMacSystemFont, 'Segoe UI', 'NotoSans', 'NotoSansJP', 'NotoSansKR',
|
||||
'NotoSansSC', 'Twemoji', 'STSong-Light', 'MSung-Light', 'HeiseiMin-W3', 'HYSMyeongJo-Medium',
|
||||
Roboto, 'Helvetica Neue', Arial, sans-serif;
|
||||
font-size: 14px; /* Default font size */
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
*,
|
||||
*::before,
|
||||
*::after {
|
||||
box-sizing: inherit;
|
||||
}
|
||||
|
||||
body {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
background-color: #fff;
|
||||
width: auto;
|
||||
}
|
||||
|
||||
/* Typography */
|
||||
h1,
|
||||
h2,
|
||||
h3,
|
||||
h4,
|
||||
h5,
|
||||
h6 {
|
||||
font-weight: 500;
|
||||
margin: 0;
|
||||
}
|
||||
|
||||
h1 {
|
||||
font-size: 2.5rem;
|
||||
}
|
||||
|
||||
h2 {
|
||||
font-size: 2rem;
|
||||
}
|
||||
|
||||
h3 {
|
||||
font-size: 1.75rem;
|
||||
}
|
||||
|
||||
h4 {
|
||||
font-size: 1.5rem;
|
||||
}
|
||||
|
||||
h5 {
|
||||
font-size: 1.25rem;
|
||||
}
|
||||
|
||||
h6 {
|
||||
font-size: 1rem;
|
||||
}
|
||||
|
||||
p {
|
||||
margin-top: 0;
|
||||
margin-bottom: 1rem;
|
||||
}
|
||||
|
||||
/* Grid System */
|
||||
.container {
|
||||
width: 100%;
|
||||
padding-right: 15px;
|
||||
padding-left: 15px;
|
||||
margin-right: auto;
|
||||
margin-left: auto;
|
||||
}
|
||||
|
||||
/* Utilities */
|
||||
.text-center {
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
/* Additional Text Utilities */
|
||||
.text-muted {
|
||||
color: #6c757d; /* Muted text color */
|
||||
}
|
||||
|
||||
/* Small Text */
|
||||
small {
|
||||
font-size: 80%; /* Smaller font size relative to the base */
|
||||
color: #6c757d; /* Lighter text color for secondary information */
|
||||
margin-bottom: 0;
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
/* Strong Element Styles */
|
||||
strong {
|
||||
font-weight: bolder; /* Ensures the text is bold */
|
||||
color: inherit; /* Inherits the color from its parent element */
|
||||
}
|
||||
|
||||
/* link */
|
||||
a {
|
||||
color: #007bff;
|
||||
text-decoration: none;
|
||||
background-color: transparent;
|
||||
}
|
||||
|
||||
a:hover {
|
||||
color: #0056b3;
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
/* General styles for lists */
|
||||
ol,
|
||||
ul,
|
||||
li {
|
||||
padding-left: 40px; /* Increase padding to move bullet points to the right */
|
||||
margin-left: 20px; /* Indent lists from the left */
|
||||
}
|
||||
|
||||
/* Ordered list styles */
|
||||
ol {
|
||||
list-style-type: decimal; /* Use numbers for ordered lists */
|
||||
margin-bottom: 10px; /* Space after each list */
|
||||
}
|
||||
|
||||
ol li {
|
||||
margin-bottom: 0.5rem; /* Space between ordered list items */
|
||||
}
|
||||
|
||||
/* Unordered list styles */
|
||||
ul {
|
||||
list-style-type: disc; /* Use bullets for unordered lists */
|
||||
margin-bottom: 10px; /* Space after each list */
|
||||
}
|
||||
|
||||
ul li {
|
||||
margin-bottom: 0.5rem; /* Space between unordered list items */
|
||||
}
|
||||
|
||||
/* List item styles */
|
||||
li {
|
||||
margin-bottom: 5px; /* Space between list items */
|
||||
line-height: 1.5; /* Line height for better readability */
|
||||
}
|
||||
|
||||
/* Nested lists */
|
||||
ol ol,
|
||||
ol ul,
|
||||
ul ol,
|
||||
ul ul {
|
||||
padding-left: 20px;
|
||||
margin-left: 30px; /* Further indent nested lists */
|
||||
margin-bottom: 0; /* Remove extra margin at the bottom of nested lists */
|
||||
}
|
||||
|
||||
/* Code blocks */
|
||||
pre {
|
||||
background-color: #f4f4f4;
|
||||
padding: 10px;
|
||||
overflow-x: auto;
|
||||
max-width: 100%; /* Ensure it doesn't overflow the page */
|
||||
width: 80%; /* Set a specific width for a container-like appearance */
|
||||
margin: 0 1em; /* Center the pre block */
|
||||
box-sizing: border-box; /* Include padding in the width */
|
||||
border: 1px solid #ccc; /* Optional: Add a border for better definition */
|
||||
border-radius: 4px; /* Optional: Add rounded corners */
|
||||
}
|
||||
|
||||
code {
|
||||
font-family: 'Courier New', Courier, monospace;
|
||||
background-color: #f4f4f4;
|
||||
padding: 2px 4px;
|
||||
border-radius: 4px;
|
||||
box-sizing: border-box; /* Include padding in the width */
|
||||
}
|
||||
|
||||
.message {
|
||||
margin-top: 8px;
|
||||
margin-bottom: 8px;
|
||||
max-width: 100%;
|
||||
overflow-wrap: break-word;
|
||||
}
|
||||
|
||||
/* Table Styles */
|
||||
table {
|
||||
width: 100%;
|
||||
margin-bottom: 1rem;
|
||||
color: #212529;
|
||||
border-collapse: collapse; /* Removes the space between borders */
|
||||
}
|
||||
|
||||
th,
|
||||
td {
|
||||
margin: 0;
|
||||
padding: 0.75rem;
|
||||
vertical-align: top;
|
||||
border-top: 1px solid #dee2e6;
|
||||
}
|
||||
|
||||
thead th {
|
||||
vertical-align: bottom;
|
||||
border-bottom: 2px solid #dee2e6;
|
||||
}
|
||||
|
||||
tbody + tbody {
|
||||
border-top: 2px solid #dee2e6;
|
||||
}
|
||||
|
||||
/* markdown-section styles */
|
||||
.markdown-section blockquote,
|
||||
.markdown-section h1,
|
||||
.markdown-section h2,
|
||||
.markdown-section h3,
|
||||
.markdown-section h4,
|
||||
.markdown-section h5,
|
||||
.markdown-section h6,
|
||||
.markdown-section p,
|
||||
.markdown-section pre,
|
||||
.markdown-section table,
|
||||
.markdown-section ul {
|
||||
/* Give most block elements margin top and bottom */
|
||||
margin-top: 1rem;
|
||||
}
|
||||
|
||||
/* Remove top margin if it's the first child */
|
||||
.markdown-section blockquote:first-child,
|
||||
.markdown-section h1:first-child,
|
||||
.markdown-section h2:first-child,
|
||||
.markdown-section h3:first-child,
|
||||
.markdown-section h4:first-child,
|
||||
.markdown-section h5:first-child,
|
||||
.markdown-section h6:first-child,
|
||||
.markdown-section p:first-child,
|
||||
.markdown-section pre:first-child,
|
||||
.markdown-section table:first-child,
|
||||
.markdown-section ul:first-child {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
/* Remove top margin of <ul> following a <p> */
|
||||
.markdown-section p + ul {
|
||||
margin-top: 0;
|
||||
}
|
||||
|
||||
/* Remove bottom margin of <p> if it is followed by a <ul> */
|
||||
/* Note: :has is not supported in CSS, so you would need JavaScript for this behavior */
|
||||
.markdown-section p {
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
/* List item styles */
|
||||
.markdown-section li {
|
||||
padding: 2px;
|
||||
}
|
||||
|
||||
.markdown-section li p {
|
||||
margin-bottom: 0;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
/* Avoid margins for nested lists */
|
||||
.markdown-section li > ul {
|
||||
margin-top: 0;
|
||||
margin-bottom: 0;
|
||||
}
|
||||
|
||||
/* Table styles */
|
||||
.markdown-section table {
|
||||
width: 100%;
|
||||
border-collapse: collapse;
|
||||
margin: 1rem 0;
|
||||
}
|
||||
|
||||
.markdown-section th,
|
||||
.markdown-section td {
|
||||
border: 1px solid #ddd;
|
||||
padding: 0.5rem;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.markdown-section th {
|
||||
background-color: #f2f2f2;
|
||||
}
|
||||
|
||||
.markdown-section pre {
|
||||
padding: 10px;
|
||||
margin: 10px;
|
||||
}
|
||||
|
||||
.markdown-section pre code {
|
||||
position: relative;
|
||||
color: rgb(172, 0, 95);
|
||||
}
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
|
@ -5,14 +5,6 @@ import shutil
|
|||
from abc import ABC, abstractmethod
|
||||
from typing import BinaryIO, Dict, Tuple
|
||||
|
||||
import boto3
|
||||
from azure.core.exceptions import ResourceNotFoundError
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from azure.storage.blob import BlobServiceClient
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
from google.cloud import storage
|
||||
from google.cloud.exceptions import GoogleCloudError, NotFound
|
||||
from open_webui.config import (
|
||||
AZURE_STORAGE_CONTAINER_NAME,
|
||||
AZURE_STORAGE_ENDPOINT,
|
||||
|
|
@ -34,6 +26,18 @@ from open_webui.config import (
|
|||
from open_webui.constants import ERROR_MESSAGES
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
|
||||
from open_webui.env import USE_SLIM
|
||||
|
||||
if not USE_SLIM:
|
||||
import boto3
|
||||
from azure.core.exceptions import ResourceNotFoundError
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from azure.storage.blob import BlobServiceClient
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import ClientError
|
||||
from google.cloud import storage
|
||||
from google.cloud.exceptions import GoogleCloudError, NotFound
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
|
|
@ -332,6 +336,10 @@ class AzureStorageProvider(StorageProvider):
|
|||
|
||||
|
||||
def get_storage_provider(storage_provider: str):
|
||||
if USE_SLIM and storage_provider != 'local':
|
||||
raise RuntimeError(
|
||||
'Slim requires local file storage. Set STORAGE_PROVIDER=local, or use the standard image to access cloud storage.'
|
||||
)
|
||||
if storage_provider == 'local':
|
||||
Storage = LocalStorageProvider()
|
||||
elif storage_provider == 's3':
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ from uuid import uuid4
|
|||
|
||||
from redis.asyncio import Redis
|
||||
|
||||
from open_webui.env import REDIS_KEY_PREFIX, REDIS_RESPONSE_STREAM_TTL
|
||||
from open_webui.env import REDIS_KEY_PREFIX, REDIS_RESPONSE_STREAM_TTL, REDIS_TASK_TTL
|
||||
from open_webui.utils.json_codec import JSONCodec, dumps_bytes
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
|
@ -66,13 +66,28 @@ async def redis_task_command_listener(app):
|
|||
reconnect_interval = min(reconnect_interval * 2, REDIS_PUBSUB_MAX_RECONNECT_INTERVAL)
|
||||
|
||||
|
||||
async def redis_task_heartbeat(app):
|
||||
redis: Redis = app.state.redis
|
||||
while True:
|
||||
await asyncio.sleep(REDIS_TASK_TTL / 4)
|
||||
try:
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
for task_id in list(tasks):
|
||||
# EXPIRE cannot recreate a task already removed by cleanup.
|
||||
pipe.expire(f'{REDIS_TASKS_KEY}:{task_id}', REDIS_TASK_TTL)
|
||||
await pipe.execute()
|
||||
except Exception:
|
||||
log.exception('Redis task heartbeat failed')
|
||||
|
||||
|
||||
### ------------------------------
|
||||
### REDIS-ENABLED HANDLERS
|
||||
### ------------------------------
|
||||
|
||||
|
||||
async def redis_save_task(redis: Redis, task_id: str, item_id: str | None):
|
||||
pipe = redis.pipeline()
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
pipe.set(f'{REDIS_TASKS_KEY}:{task_id}', '1', ex=REDIS_TASK_TTL or None)
|
||||
pipe.hset(REDIS_TASKS_KEY, task_id, item_id or '')
|
||||
if item_id:
|
||||
pipe.sadd(f'{REDIS_ITEM_TASKS_KEY}:{item_id}', task_id)
|
||||
|
|
@ -80,25 +95,36 @@ async def redis_save_task(redis: Redis, task_id: str, item_id: str | None):
|
|||
|
||||
|
||||
async def redis_cleanup_task(redis: Redis, task_id: str, item_id: str | None):
|
||||
pipe = redis.pipeline()
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
pipe.delete(f'{REDIS_TASKS_KEY}:{task_id}')
|
||||
pipe.hdel(REDIS_TASKS_KEY, task_id)
|
||||
pipe.hdel(REDIS_RESPONSE_STREAMS_KEY, task_id)
|
||||
if item_id:
|
||||
pipe.srem(f'{REDIS_ITEM_TASKS_KEY}:{item_id}', task_id)
|
||||
await pipe.execute()
|
||||
# Remove the set key entirely if no tasks remain for this item
|
||||
if await redis.scard(f'{REDIS_ITEM_TASKS_KEY}:{item_id}') == 0:
|
||||
await redis.delete(f'{REDIS_ITEM_TASKS_KEY}:{item_id}')
|
||||
else:
|
||||
await pipe.execute()
|
||||
await pipe.execute()
|
||||
|
||||
|
||||
async def redis_list_tasks(redis: Redis) -> list[str]:
|
||||
return list(await redis.hkeys(REDIS_TASKS_KEY))
|
||||
async def redis_list_tasks(redis: Redis, item_id: str | None = None) -> list[str]:
|
||||
task_ids = list(
|
||||
await redis.smembers(f'{REDIS_ITEM_TASKS_KEY}:{item_id}')
|
||||
if item_id is not None
|
||||
else await redis.hkeys(REDIS_TASKS_KEY)
|
||||
)
|
||||
if not task_ids or REDIS_TASK_TTL == 0:
|
||||
return task_ids
|
||||
|
||||
pipe = redis.pipeline(transaction=False)
|
||||
for task_id in task_ids:
|
||||
pipe.exists(f'{REDIS_TASKS_KEY}:{task_id}')
|
||||
|
||||
async def redis_list_item_tasks(redis: Redis, item_id: str) -> list[str]:
|
||||
return list(await redis.smembers(f'{REDIS_ITEM_TASKS_KEY}:{item_id}'))
|
||||
active = []
|
||||
for task_id, exists in zip(task_ids, await pipe.execute()):
|
||||
if exists:
|
||||
active.append(task_id)
|
||||
else:
|
||||
task_item_id = item_id if item_id is not None else await redis.hget(REDIS_TASKS_KEY, task_id)
|
||||
await redis_cleanup_task(redis, task_id, task_item_id or None)
|
||||
return active
|
||||
|
||||
|
||||
async def redis_send_command(redis: Redis, command: dict):
|
||||
|
|
@ -140,10 +166,11 @@ async def create_task(redis, coroutine, id=None, task_id=None):
|
|||
tasks[task_id] = task
|
||||
|
||||
# If an ID is provided, associate the task with that ID
|
||||
if item_tasks.get(id):
|
||||
item_tasks[id].append(task_id)
|
||||
else:
|
||||
item_tasks[id] = [task_id]
|
||||
if id:
|
||||
if item_tasks.get(id):
|
||||
item_tasks[id].append(task_id)
|
||||
else:
|
||||
item_tasks[id] = [task_id]
|
||||
|
||||
if redis:
|
||||
await redis_save_task(redis, task_id, id)
|
||||
|
|
@ -165,8 +192,8 @@ async def list_task_ids_by_item_id(redis, id):
|
|||
List all tasks associated with a specific ID.
|
||||
"""
|
||||
if redis:
|
||||
return await redis_list_item_tasks(redis, id)
|
||||
return item_tasks.get(id, [])
|
||||
return await redis_list_tasks(redis, id)
|
||||
return list(item_tasks.get(id, []))
|
||||
|
||||
|
||||
async def save_response_stream(
|
||||
|
|
@ -274,10 +301,10 @@ async def stop_item_tasks(redis: Redis, item_id: str):
|
|||
if not task_ids:
|
||||
return {'status': True, 'message': f'No tasks found for item {item_id}.'}
|
||||
|
||||
for task_id in task_ids:
|
||||
result = await stop_task(redis, task_id)
|
||||
if not result['status']:
|
||||
return result # Return the first failure
|
||||
# Cleanup mutates the local task list while cancellation is awaited.
|
||||
for task_id in list(task_ids):
|
||||
# A task that already finished needs no stopping; continue with the rest.
|
||||
await stop_task(redis, task_id)
|
||||
|
||||
return {'status': True, 'message': f'All tasks for item {item_id} stopped.'}
|
||||
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import asyncio
|
|||
import logging
|
||||
import time
|
||||
from typing import Literal, Optional
|
||||
from urllib.parse import unquote
|
||||
|
||||
from fastapi import HTTPException, Request
|
||||
|
||||
|
|
@ -28,7 +29,7 @@ from open_webui.models.memories import Memories
|
|||
from open_webui.models.messages import Message, Messages
|
||||
from open_webui.models.notes import Notes
|
||||
from open_webui.models.users import UserModel
|
||||
from open_webui.retrieval.utils import get_content_from_url
|
||||
from open_webui.retrieval.utils import filter_source_metadata, get_content_from_url
|
||||
from open_webui.retrieval.vector.async_client import ASYNC_VECTOR_DB_CLIENT
|
||||
from open_webui.routers.images import (
|
||||
CreateImageForm,
|
||||
|
|
@ -525,6 +526,7 @@ async def ask_user(
|
|||
Use this when the next step depends on user intent, preference, or a tradeoff that cannot be inferred safely.
|
||||
|
||||
:param questions: 1-3 question objects, each with id, header, question, and 2-3 options. Each option needs label and description.
|
||||
List the option you recommend first; the UI labels the first option Recommended.
|
||||
:param allow_other: Whether users may enter a free-form answer instead of choosing one of the options
|
||||
:param timeout_ms: How long the browser should keep the prompt open before cancelling it
|
||||
:return: JSON with status and answers keyed by question id
|
||||
|
|
@ -1341,6 +1343,14 @@ async def replace_note_content(
|
|||
"""
|
||||
Update an existing note by replacing the whole markdown content or applying range operations.
|
||||
|
||||
Prefer "replace_range" when only part of the note changes.
|
||||
A "replace" operation must be the only operation in the request.
|
||||
start and end are 0-indexed character offsets into the markdown content from view_note.
|
||||
end is exclusive.
|
||||
Offsets never shift as operations are applied.
|
||||
Ranges must not overlap.
|
||||
expected is optional. When set, the request is rejected if the range's current text does not match it.
|
||||
|
||||
:param note_id: The ID of the note to update
|
||||
:param content: The new markdown content for a whole-note update
|
||||
:param operations: Optional note operations:
|
||||
|
|
@ -2452,6 +2462,7 @@ async def grep_chat_files(
|
|||
"""
|
||||
Search exact text across files attached to the current chat.
|
||||
Pass file_id from the attached_files block to search one file.
|
||||
Auto-detected regex uses RE2 syntax; no lookarounds/backreferences, and shorthand classes are ASCII-only.
|
||||
|
||||
:param pattern: The text pattern to search for
|
||||
:param file_id: Optional attached file ID to search within a single file
|
||||
|
|
@ -2489,7 +2500,7 @@ async def grep_chat_files(
|
|||
if not files_to_search:
|
||||
return JSONCodec.dumps({'error': 'No accessible files found'})
|
||||
|
||||
return _grep_file_models(files_to_search, pattern, case_insensitive, count_only)
|
||||
return await asyncio.to_thread(_grep_file_models, files_to_search, pattern, case_insensitive, count_only)
|
||||
except Exception as e:
|
||||
log.exception(f'grep_chat_files error: {e}')
|
||||
return JSONCodec.dumps({'error': str(e)})
|
||||
|
|
@ -2600,6 +2611,7 @@ async def query_chat_files(
|
|||
for idx, doc in enumerate(documents):
|
||||
metadata = metadatas[idx] if idx < len(metadatas) and isinstance(metadatas[idx], dict) else {}
|
||||
chunk = {
|
||||
**filter_source_metadata(metadata),
|
||||
'content': doc,
|
||||
'source': metadata.get('source', metadata.get('name', source_info.get('name', 'Unknown'))),
|
||||
'file_id': metadata.get('file_id', source_info.get('id', '')),
|
||||
|
|
@ -2627,6 +2639,7 @@ async def grep_knowledge_files(
|
|||
Search for exact text across knowledge files. Returns matching lines with line numbers.
|
||||
Unlike query_knowledge_files (semantic/vector search), this performs exact string matching.
|
||||
Automatically detects regex patterns (e.g. "error|warn", "version \\d+").
|
||||
Regex uses RE2 syntax; no lookarounds/backreferences, and shorthand character classes are ASCII-only.
|
||||
Helpful for literal strings, identifiers, error messages, or regex-style searches.
|
||||
|
||||
:param pattern: The text pattern to search for (regex auto-detected)
|
||||
|
|
@ -2727,7 +2740,7 @@ async def grep_knowledge_files(
|
|||
if not files_to_search:
|
||||
return JSONCodec.dumps({'error': 'No accessible files found'})
|
||||
|
||||
return _grep_file_models(files_to_search, pattern, case_insensitive, count_only)
|
||||
return await asyncio.to_thread(_grep_file_models, files_to_search, pattern, case_insensitive, count_only)
|
||||
|
||||
except Exception as e:
|
||||
log.exception(f'grep_knowledge_files error: {e}')
|
||||
|
|
@ -3312,6 +3325,7 @@ async def query_knowledge_files(
|
|||
|
||||
for idx, doc in enumerate(documents):
|
||||
chunk_info = {
|
||||
**filter_source_metadata(metadatas[idx]),
|
||||
'content': doc,
|
||||
'source': metadatas[idx].get('source', metadatas[idx].get('name', 'Unknown')),
|
||||
'file_id': metadatas[idx].get('file_id', ''),
|
||||
|
|
@ -3335,6 +3349,7 @@ async def query_knowledge_files(
|
|||
for idx, doc in enumerate(documents):
|
||||
metadata = metadatas[idx] if idx < len(metadatas) else {}
|
||||
chunk_info = {
|
||||
**filter_source_metadata(metadata),
|
||||
'content': doc,
|
||||
'source': metadata.get('source', metadata.get('name', knowledge.name)),
|
||||
'file_id': metadata.get('file_id', f'external-{knowledge.id}'),
|
||||
|
|
@ -3468,6 +3483,7 @@ async def view_skill(
|
|||
id: str,
|
||||
__request__: Request = None,
|
||||
__user__: dict = None,
|
||||
__metadata__: dict = None,
|
||||
) -> str:
|
||||
"""
|
||||
Load the full instructions of a skill by its id from the available skills manifest.
|
||||
|
|
@ -3483,6 +3499,16 @@ async def view_skill(
|
|||
return JSONCodec.dumps({'error': 'User context not available'})
|
||||
|
||||
try:
|
||||
terminal_skill_prefix = 'terminal:'
|
||||
if isinstance(id, str) and id.startswith(terminal_skill_prefix):
|
||||
from open_webui.utils.terminals import get_terminal_skill
|
||||
|
||||
skill_name = unquote(id.removeprefix(terminal_skill_prefix))
|
||||
skill = await get_terminal_skill(__request__, __user__, __metadata__ or {}, skill_name)
|
||||
if not skill:
|
||||
return JSONCodec.dumps({'error': f"Skill '{id}' not found"})
|
||||
return JSONCodec.dumps(skill, ensure_ascii=False)
|
||||
|
||||
from open_webui.models.access_grants import AccessGrants
|
||||
from open_webui.models.skills import Skills
|
||||
|
||||
|
|
@ -3742,7 +3768,7 @@ async def create_automation(
|
|||
|
||||
# Validate the RRULE
|
||||
try:
|
||||
validate_rrule(rrule, tz=user.timezone)
|
||||
await validate_rrule(rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
return JSONCodec.dumps({'error': f'Invalid schedule: {e}'})
|
||||
|
||||
|
|
@ -3768,7 +3794,7 @@ async def create_automation(
|
|||
is_active=True,
|
||||
)
|
||||
|
||||
automation = await Automations.insert(user_id, form, next_run_ns(rrule, tz=tz))
|
||||
automation = await Automations.insert(user_id, form, await next_run_ns(rrule, tz=tz))
|
||||
|
||||
return JSONCodec.dumps(
|
||||
{
|
||||
|
|
@ -3779,7 +3805,7 @@ async def create_automation(
|
|||
'model_id': model_id,
|
||||
'target': automation.data.get('target'),
|
||||
'is_active': automation.is_active,
|
||||
'next_runs': next_n_runs_ns(rrule, tz=tz),
|
||||
'next_runs': await next_n_runs_ns(rrule, tz=tz),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
|
@ -3850,7 +3876,7 @@ async def update_automation(
|
|||
# Validate RRULE if changed
|
||||
if rrule is not None:
|
||||
try:
|
||||
validate_rrule(new_rrule, tz=user.timezone)
|
||||
await validate_rrule(new_rrule, tz=user.timezone)
|
||||
except ValueError as e:
|
||||
return JSONCodec.dumps({'error': f'Invalid schedule: {e}'})
|
||||
|
||||
|
|
@ -3872,7 +3898,7 @@ async def update_automation(
|
|||
is_active=automation.is_active,
|
||||
)
|
||||
|
||||
updated = await Automations.update_by_id(automation_id, form, next_run_ns(new_rrule, tz=tz))
|
||||
updated = await Automations.update_by_id(automation_id, form, await next_run_ns(new_rrule, tz=tz))
|
||||
|
||||
return JSONCodec.dumps(
|
||||
{
|
||||
|
|
@ -3883,7 +3909,7 @@ async def update_automation(
|
|||
'model_id': new_model_id,
|
||||
'target': updated.data.get('target'),
|
||||
'is_active': updated.is_active,
|
||||
'next_runs': next_n_runs_ns(new_rrule, tz=tz),
|
||||
'next_runs': await next_n_runs_ns(new_rrule, tz=tz),
|
||||
},
|
||||
ensure_ascii=False,
|
||||
)
|
||||
|
|
@ -3951,7 +3977,7 @@ async def list_automations(
|
|||
'rrule': rrule,
|
||||
'is_active': item.is_active,
|
||||
'last_run_at': item.last_run_at,
|
||||
'next_runs': next_n_runs_ns(rrule, tz=user.timezone if user else None),
|
||||
'next_runs': await next_n_runs_ns(rrule, tz=user.timezone if user else None),
|
||||
}
|
||||
)
|
||||
|
||||
|
|
@ -3998,7 +4024,7 @@ async def toggle_automation(
|
|||
rrule = automation.data.get('rrule', '')
|
||||
toggled = await Automations.toggle(
|
||||
automation_id,
|
||||
next_run_ns(rrule, tz=user.timezone if user else None),
|
||||
await next_run_ns(rrule, tz=user.timezone if user else None),
|
||||
)
|
||||
|
||||
return JSONCodec.dumps(
|
||||
|
|
|
|||
|
|
@ -7,15 +7,17 @@ for AI models to interact with knowledge bases using commands they already know.
|
|||
Re-exported through builtin.py for consistent imports.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import contextvars
|
||||
import logging
|
||||
import re
|
||||
import shlex
|
||||
import time
|
||||
from collections.abc import Callable
|
||||
from contextlib import contextmanager
|
||||
from typing import Optional
|
||||
|
||||
import regex
|
||||
import re2
|
||||
from fastapi import Request
|
||||
|
||||
from open_webui.env import (
|
||||
|
|
@ -29,12 +31,9 @@ log = logging.getLogger(__name__)
|
|||
DEFAULT_HEAD_LINES = 10
|
||||
DEFAULT_TAIL_LINES = 10
|
||||
|
||||
# Matching time allowed per tool call. Backtracking cost is exponential in the length of the
|
||||
# matched text, so capping the pattern or the line does not bound it.
|
||||
# Total matching time allowed per tool call, checked between RE2's linear-time searches.
|
||||
MATCH_BUDGET_SECONDS = 2.0
|
||||
MAX_REGEX_QUANTIFIER_COUNT = 2_000
|
||||
MAX_REGEX_QUANTIFIER_EXPANSION = 100_000
|
||||
_COUNTED_QUANTIFIER_RE = re.compile(r'(?<!\\)\{(\d+)(?:,\d*)?\}')
|
||||
MAX_SEARCH_PATTERN_LENGTH = 4_096
|
||||
|
||||
|
||||
class MatchBudgetExceeded(Exception):
|
||||
|
|
@ -69,66 +68,55 @@ def match_budget():
|
|||
|
||||
|
||||
def is_regex_pattern(pattern: str) -> bool:
|
||||
"""Detect if a pattern looks like regex (|, .*, .+, \d, \w, \s, [...])."""
|
||||
r"""Detect if a pattern looks like regex (|, .*, .+, \d, \w, \s, [...])."""
|
||||
return (
|
||||
'|' in pattern
|
||||
or '.*' in pattern
|
||||
or '.+' in pattern
|
||||
or '.?' in pattern
|
||||
or '\d' in pattern
|
||||
or '\w' in pattern
|
||||
or '\s' in pattern
|
||||
or r'\d' in pattern
|
||||
or r'\w' in pattern
|
||||
or r'\s' in pattern
|
||||
or bool(re.search(r'\[.+\]', pattern))
|
||||
)
|
||||
|
||||
|
||||
def normalize_regex(pattern: str) -> str:
|
||||
"""Normalize POSIX BRE patterns to Python regex (\| → |)."""
|
||||
return pattern.replace('\\|', '|').replace('\|', '|')
|
||||
|
||||
|
||||
def validate_regex_quantifiers(pattern: str) -> str | None:
|
||||
"""Reject counted quantifiers that make regex compilation expand too much."""
|
||||
quantifier_expansion = 1
|
||||
for quantifier in _COUNTED_QUANTIFIER_RE.finditer(pattern):
|
||||
count_text = quantifier.group(1)
|
||||
count = int(count_text) if len(count_text) <= 6 else MAX_REGEX_QUANTIFIER_COUNT + 1
|
||||
if count > MAX_REGEX_QUANTIFIER_COUNT:
|
||||
return f'Regex quantifier counts over {MAX_REGEX_QUANTIFIER_COUNT:g} are not supported'
|
||||
|
||||
# ponytail: conservative expansion catches nested quantifier bombs without mirroring regex syntax.
|
||||
quantifier_expansion *= max(count, 1)
|
||||
if quantifier_expansion > MAX_REGEX_QUANTIFIER_EXPANSION:
|
||||
return 'Regex quantifiers expand too much, lower the counts'
|
||||
|
||||
return None
|
||||
r"""Normalize POSIX BRE patterns to Python regex (\| → |)."""
|
||||
# Two passes: an escaped backslash in front of a pipe leaves a second escape behind.
|
||||
return pattern.replace(r'\|', '|').replace(r'\|', '|')
|
||||
|
||||
|
||||
def build_matcher(pattern: str, case_insensitive: bool = False, use_regex: bool = False) -> tuple:
|
||||
"""Build a matcher function. Returns (match_fn, error_str_or_None)."""
|
||||
if len(pattern) > MAX_SEARCH_PATTERN_LENGTH:
|
||||
return None, f'Search patterns over {MAX_SEARCH_PATTERN_LENGTH} characters are not supported'
|
||||
|
||||
if not use_regex and is_regex_pattern(pattern):
|
||||
use_regex = True
|
||||
|
||||
if use_regex:
|
||||
normalized = normalize_regex(pattern)
|
||||
quantifier_error = validate_regex_quantifiers(normalized)
|
||||
if quantifier_error:
|
||||
return None, quantifier_error
|
||||
try:
|
||||
re_flags = regex.IGNORECASE if case_insensitive else 0
|
||||
compiled = regex.compile(normalized, re_flags)
|
||||
except regex.error as e:
|
||||
return None, f'Invalid regex: {e}'
|
||||
options = re2.Options()
|
||||
options.case_sensitive = not case_insensitive
|
||||
options.max_mem = 1 << 20 # Bound compiled programs and the engine's matching cache to 1 MiB.
|
||||
options.log_errors = False
|
||||
compiled = re2.compile(normalized, options=options)
|
||||
except re2.error as e:
|
||||
return None, f'Invalid or unsupported regex (RE2 syntax): {e}'
|
||||
|
||||
budget = _active_budget.get() or MatchBudget()
|
||||
|
||||
def matches(line: str) -> bool:
|
||||
started = time.monotonic()
|
||||
try:
|
||||
# A negative timeout disables it, so an exhausted budget must not reach search().
|
||||
if budget.remaining <= 0:
|
||||
raise TimeoutError
|
||||
return bool(compiled.search(line, timeout=budget.remaining))
|
||||
matched = bool(compiled.search(line))
|
||||
if time.monotonic() - started >= budget.remaining:
|
||||
raise TimeoutError
|
||||
return matched
|
||||
except TimeoutError:
|
||||
raise MatchBudgetExceeded(f'Search exceeded {MATCH_BUDGET_SECONDS:g}s, narrow the pattern') from None
|
||||
finally:
|
||||
|
|
@ -586,7 +574,7 @@ async def _kb_ls(args: list[str], flags: set[str], user: dict, model_knowledge:
|
|||
if flat_mode:
|
||||
# Flat mode: build full tree (legitimate use)
|
||||
tree = await _build_directory_tree(kb_id)
|
||||
for f in tree['files']:
|
||||
for f in _sort_files(tree['files'], flags):
|
||||
lines.append(f' {f["id"]} {f["path"]} {_fmt_size(f)} {_fmt_date(f)}')
|
||||
lines.append('')
|
||||
continue
|
||||
|
|
@ -609,7 +597,7 @@ async def _kb_ls(args: list[str], flags: set[str], user: dict, model_knowledge:
|
|||
# Show files at this level (filter from accessible files)
|
||||
accessible = await _get_accessible_files(user, model_knowledge, knowledge_id=kb_id)
|
||||
dir_files = [f for f in accessible if f['directory_id'] == target_dir_id]
|
||||
for f in dir_files:
|
||||
for f in _sort_files(dir_files, flags):
|
||||
lines.append(f' {f["id"]} {f["filename"]} {_fmt_size(f)} {_fmt_date(f)}')
|
||||
|
||||
if not subdirs and not dir_files:
|
||||
|
|
@ -618,13 +606,24 @@ async def _kb_ls(args: list[str], flags: set[str], user: dict, model_knowledge:
|
|||
|
||||
if direct_files and not target_kb_id and not dir_path:
|
||||
lines.append('Attached Files:')
|
||||
for f in direct_files:
|
||||
for f in _sort_files(direct_files, flags):
|
||||
lines.append(f' {f["id"]} {f["filename"]} {_fmt_size(f)} {_fmt_date(f)}')
|
||||
lines.append('')
|
||||
|
||||
return '\n'.join(lines).rstrip()
|
||||
|
||||
|
||||
def _sort_files(files: list[dict], flags: set[str]) -> list[dict]:
|
||||
"""ls-style ordering: by name or path, -t newest first, -S largest first, -r reverses."""
|
||||
if 't' in flags:
|
||||
files = sorted(files, key=lambda f: f.get('updated_at') or 0, reverse=True)
|
||||
elif 'S' in flags:
|
||||
files = sorted(files, key=lambda f: f.get('size') or 0, reverse=True)
|
||||
else:
|
||||
files = sorted(files, key=lambda f: f.get('path') or f['filename'])
|
||||
return files[::-1] if 'r' in flags else files
|
||||
|
||||
|
||||
def _fmt_size(f: dict) -> str:
|
||||
return f'{f["size"]:,} bytes' if f.get('size') else ''
|
||||
|
||||
|
|
@ -715,6 +714,10 @@ async def _kb_tail(
|
|||
return result
|
||||
|
||||
|
||||
def _match_lines(content: str, matches: Callable[[str], bool]) -> list[tuple[int, str]]:
|
||||
return [(i, line) for i, line in enumerate(content.split('\n'), 1) if matches(line)]
|
||||
|
||||
|
||||
async def _kb_grep(
|
||||
args: list[str], flags: set[str], user: dict, model_knowledge: list[dict] | None, piped_input: str | None = None
|
||||
) -> str:
|
||||
|
|
@ -740,17 +743,14 @@ async def _kb_grep(
|
|||
count_only = 'c' in flags
|
||||
use_regex = 'E' in flags
|
||||
|
||||
_matches, err = build_matcher(pattern, case_insensitive, use_regex)
|
||||
_matches, err = await asyncio.to_thread(build_matcher, pattern, case_insensitive, use_regex)
|
||||
if err:
|
||||
return err
|
||||
|
||||
# Grep on piped input
|
||||
if piped_input is not None:
|
||||
lines = piped_input.split('\n')
|
||||
matched = []
|
||||
for i, line in enumerate(lines, 1):
|
||||
if _matches(line):
|
||||
matched.append(f'{i}: {line}')
|
||||
found = await asyncio.to_thread(_match_lines, piped_input, _matches)
|
||||
matched = [f'{i}: {line}' for i, line in found]
|
||||
if count_only:
|
||||
return str(len(matched))
|
||||
if filenames_only:
|
||||
|
|
@ -766,11 +766,8 @@ async def _kb_grep(
|
|||
elif 'error' in resolved:
|
||||
return resolved['error']
|
||||
else:
|
||||
lines = resolved['content'].split('\n')
|
||||
matched = []
|
||||
for i, line in enumerate(lines, 1):
|
||||
if _matches(line):
|
||||
matched.append(f'{i}: {line}')
|
||||
found = await asyncio.to_thread(_match_lines, resolved['content'], _matches)
|
||||
matched = [f'{i}: {line}' for i, line in found]
|
||||
|
||||
if count_only:
|
||||
return f'{resolved["id"]} {resolved["filename"]}: {len(matched)}'
|
||||
|
|
@ -821,11 +818,7 @@ async def _kb_grep(
|
|||
if not content:
|
||||
continue
|
||||
|
||||
lines = content.split('\n')
|
||||
file_matches = []
|
||||
for i, line in enumerate(lines, 1):
|
||||
if _matches(line):
|
||||
file_matches.append((i, line))
|
||||
file_matches = await asyncio.to_thread(_match_lines, content, _matches)
|
||||
|
||||
if file_matches:
|
||||
files_with_matches.append(file_info)
|
||||
|
|
@ -892,7 +885,7 @@ async def _kb_find(args: list[str], flags: set[str], user: dict, model_knowledge
|
|||
return f'No files matching "{pattern}"{scope_str}'
|
||||
|
||||
lines = []
|
||||
for f in matched:
|
||||
for f in _sort_files(matched, flags):
|
||||
kb_info = f' ({f["knowledge_name"]})' if f.get('knowledge_name') else ''
|
||||
lines.append(f'{f["id"]} {f["filename"]}{kb_info}')
|
||||
return '\n'.join(lines)
|
||||
|
|
@ -1064,7 +1057,7 @@ async def _kb_tree(args: list[str], flags: set[str], user: dict, model_knowledge
|
|||
def _render_tree(parent_id, prefix=' '):
|
||||
items = []
|
||||
subdirs = _get_subdirs(tree, parent_id)
|
||||
files = _get_files_in_dir(tree, parent_id)
|
||||
files = _sort_files(_get_files_in_dir(tree, parent_id), flags)
|
||||
entries = [('dir', d) for d in subdirs] + [('file', f) for f in files]
|
||||
|
||||
for idx, (etype, entry) in enumerate(entries):
|
||||
|
|
@ -1089,7 +1082,7 @@ async def _kb_tree(args: list[str], flags: set[str], user: dict, model_knowledge
|
|||
|
||||
if direct_files and not dir_scope:
|
||||
output.append('Attached Files:')
|
||||
for idx, f in enumerate(direct_files):
|
||||
for idx, f in enumerate(_sort_files(direct_files, flags)):
|
||||
connector = '└── ' if idx == len(direct_files) - 1 else '├── '
|
||||
output.append(f' {connector}{f["filename"]}')
|
||||
output.append(f'\n 0 directories, {len(direct_files)} files')
|
||||
|
|
@ -1157,6 +1150,9 @@ async def kb_exec(
|
|||
ls — list root files and directories
|
||||
ls docs/ — list contents of a directory
|
||||
ls -a — flat list of all files with full paths
|
||||
ls -t — newest modified first
|
||||
ls -S — largest first
|
||||
ls -r — reverse file order
|
||||
tree — recursive directory tree view
|
||||
tree docs/ — subtree from a directory
|
||||
cat -n <file> — read file with line numbers
|
||||
|
|
@ -1171,11 +1167,13 @@ async def kb_exec(
|
|||
grep "text" *.py — filter by extension
|
||||
find "*.md" — find files by glob
|
||||
find docs/ "*.md" — find within a directory
|
||||
find -t "*.md", tree -t — same sort flags as ls
|
||||
wc <file> — line/word/char counts
|
||||
stat <file> — file metadata
|
||||
|
||||
Pipes: grep "auth" | head -5
|
||||
Files: reference by path (docs/api/auth.md), filename, or file ID
|
||||
Regex: RE2 syntax; no lookarounds/backreferences. Shorthand character classes are ASCII-only.
|
||||
|
||||
:param command: A filesystem command string
|
||||
:return: Command output as text
|
||||
|
|
|
|||
|
|
@ -1,3 +1,4 @@
|
|||
import logging
|
||||
from typing import Any
|
||||
|
||||
from open_webui.config import DEFAULT_USER_PERMISSIONS
|
||||
|
|
@ -14,6 +15,8 @@ from open_webui.models.users import UserModel
|
|||
from open_webui.utils.json_codec import JSONCodec
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def fill_missing_permissions(permissions: dict[str, Any], default_permissions: dict[str, Any]) -> dict[str, Any]:
|
||||
"""
|
||||
|
|
@ -325,6 +328,13 @@ async def has_base_model_access(
|
|||
seen.add(base_model_id)
|
||||
base_model_info = await Models.get_model_by_id(base_model_id, db=db)
|
||||
if base_model_info is None:
|
||||
if user_role != 'admin':
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r base_model_id=%r reason=base_model_unregistered',
|
||||
user_id,
|
||||
model_info.id,
|
||||
base_model_id,
|
||||
)
|
||||
return user_role == 'admin'
|
||||
if not (
|
||||
user_id == base_model_info.user_id
|
||||
|
|
@ -337,6 +347,12 @@ async def has_base_model_access(
|
|||
db=db,
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r base_model_id=%r reason=base_model_read_denied',
|
||||
user_id,
|
||||
model_info.id,
|
||||
base_model_id,
|
||||
)
|
||||
return False
|
||||
base_model_id = getattr(base_model_info, 'base_model_id', None)
|
||||
return True
|
||||
|
|
@ -381,6 +397,11 @@ async def check_model_access(
|
|||
user_group_ids=user_group_ids,
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=model_read_denied',
|
||||
user.id,
|
||||
model_info.id,
|
||||
)
|
||||
raise HTTPException(status_code=403, detail='Model not found')
|
||||
|
||||
# Enforce access on chained base models
|
||||
|
|
@ -388,4 +409,5 @@ async def check_model_access(
|
|||
raise HTTPException(status_code=403, detail='Model not found')
|
||||
else:
|
||||
if user.role != 'admin':
|
||||
log.warning('Model access denied: user_id=%r reason=model_unregistered', user.id)
|
||||
raise HTTPException(status_code=403, detail='Model not found')
|
||||
|
|
|
|||
|
|
@ -63,28 +63,6 @@ async def has_access_to_file(
|
|||
) and (access_type == 'read' or knowledge_base.user_id == file.user_id):
|
||||
return True
|
||||
|
||||
knowledge_base_id = file.meta.get('collection_name') if file.meta else None
|
||||
if knowledge_base_id:
|
||||
# Fetch the one referenced knowledge base instead of listing every
|
||||
# knowledge base the user can access just to scan for this id.
|
||||
knowledge_base = await Knowledges.get_knowledge_by_id(knowledge_base_id, db=db)
|
||||
if (
|
||||
knowledge_base
|
||||
and (access_type == 'read' or knowledge_base.user_id == file.user_id)
|
||||
and (
|
||||
knowledge_base.user_id == user.id
|
||||
or await AccessGrants.has_access(
|
||||
user_id=user.id,
|
||||
resource_type='knowledge',
|
||||
resource_id=knowledge_base.id,
|
||||
permission=access_type,
|
||||
user_group_ids=user_group_ids,
|
||||
db=db,
|
||||
)
|
||||
)
|
||||
):
|
||||
return True
|
||||
|
||||
# Check if the file is associated with any channels the user has access to
|
||||
channels = await Channels.get_channels_by_file_id_and_user_id(file_id, user.id, db=db)
|
||||
if access_type == 'read' and channels:
|
||||
|
|
@ -106,7 +84,7 @@ async def has_access_to_file(
|
|||
|
||||
# Check if the file is directly attached to a shared workspace model (per the ownership
|
||||
# note above, model write is conferred only for files the model owner owns).
|
||||
model_owners = await Models.get_model_owners_attaching_file(file.id, db=db)
|
||||
model_owners = await Models.get_model_owner_ids_by_file_id(file.id, db=db, include_background=access_type == 'read')
|
||||
if access_type != 'read':
|
||||
model_owners = {model_id: owner_id for model_id, owner_id in model_owners.items() if owner_id == file.user_id}
|
||||
if user.id in model_owners.values():
|
||||
|
|
|
|||
|
|
@ -202,7 +202,9 @@ def convert_anthropic_to_openai_payload(
|
|||
)
|
||||
)
|
||||
elif block_type in ('thinking', 'redacted_thinking'):
|
||||
openai_content.append(_copy_cache_control(block, dict(block)))
|
||||
# Unsigned thinking cannot be replayed upstream
|
||||
if block_type == 'redacted_thinking' or block.get('signature'):
|
||||
openai_content.append(_copy_cache_control(block, dict(block)))
|
||||
elif block_type == 'image':
|
||||
source = block.get('source', {})
|
||||
if source.get('type') == 'base64':
|
||||
|
|
@ -369,7 +371,7 @@ def convert_anthropic_to_openai_payload(
|
|||
msg_dict['content'] = ''
|
||||
msg_dict['tool_calls'] = tool_calls
|
||||
messages.append(msg_dict)
|
||||
elif openai_content:
|
||||
elif openai_content or role == 'assistant':
|
||||
messages.append({'role': role, 'content': _finalize_openai_content(openai_content)})
|
||||
else:
|
||||
messages.append({'role': role, 'content': str(content) if content else ''})
|
||||
|
|
|
|||
|
|
@ -8,6 +8,8 @@ import logging
|
|||
import os
|
||||
import uuid
|
||||
from datetime import datetime, timedelta
|
||||
from threading import Lock
|
||||
from time import monotonic
|
||||
from typing import Optional, Union
|
||||
|
||||
import bcrypt
|
||||
|
|
@ -42,6 +44,7 @@ from open_webui.utils.access_control import has_permission
|
|||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.misc import parse_duration
|
||||
from pytz import UTC
|
||||
from redis.exceptions import RedisError
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
|
@ -248,14 +251,41 @@ def decode_token(token: str) -> dict | None:
|
|||
return None
|
||||
|
||||
|
||||
class RateLimitFilter(logging.Filter):
|
||||
"""Limit a logger to one record per interval per process."""
|
||||
|
||||
def __init__(self, interval=60):
|
||||
super().__init__()
|
||||
self.interval = interval
|
||||
self.next_allowed = float('-inf')
|
||||
self.lock = Lock()
|
||||
|
||||
def filter(self, record):
|
||||
with self.lock:
|
||||
now = monotonic()
|
||||
if now < self.next_allowed:
|
||||
return False
|
||||
self.next_allowed = now + self.interval
|
||||
return True
|
||||
|
||||
|
||||
revocation_log = logging.getLogger(f'{__name__}.revocation')
|
||||
revocation_log.addFilter(RateLimitFilter())
|
||||
|
||||
|
||||
async def is_valid_token(decoded, redis=None) -> bool:
|
||||
"""
|
||||
Check whether a JWT has been revoked. Two mechanisms:
|
||||
1. Per-token (jti) — used by user-initiated sign-out (known jti).
|
||||
2. Per-user (revoked_at) — used by password changes and OIDC back-channel
|
||||
logout when individual jti values are unknown; rejects tokens with iat <= revoked_at.
|
||||
|
||||
Fail open on Redis errors to preserve availability; revoked tokens may be accepted.
|
||||
"""
|
||||
if redis:
|
||||
if not redis:
|
||||
return True
|
||||
|
||||
try:
|
||||
# Per-token revocation
|
||||
jti = decoded.get('jti')
|
||||
if jti:
|
||||
|
|
@ -276,6 +306,8 @@ async def is_valid_token(decoded, redis=None) -> bool:
|
|||
return False
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
except RedisError as e:
|
||||
revocation_log.warning('Revocation check failed; accepting token: %s', e)
|
||||
|
||||
return True
|
||||
|
||||
|
|
@ -296,6 +328,10 @@ async def invalidate_token(request, token):
|
|||
ttl = exp - int(datetime.now(UTC).timestamp()) # Calculate time-to-live for the token
|
||||
|
||||
if ttl > 0:
|
||||
# Revoked tokens must not be able to disconnect newer sessions.
|
||||
if not await is_valid_token(decoded, request.app.state.redis):
|
||||
return
|
||||
|
||||
# Store the revoked token in Redis with an expiration time
|
||||
await request.app.state.redis.set(
|
||||
f'{REDIS_KEY_PREFIX}:auth:token:{jti}:revoked',
|
||||
|
|
@ -303,6 +339,12 @@ async def invalidate_token(request, token):
|
|||
ex=ttl,
|
||||
)
|
||||
|
||||
user_id = decoded.get('id')
|
||||
if user_id:
|
||||
from open_webui.socket.main import disconnect_user_sessions
|
||||
|
||||
await disconnect_user_sessions(user_id)
|
||||
|
||||
|
||||
async def revoke_user_tokens(request, user_id: str):
|
||||
"""Reject every token already issued to a user. Requires Redis."""
|
||||
|
|
@ -324,6 +366,10 @@ async def revoke_user_tokens(request, user_id: str):
|
|||
ex=int(expires_delta.total_seconds()) if expires_delta else None,
|
||||
)
|
||||
|
||||
from open_webui.socket.main import disconnect_user_sessions
|
||||
|
||||
await disconnect_user_sessions(user_id)
|
||||
|
||||
|
||||
def extract_token_from_auth_header(auth_header: str):
|
||||
return auth_header[len('Bearer ') :]
|
||||
|
|
@ -386,6 +432,7 @@ async def get_current_user(
|
|||
|
||||
# Scope-backed, so outer middleware (audit) can reuse the resolved user
|
||||
request.state.user = user
|
||||
request.state.auth_type = 'api_key'
|
||||
return user
|
||||
|
||||
# auth by jwt token
|
||||
|
|
@ -437,6 +484,7 @@ async def get_current_user(
|
|||
|
||||
# Scope-backed, so outer middleware (audit) can reuse the resolved user
|
||||
request.state.user = user
|
||||
request.state.auth_type = 'jwt'
|
||||
return user
|
||||
else:
|
||||
raise HTTPException(
|
||||
|
|
|
|||
|
|
@ -20,12 +20,10 @@ import logging
|
|||
import os
|
||||
import random
|
||||
import time
|
||||
from datetime import datetime, timedelta
|
||||
from datetime import timedelta
|
||||
from typing import Optional
|
||||
from uuid import uuid4
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
from dateutil.rrule import HOURLY, MINUTELY, SECONDLY, rruleset, rrulestr
|
||||
from fastapi import Request
|
||||
from fastapi.security import HTTPAuthorizationCredentials
|
||||
from open_webui.constants import ERROR_MESSAGES
|
||||
|
|
@ -39,6 +37,13 @@ from open_webui.models.messages import MessageForm
|
|||
from open_webui.models.users import Users
|
||||
from open_webui.utils.auth import create_token
|
||||
from open_webui.utils.misc import parse_duration
|
||||
from open_webui.utils.recurrence import (
|
||||
_resolve_tz,
|
||||
next_n_runs_ns,
|
||||
next_run_ns,
|
||||
rrule_interval_seconds,
|
||||
validate_rrule,
|
||||
)
|
||||
from open_webui.utils.task import prompt_template
|
||||
from open_webui.utils.terminals import get_terminal_server_url
|
||||
from starlette.datastructures import Headers
|
||||
|
|
@ -50,153 +55,6 @@ TIMER_POLL_INTERVAL = int(os.getenv('TIMER_POLL_INTERVAL', '1'))
|
|||
CALENDAR_ALERT_LOOKAHEAD_MINUTES = int(os.getenv('CALENDAR_ALERT_LOOKAHEAD_MINUTES', '10'))
|
||||
|
||||
|
||||
####################
|
||||
# RRULE Helpers
|
||||
####################
|
||||
|
||||
|
||||
def _resolve_tz(tz: str = None) -> Optional[ZoneInfo]:
|
||||
"""Safely resolve a timezone string to ZoneInfo.
|
||||
|
||||
Returns None (→ server-local fallback) when *tz* is empty, None,
|
||||
or an unrecognised IANA key. Logs a warning on bad keys so
|
||||
misconfiguration is visible in the server logs.
|
||||
"""
|
||||
if not tz:
|
||||
return None
|
||||
try:
|
||||
return ZoneInfo(tz)
|
||||
except (KeyError, Exception):
|
||||
log.warning('Unknown timezone %r — falling back to server time', tz)
|
||||
return None
|
||||
|
||||
|
||||
def _parse_rule(s: str, now: Optional[datetime] = None):
|
||||
"""Parse RRULE with clock-aligned DTSTART for sub-daily frequencies.
|
||||
|
||||
SECONDLY/MINUTELY/HOURLY rules use a fixed epoch DTSTART (2000-01-01 00:00)
|
||||
so intervals snap to clock boundaries (e.g. every 5min = :00, :05, :10).
|
||||
"""
|
||||
upper = s.upper()
|
||||
if 'EXRULE' in upper:
|
||||
raise ValueError('EXRULE is not supported in recurrence rules')
|
||||
|
||||
parsed = rrulestr(s, ignoretz=True)
|
||||
rules = parsed._rrule if isinstance(parsed, rruleset) else [parsed]
|
||||
if len(rules) > 1:
|
||||
raise ValueError('only one RRULE is supported per recurrence rule')
|
||||
|
||||
rule = rules[0]
|
||||
start = rule._dtstart.replace(tzinfo=None)
|
||||
anchor = now or datetime.now()
|
||||
lines = s.splitlines()
|
||||
stripped = '\n'.join(line for line in lines if not line.upper().startswith('DTSTART')) or s
|
||||
has_dtstart = any(line.upper().startswith('DTSTART') for line in lines)
|
||||
step = {
|
||||
SECONDLY: timedelta(seconds=rule._interval),
|
||||
MINUTELY: timedelta(minutes=rule._interval),
|
||||
HOURLY: timedelta(hours=rule._interval),
|
||||
}.get(rule._freq)
|
||||
|
||||
if step is None:
|
||||
if not rule._dtstart.tzinfo:
|
||||
return parsed
|
||||
return rrulestr(stripped, dtstart=start, ignoretz=True)
|
||||
|
||||
if rule._interval < 1:
|
||||
raise ValueError('RRULE INTERVAL must be a positive integer')
|
||||
dtstart = None
|
||||
if has_dtstart:
|
||||
emitted = ((anchor - start) // step) if anchor > start else 0
|
||||
emitted *= len(rule._byminute or (0,)) * len(rule._bysecond or (0,))
|
||||
if emitted <= 100_000:
|
||||
if rule._dtstart.tzinfo:
|
||||
dtstart = start
|
||||
else:
|
||||
return parsed
|
||||
if not has_dtstart or dtstart is None:
|
||||
epoch = datetime(2000, 1, 1)
|
||||
dtstart = epoch + ((anchor - epoch) // step) * step
|
||||
|
||||
return rrulestr(stripped, dtstart=dtstart, ignoretz=True)
|
||||
|
||||
|
||||
def validate_rrule(s: str, tz: str = None) -> None:
|
||||
"""Raise ValueError if the RRULE is malformed or exhausted.
|
||||
|
||||
When *tz* is provided the "now" reference uses the user's local
|
||||
clock so that near-future schedules are not incorrectly rejected
|
||||
on servers whose system clock is ahead (e.g. UTC vs US timezones).
|
||||
"""
|
||||
upper = s.upper()
|
||||
if 'COUNT=' in upper and 'DTSTART' not in upper:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_COUNT_REQUIRES_DTSTART)
|
||||
zi = _resolve_tz(tz)
|
||||
now = datetime.now(zi).replace(tzinfo=None) if zi else datetime.now()
|
||||
try:
|
||||
rule = _parse_rule(s, now)
|
||||
except Exception as e:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_INVALID_RRULE(e))
|
||||
if rule.after(now) is None:
|
||||
raise ValueError(ERROR_MESSAGES.AUTOMATION_NO_FUTURE_RUNS)
|
||||
|
||||
|
||||
def next_run_ns(s: str, tz: str = None) -> Optional[int]:
|
||||
"""Next occurrence as epoch nanoseconds, respecting user timezone."""
|
||||
zi = _resolve_tz(tz)
|
||||
now = datetime.now(zi) if zi else datetime.now()
|
||||
now_naive = now.replace(tzinfo=None)
|
||||
dt = _parse_rule(s, now_naive).after(now_naive)
|
||||
if dt is None:
|
||||
return None
|
||||
if zi:
|
||||
dt = dt.replace(tzinfo=zi)
|
||||
return int(dt.timestamp() * 1_000_000_000)
|
||||
|
||||
|
||||
def next_n_runs_ns(s: str, n: int = 5, tz: str = None) -> list[int]:
|
||||
"""Compute next N occurrences for UI preview.
|
||||
|
||||
Uses the user's timezone for the starting "now" so that the
|
||||
preview matches the user's local clock (same as next_run_ns).
|
||||
"""
|
||||
zi = _resolve_tz(tz)
|
||||
result = []
|
||||
now = datetime.now(zi).replace(tzinfo=None) if zi else datetime.now()
|
||||
rule = _parse_rule(s, now)
|
||||
dt = now
|
||||
for _ in range(n):
|
||||
dt = rule.after(dt)
|
||||
if not dt:
|
||||
break
|
||||
if zi:
|
||||
dt_tz = dt.replace(tzinfo=zi)
|
||||
result.append(int(dt_tz.timestamp() * 1_000_000_000))
|
||||
else:
|
||||
result.append(int(dt.timestamp() * 1_000_000_000))
|
||||
return result
|
||||
|
||||
|
||||
def rrule_interval_seconds(s: str) -> Optional[int]:
|
||||
"""Approximate interval between recurrences in seconds.
|
||||
|
||||
Returns None for one-shot (COUNT=1) schedules or rules
|
||||
with fewer than two future occurrences.
|
||||
"""
|
||||
if 'COUNT=1' in s:
|
||||
return None
|
||||
s = '\n'.join(line for line in s.splitlines() if not line.upper().startswith('DTSTART')) or s
|
||||
now = datetime.now()
|
||||
rule = _parse_rule(s, now)
|
||||
first = rule.after(now)
|
||||
if first is None:
|
||||
return None
|
||||
second = rule.after(first)
|
||||
if second is None:
|
||||
return None
|
||||
return int((second - first).total_seconds())
|
||||
|
||||
|
||||
############################
|
||||
# Worker Loop
|
||||
############################
|
||||
|
|
@ -305,17 +163,20 @@ def _build_request(
|
|||
return request
|
||||
|
||||
|
||||
async def _resolve_model_defaults(app, model_id: str) -> tuple[list[str], dict, list[str], Optional[str]]:
|
||||
async def _resolve_model_defaults(app, model_id: str) -> dict:
|
||||
models = getattr(app.state, 'MODELS', {})
|
||||
model = models.get(model_id, {})
|
||||
meta = model.get('info', {}).get('meta', {})
|
||||
|
||||
tool_ids = list(meta.get('toolIds') or [])
|
||||
filter_ids = list(meta.get('defaultFilterIds') or [])
|
||||
terminal_id = meta.get('terminalId') or None
|
||||
defaults = {
|
||||
'tool_ids': list(meta.get('toolIds') or []),
|
||||
'filter_ids': list(meta.get('defaultFilterIds') or []),
|
||||
'terminal_id': meta.get('terminalId'),
|
||||
}
|
||||
defaults = {key: value for key, value in defaults.items() if value}
|
||||
default_feature_ids = meta.get('defaultFeatureIds', [])
|
||||
if not default_feature_ids:
|
||||
return tool_ids, {}, filter_ids, terminal_id
|
||||
return defaults
|
||||
|
||||
capabilities = meta.get('capabilities') or {}
|
||||
features = {}
|
||||
|
|
@ -333,7 +194,9 @@ async def _resolve_model_defaults(app, model_id: str) -> tuple[list[str], dict,
|
|||
if capabilities.get(feature_id) and feature_checks[feature_id]:
|
||||
features[feature_id] = True
|
||||
|
||||
return tool_ids, features, filter_ids, terminal_id
|
||||
if features:
|
||||
defaults['features'] = features
|
||||
return defaults
|
||||
|
||||
|
||||
async def _set_terminal_cwd(app, server_id: str, user, cwd: str, chat_id: str) -> None:
|
||||
|
|
@ -436,9 +299,8 @@ async def _execute_channel_automation(
|
|||
db,
|
||||
)
|
||||
|
||||
tool_ids, features, filter_ids, _ = await _resolve_model_defaults(app, model_id)
|
||||
|
||||
form_data = {
|
||||
**await _resolve_model_defaults(app, model_id),
|
||||
'model': model_id,
|
||||
'messages': [
|
||||
{
|
||||
|
|
@ -454,13 +316,6 @@ async def _execute_channel_automation(
|
|||
'automation_id': automation.id,
|
||||
'background_tasks': {},
|
||||
}
|
||||
if tool_ids:
|
||||
form_data['tool_ids'] = tool_ids
|
||||
if features:
|
||||
form_data['features'] = features
|
||||
if filter_ids:
|
||||
form_data['filter_ids'] = filter_ids
|
||||
|
||||
await app.state.CHAT_COMPLETION_HANDLER(request, form_data, user=user)
|
||||
|
||||
from open_webui.socket.main import sio
|
||||
|
|
@ -615,11 +470,9 @@ async def execute_automation(app, automation: AutomationModel) -> None:
|
|||
room=f'user:{automation.user_id}',
|
||||
)
|
||||
|
||||
# Resolve model defaults (frontend does this, backend doesn't)
|
||||
tool_ids, features, filter_ids, terminal_id = await _resolve_model_defaults(app, model_id)
|
||||
|
||||
# Build the same payload the frontend sends to /api/chat/completions
|
||||
form_data = {
|
||||
**await _resolve_model_defaults(app, model_id),
|
||||
'model': model_id,
|
||||
'messages': [{'role': 'user', 'content': prompt}],
|
||||
'stream': True,
|
||||
|
|
@ -636,15 +489,6 @@ async def execute_automation(app, automation: AutomationModel) -> None:
|
|||
'automation_id': automation.id,
|
||||
'background_tasks': {},
|
||||
}
|
||||
if tool_ids:
|
||||
form_data['tool_ids'] = tool_ids
|
||||
if features:
|
||||
form_data['features'] = features
|
||||
if filter_ids:
|
||||
form_data['filter_ids'] = filter_ids
|
||||
if terminal_id:
|
||||
form_data['terminal_id'] = terminal_id
|
||||
|
||||
# Call the full chat completion pipeline (same as POST /api/chat/completions).
|
||||
# The handler reference is stored on app.state to avoid circular imports.
|
||||
request = _build_request(app, token=token)
|
||||
|
|
|
|||
|
|
@ -44,7 +44,7 @@ def expand_recurring_event(
|
|||
|
||||
original_start_ns = event_dict['start_at']
|
||||
original_start = to_local_datetime(original_start_ns)
|
||||
rule_str = '\n'.join(line for line in rrule_str.splitlines() if not line.upper().startswith('DTSTART')) or rrule_str
|
||||
rule_str = '\n'.join(part for part in rrule_str.split() if not part.upper().startswith('DTSTART')) or rrule_str
|
||||
|
||||
try:
|
||||
# Anchor to the event's real start so day-of-week / day-of-month are correct
|
||||
|
|
|
|||
|
|
@ -23,9 +23,9 @@ from open_webui.routers.pipelines import (
|
|||
process_pipeline_outlet_filter,
|
||||
)
|
||||
from open_webui.socket.main import (
|
||||
EVENT_QUEUES,
|
||||
get_event_call,
|
||||
get_event_emitter,
|
||||
sio,
|
||||
)
|
||||
from open_webui.utils.filter import (
|
||||
get_filter_functions,
|
||||
|
|
@ -71,39 +71,36 @@ async def generate_direct_chat_completion(
|
|||
logging.info('WebSocket channel: %s', channel)
|
||||
|
||||
if form_data.get('stream'):
|
||||
q = asyncio.Queue()
|
||||
|
||||
async def message_listener(sid, data):
|
||||
"""
|
||||
Handle received socket messages and push them into the queue.
|
||||
"""
|
||||
await q.put(data)
|
||||
|
||||
# Register the listener
|
||||
sio.on(channel, message_listener)
|
||||
queue = asyncio.Queue()
|
||||
EVENT_QUEUES[channel] = queue
|
||||
|
||||
# Start processing chat completion in background
|
||||
res = await event_caller(
|
||||
{
|
||||
'type': 'request:chat:completion',
|
||||
'data': {
|
||||
'form_data': form_data,
|
||||
'model': models[form_data['model']],
|
||||
'channel': channel,
|
||||
'session_id': session_id,
|
||||
},
|
||||
}
|
||||
)
|
||||
try:
|
||||
res = await event_caller(
|
||||
{
|
||||
'type': 'request:chat:completion',
|
||||
'data': {
|
||||
'form_data': form_data,
|
||||
'model': models[form_data['model']],
|
||||
'channel': channel,
|
||||
'session_id': session_id,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
log.info('res: %s', res)
|
||||
log.info('res: %s', res)
|
||||
|
||||
if res.get('status', False):
|
||||
status = res.get('status', False)
|
||||
except BaseException:
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
raise
|
||||
|
||||
if status:
|
||||
# Define a generator to stream responses
|
||||
async def event_generator():
|
||||
nonlocal q
|
||||
try:
|
||||
while True:
|
||||
data = await q.get() # Wait for new messages
|
||||
data = await queue.get() # Wait for new messages
|
||||
if isinstance(data, dict):
|
||||
if 'done' in data and data['done']:
|
||||
break # Stop streaming when 'done' is received
|
||||
|
|
@ -117,17 +114,17 @@ async def generate_direct_chat_completion(
|
|||
except Exception as e:
|
||||
log.debug('Error in event generator: %s', e)
|
||||
pass
|
||||
finally:
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
|
||||
# Define a background task to run the event generator
|
||||
async def background():
|
||||
try:
|
||||
del sio.handlers['/'][channel]
|
||||
except Exception as e:
|
||||
pass
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
|
||||
# Return the streaming response
|
||||
return StreamingResponse(event_generator(), media_type='text/event-stream', background=background)
|
||||
else:
|
||||
EVENT_QUEUES.pop(channel, None)
|
||||
raise Exception(str(res))
|
||||
else:
|
||||
res = await event_caller(
|
||||
|
|
@ -260,24 +257,25 @@ async def generate_chat_completion(
|
|||
bypass_filter=True,
|
||||
bypass_system_prompt=bypass_system_prompt,
|
||||
)
|
||||
# Upstream errors come back as a response object.
|
||||
if not isinstance(response, StreamingResponse):
|
||||
return response
|
||||
return StreamingResponse(
|
||||
stream_wrapper(response.body_iterator),
|
||||
media_type='text/event-stream',
|
||||
background=response.background,
|
||||
)
|
||||
else:
|
||||
return {
|
||||
**(
|
||||
await generate_chat_completion(
|
||||
request,
|
||||
form_data,
|
||||
user,
|
||||
bypass_filter=True,
|
||||
bypass_system_prompt=bypass_system_prompt,
|
||||
)
|
||||
),
|
||||
'selected_model_id': selected_model_id,
|
||||
}
|
||||
response = await generate_chat_completion(
|
||||
request,
|
||||
form_data,
|
||||
user,
|
||||
bypass_filter=True,
|
||||
bypass_system_prompt=bypass_system_prompt,
|
||||
)
|
||||
if not isinstance(response, dict):
|
||||
return response
|
||||
return {**response, 'selected_model_id': selected_model_id}
|
||||
|
||||
if model.get('pipe'):
|
||||
# Below does not require bypass_filter because this is the only route the uses this function and it is already bypassing the filter
|
||||
|
|
|
|||
|
|
@ -32,6 +32,7 @@ from open_webui.storage.provider import Storage
|
|||
|
||||
BASE64_IMAGE_URL_PREFIX = re.compile(r'data:image/\w+;base64,', re.IGNORECASE)
|
||||
MARKDOWN_IMAGE_URL_PATTERN = re.compile(r'!\[(.*?)\]\((.+?)\)', re.IGNORECASE)
|
||||
FILE_CONTENT_URL_PATTERN = re.compile(r'^/api/v1/files/([^/?#]+)/content(?:[?#]|$)')
|
||||
|
||||
# Extension-based MIME fallback, only used when ENABLE_IMAGE_CONTENT_TYPE_EXTENSION_FALLBACK is True.
|
||||
_IMAGE_MIME_FALLBACK = {
|
||||
|
|
@ -74,9 +75,16 @@ async def get_image_base64_from_url(url: str, user=None) -> Optional[str]:
|
|||
# rebinding DNS answer that passed validate_url cannot reach an internal address.
|
||||
async with get_ssrf_safe_session() as session:
|
||||
async with session.get(
|
||||
url, ssl=AIOHTTP_CLIENT_SESSION_SSL, allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS
|
||||
url,
|
||||
ssl=AIOHTTP_CLIENT_SESSION_SSL,
|
||||
allow_redirects=AIOHTTP_CLIENT_ALLOW_REDIRECTS,
|
||||
headers={'Accept-Encoding': 'identity'},
|
||||
) as response:
|
||||
response.raise_for_status()
|
||||
# Accept-Encoding is only a request; the sender can still compress and pick our decompressed size.
|
||||
encodings = response.headers.getall('Content-Encoding', ())
|
||||
if any(encoding.lower() not in ('', 'identity') for encoding in encodings):
|
||||
return None
|
||||
image_data = bytearray()
|
||||
total = 0
|
||||
async for chunk in response.content.iter_chunked(64 * 1024):
|
||||
|
|
@ -90,7 +98,8 @@ async def get_image_base64_from_url(url: str, user=None) -> Optional[str]:
|
|||
else:
|
||||
# Non-URL string — treat as file_id. Delegate to the canonical
|
||||
# file-ID resolver which enforces ownership/access checks.
|
||||
return await get_image_base64_from_file_id(url, user=user)
|
||||
file_id_match = FILE_CONTENT_URL_PATTERN.match(url)
|
||||
return await get_image_base64_from_file_id(file_id_match.group(1) if file_id_match else url, user=user)
|
||||
|
||||
except Exception:
|
||||
return None
|
||||
|
|
|
|||
|
|
@ -1,10 +1,12 @@
|
|||
import logging
|
||||
import time
|
||||
from string import punctuation
|
||||
from typing import Any, Optional
|
||||
from urllib.parse import quote
|
||||
|
||||
import jwt
|
||||
from open_webui.env import (
|
||||
FORWARD_USER_INFO_HEADER_AUTH_TYPE,
|
||||
FORWARD_USER_INFO_HEADER_JWT,
|
||||
FORWARD_USER_INFO_HEADER_JWT_EXPIRES_SECONDS,
|
||||
FORWARD_USER_INFO_HEADER_JWT_SECRET,
|
||||
|
|
@ -47,15 +49,20 @@ def _mint_forward_user_jwt(user: Any) -> str:
|
|||
return jwt.encode(payload, FORWARD_USER_INFO_HEADER_JWT_SECRET, algorithm='HS256')
|
||||
|
||||
|
||||
def include_user_info_headers(headers: dict, user: Optional[Any] = None) -> dict:
|
||||
def include_user_info_headers(headers: dict, user: Optional[Any] = None, *, request=None) -> dict:
|
||||
"""
|
||||
Forward user identity to external backends: signed JWT in
|
||||
FORWARD_USER_INFO_HEADER_JWT if FORWARD_USER_INFO_HEADER_JWT_SECRET is set;
|
||||
otherwise the legacy X-OpenWebUI-User-* headers.
|
||||
Include the verified incoming auth type when a request provides it.
|
||||
"""
|
||||
if user is None:
|
||||
return headers
|
||||
|
||||
auth_type = getattr(getattr(request, 'state', None), 'auth_type', None)
|
||||
if auth_type in ('api_key', 'jwt'):
|
||||
headers = {**headers, FORWARD_USER_INFO_HEADER_AUTH_TYPE: auth_type}
|
||||
|
||||
if FORWARD_USER_INFO_HEADER_JWT_SECRET:
|
||||
try:
|
||||
token = _mint_forward_user_jwt(user)
|
||||
|
|
@ -141,6 +148,7 @@ def parse_custom_headers(
|
|||
'{{USER_GROUPS}}': ','.join(group.name.strip() for group in user_groups) if user_groups else '',
|
||||
'{{USER_GROUP_IDS}}': ','.join(group.id for group in user_groups) if user_groups else '',
|
||||
'{{USER_AGENT}}': user_agent,
|
||||
'{{AUTH_TYPE}}': getattr(getattr(request, 'state', None), 'auth_type', None) or '',
|
||||
}
|
||||
|
||||
parsed_headers = {}
|
||||
|
|
@ -149,6 +157,7 @@ def parse_custom_headers(
|
|||
value = str(value)
|
||||
for token, val in template_vars.items():
|
||||
value = value.replace(token, val)
|
||||
parsed_headers[key] = value
|
||||
# Encode Unicode and controls after substitution; preserve ASCII header syntax and existing escapes.
|
||||
parsed_headers[key] = quote(value, safe=punctuation + ' \t')
|
||||
|
||||
return parsed_headers
|
||||
|
|
|
|||
|
|
@ -89,8 +89,14 @@ class MCPClient:
|
|||
if not self.session:
|
||||
raise RuntimeError('MCP client is not connected.')
|
||||
|
||||
result = await self.session.list_tools()
|
||||
tools = result.tools
|
||||
tools = []
|
||||
cursor = None
|
||||
while True:
|
||||
result = await self.session.list_tools(cursor=cursor)
|
||||
tools.extend(result.tools)
|
||||
cursor = result.nextCursor
|
||||
if cursor is None:
|
||||
break
|
||||
|
||||
tool_specs = []
|
||||
for tool in tools:
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ from typing import Any
|
|||
from fastapi import HTTPException
|
||||
from open_webui.models.config import Config
|
||||
from open_webui.models.memories import Memories
|
||||
from open_webui.utils.access_control import has_permission
|
||||
from open_webui.utils.json_codec import JSONCodec
|
||||
from open_webui.utils.misc import add_or_update_system_message, get_content_from_message
|
||||
|
||||
|
|
@ -428,10 +429,12 @@ async def review_memory_after_turn(
|
|||
return
|
||||
|
||||
config = await Config.get_many(
|
||||
'memories.enable',
|
||||
'memories.background_review.enable',
|
||||
'memories.review_interval_turns',
|
||||
'user.permissions',
|
||||
)
|
||||
if not config.get('memories.background_review.enable'):
|
||||
if not config.get('memories.enable') or not config.get('memories.background_review.enable'):
|
||||
return
|
||||
|
||||
try:
|
||||
|
|
@ -443,6 +446,10 @@ async def review_memory_after_turn(
|
|||
if user_turns == 0 or user_turns % interval != 0:
|
||||
return
|
||||
|
||||
# features is client-supplied; re-check the permission the memory routes enforce.
|
||||
if user.role != 'admin' and not await has_permission(user.id, 'features.memories', config.get('user.permissions')):
|
||||
return
|
||||
|
||||
task = asyncio.create_task(
|
||||
_review_memory(
|
||||
request=request,
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -1223,6 +1223,12 @@ def strict_match_mime_type(supported: list[str] | str, header: str) -> str | Non
|
|||
return None
|
||||
|
||||
|
||||
def is_raster_image_content_type(content_type: str | None) -> bool:
|
||||
"""Return True if the content type is an image that decodes as a bitmap; SVG is XML."""
|
||||
base_content_type = (content_type or '').split(';')[0].strip().lower()
|
||||
return base_content_type.startswith('image/') and base_content_type != 'image/svg+xml'
|
||||
|
||||
|
||||
def extract_urls(text: str) -> list[str]:
|
||||
# Regex pattern to match URLs
|
||||
url_pattern = re.compile(r'(https?://[^\s]+)', re.IGNORECASE) # Matches http and https URLs
|
||||
|
|
|
|||
|
|
@ -187,6 +187,8 @@ async def get_all_models(request, refresh: bool = False, user: UserModel = None)
|
|||
schema = get_chat_variables_schema(custom_model.params.model_dump().get('system'))
|
||||
if schema:
|
||||
model['info'].setdefault('meta', {})['chat_variables_schema'] = schema
|
||||
elif isinstance(model['info'].get('meta'), dict):
|
||||
model['info']['meta'].pop('chat_variables_schema', None)
|
||||
|
||||
action_ids = []
|
||||
filter_ids = []
|
||||
|
|
@ -239,6 +241,8 @@ async def get_all_models(request, refresh: bool = False, user: UserModel = None)
|
|||
schema = get_chat_variables_schema(custom_model.params.model_dump().get('system'))
|
||||
if schema:
|
||||
info.setdefault('meta', {})['chat_variables_schema'] = schema
|
||||
elif isinstance(info.get('meta'), dict):
|
||||
info['meta'].pop('chat_variables_schema', None)
|
||||
if 'params' in info:
|
||||
# Remove params to avoid exposing sensitive info
|
||||
del info['params']
|
||||
|
|
@ -438,7 +442,14 @@ async def get_all_models(request, refresh: bool = False, user: UserModel = None)
|
|||
|
||||
log.debug('get_all_models() returned %s models', len(models))
|
||||
|
||||
models_dict = {model['id']: model for model in models}
|
||||
models_dict = {}
|
||||
for model in models:
|
||||
model = model.copy()
|
||||
if model.get('ollama'):
|
||||
# Keep the moving expiry in the API response, outside the registry signature.
|
||||
model['ollama'] = model['ollama'].copy()
|
||||
model['ollama'].pop('expires_at', None)
|
||||
models_dict[model['id']] = model
|
||||
if isinstance(request.app.state.MODELS, RedisDict):
|
||||
try:
|
||||
request.app.state.MODELS.set(models_dict)
|
||||
|
|
@ -461,12 +472,22 @@ async def check_model_access(user, model, model_info=None, db=None):
|
|||
access_grants=access_grants,
|
||||
db=db,
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=arena_read_denied',
|
||||
user.id,
|
||||
model.get('id'),
|
||||
)
|
||||
raise Exception('Model not found')
|
||||
else:
|
||||
# Callers that already fetched the row (chat completion entry) pass it in
|
||||
if model_info is None or model_info.id != model.get('id'):
|
||||
model_info = await Models.get_model_by_id(model.get('id'), db=db)
|
||||
if not model_info:
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=model_unregistered',
|
||||
user.id,
|
||||
model.get('id'),
|
||||
)
|
||||
raise Exception('Model not found')
|
||||
|
||||
# One group-membership fetch shared by the direct check and every
|
||||
|
|
@ -486,6 +507,11 @@ async def check_model_access(user, model, model_info=None, db=None):
|
|||
db=db,
|
||||
)
|
||||
):
|
||||
log.warning(
|
||||
'Model access denied: user_id=%r model_id=%r reason=model_read_denied',
|
||||
user.id,
|
||||
model_info.id,
|
||||
)
|
||||
raise Exception('Model not found')
|
||||
|
||||
# Enforce access on chained base models
|
||||
|
|
|
|||
|
|
@ -46,8 +46,8 @@ def _normalize_target(target: dict[str, Any], existing: dict[str, Any] | None =
|
|||
|
||||
target_id = str(target.get('id') or existing.get('id') or '').strip()
|
||||
if not target_id:
|
||||
hostname = urlparse(url).hostname or 'webhook'
|
||||
target_id = re.sub(r'[^a-zA-Z0-9_-]+', '-', hostname).strip('-').lower() or 'target'
|
||||
target_id = urlparse(url).hostname or 'webhook'
|
||||
target_id = re.sub(r'[^a-zA-Z0-9_-]+', '-', target_id).strip('-').lower() or 'target'
|
||||
|
||||
events = target['events'] if 'events' in target else existing.get('events', [])
|
||||
if events is None:
|
||||
|
|
|
|||
|
|
@ -91,7 +91,7 @@ from open_webui.utils.auth import (
|
|||
)
|
||||
from open_webui.utils.groups import apply_default_group_assignment
|
||||
from open_webui.utils.misc import parse_duration
|
||||
from open_webui.utils.validate import validate_profile_image_url
|
||||
from open_webui.utils.validate import validate_image_url
|
||||
from starlette.responses import RedirectResponse
|
||||
|
||||
# Some IdPs put private params in ID token JOSE headers (CAS: client_id, CyberArk: app_id).
|
||||
|
|
@ -188,6 +188,16 @@ def _default_value(value):
|
|||
return getattr(value, 'value', value)
|
||||
|
||||
|
||||
def _get_roles_claim(claims: dict, claim: str) -> list | str | int | None:
|
||||
"""Read nested or flat claims, preserving explicit empty values and zero."""
|
||||
value = claims
|
||||
for key in claim.split('.'):
|
||||
value = value.get(key) if isinstance(value, dict) else None
|
||||
if not isinstance(value, (list, str, int)):
|
||||
value = claims.get(claim)
|
||||
return value if isinstance(value, (list, str, int)) else None
|
||||
|
||||
|
||||
async def get_oauth_runtime_config() -> SimpleNamespace:
|
||||
keys = [key for key, _default in OAUTH_RUNTIME_CONFIG.values()]
|
||||
stored = await Config.get_many(*keys)
|
||||
|
|
@ -202,7 +212,7 @@ NON_EXPIRING_TOKEN_EXPIRES_AT = 253402300799 # 9999-12-31 23:59:59 UTC
|
|||
|
||||
|
||||
def _normalize_token_expiry(token: dict) -> dict:
|
||||
"""Ensure a token dict always has a numeric ``expires_at``.
|
||||
"""Ensure a token dict always has a numeric access-token ``expires_at``.
|
||||
|
||||
Resolution order:
|
||||
1. If *expires_at* is already present and non-None, trust it.
|
||||
|
|
@ -233,16 +243,6 @@ def _normalize_token_expiry(token: dict) -> dict:
|
|||
)
|
||||
expires_at = NON_EXPIRING_TOKEN_EXPIRES_AT
|
||||
|
||||
id_token = token.get('id_token')
|
||||
if id_token:
|
||||
# Cap at the id_token expiry so pipes and tools never receive an expired JWT
|
||||
try:
|
||||
exp = jwt.decode(id_token, options={'verify_signature': False}).get('exp')
|
||||
if exp is not None:
|
||||
expires_at = min(expires_at, int(exp))
|
||||
except Exception as e:
|
||||
log.debug('Could not read exp from id_token: %s', e)
|
||||
|
||||
token['expires_at'] = expires_at
|
||||
return token
|
||||
|
||||
|
|
@ -351,6 +351,19 @@ def is_in_blocked_groups(group_name: str, groups: list) -> bool:
|
|||
return False
|
||||
|
||||
|
||||
def _parse_blocked_groups(value) -> list[str]:
|
||||
"""Accept JSON arrays, persisted lists, and comma-separated admin input."""
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
parsed = JSONCodec.loads(value)
|
||||
except JSONCodec.JSONDecodeError:
|
||||
parsed = None
|
||||
value = parsed if isinstance(parsed, list) else [group.strip() for group in value.split(',')]
|
||||
if not isinstance(value, list):
|
||||
return []
|
||||
return [group for group in value if isinstance(group, str) and group]
|
||||
|
||||
|
||||
def get_parsed_and_base_url(server_url) -> tuple[urllib.parse.ParseResult, str]:
|
||||
parsed = urllib.parse.urlparse(server_url)
|
||||
base_url = f'{parsed.scheme}://{parsed.netloc}'
|
||||
|
|
@ -1263,7 +1276,7 @@ class OAuthClientManager:
|
|||
if token and not token.get('access_token'):
|
||||
error_desc = token.get('error_description', token.get('error', 'Unknown error'))
|
||||
error_message = f'Token exchange failed: {error_desc}'
|
||||
log.error(f'Invalid token response for client_id {client_id}: {token}')
|
||||
log.error('Invalid token response for client_id %s: %s', client_id, error_desc)
|
||||
token = None
|
||||
|
||||
if token:
|
||||
|
|
@ -1372,10 +1385,21 @@ class OAuthManager:
|
|||
)
|
||||
return None
|
||||
|
||||
# SSO integrations may consume the ID token as well as the access token.
|
||||
expires_at = session.expires_at
|
||||
id_token = session.token.get('id_token')
|
||||
if id_token and expires_at is not None:
|
||||
try:
|
||||
exp = jwt.decode(id_token, options={'verify_signature': False}).get('exp')
|
||||
if exp is not None:
|
||||
expires_at = min(expires_at, int(exp))
|
||||
except Exception as e:
|
||||
log.debug('Could not read exp from id_token: %s', e)
|
||||
|
||||
if (
|
||||
force_refresh
|
||||
or session.expires_at is None
|
||||
or datetime.now() + timedelta(minutes=5) >= datetime.fromtimestamp(session.expires_at)
|
||||
or expires_at is None
|
||||
or datetime.now() + timedelta(minutes=5) >= datetime.fromtimestamp(expires_at)
|
||||
):
|
||||
log.debug('Token refresh needed for user %s, provider %s', user_id, session.provider)
|
||||
refreshed_token = await self._refresh_token(session)
|
||||
|
|
@ -1504,7 +1528,7 @@ class OAuthManager:
|
|||
log.error(f'Exception during token refresh for provider {provider}: {e}')
|
||||
return None
|
||||
|
||||
async def get_user_role(self, user, user_data):
|
||||
async def get_user_role(self, user, user_data, *, access_token: str | None = None):
|
||||
auth_config = await get_oauth_runtime_config()
|
||||
user_count = await Users.get_num_users()
|
||||
if user and user_count == 1:
|
||||
|
|
@ -1528,18 +1552,15 @@ class OAuthManager:
|
|||
# Keep existing users at their current role unless the provider sent roles.
|
||||
role = user.role if user else auth_config.DEFAULT_USER_ROLE
|
||||
|
||||
# Next block extracts the roles from the user data, accepting nested claims of any depth
|
||||
if oauth_claim and oauth_allowed_roles and oauth_admin_roles:
|
||||
claim_data = user_data
|
||||
nested_claims = oauth_claim.split('.')
|
||||
for nested_claim in nested_claims:
|
||||
claim_data = claim_data.get(nested_claim, {})
|
||||
|
||||
# Try flat claim structure as alternative
|
||||
if not claim_data:
|
||||
claim_data = user_data.get(oauth_claim, {})
|
||||
|
||||
oauth_roles = []
|
||||
if oauth_claim:
|
||||
claim_data = _get_roles_claim(user_data, oauth_claim)
|
||||
if claim_data is None and access_token is not None:
|
||||
# The exchange endpoint has already validated this token with the provider's userinfo endpoint.
|
||||
try:
|
||||
token_claims = jwt.decode(access_token, options={'verify_signature': False})
|
||||
claim_data = _get_roles_claim(token_claims, oauth_claim)
|
||||
except jwt.PyJWTError as e:
|
||||
log.debug('Token exchange: cannot decode token claims: %s', e)
|
||||
|
||||
if isinstance(claim_data, list):
|
||||
oauth_roles = claim_data
|
||||
|
|
@ -1552,6 +1573,10 @@ class OAuthManager:
|
|||
elif isinstance(claim_data, int):
|
||||
oauth_roles = [str(claim_data)]
|
||||
|
||||
if access_token is not None and not oauth_roles and oauth_allowed_roles and '*' not in oauth_allowed_roles:
|
||||
log.warning('Token exchange denied: no readable roles claim in userinfo or the token')
|
||||
raise HTTPException(status.HTTP_403_FORBIDDEN, detail=ERROR_MESSAGES.ACCESS_PROHIBITED)
|
||||
|
||||
log.debug('Oauth Roles claim: %s', oauth_claim)
|
||||
log.debug('User roles from oauth: %s', oauth_roles)
|
||||
log.debug('Accepted user roles: %s', oauth_allowed_roles)
|
||||
|
|
@ -1598,9 +1623,10 @@ class OAuthManager:
|
|||
user_data,
|
||||
provider,
|
||||
*,
|
||||
access_token: str | None = None,
|
||||
db=None,
|
||||
):
|
||||
determined_role = await self.get_user_role(user, user_data)
|
||||
determined_role = await self.get_user_role(user, user_data, access_token=access_token)
|
||||
if user.role == determined_role:
|
||||
return user
|
||||
|
||||
|
|
@ -1623,11 +1649,7 @@ class OAuthManager:
|
|||
log.debug('Running OAUTH Group management')
|
||||
oauth_claim = auth_config.OAUTH_GROUPS_CLAIM
|
||||
|
||||
try:
|
||||
blocked_groups = JSONCodec.loads(auth_config.OAUTH_BLOCKED_GROUPS)
|
||||
except Exception as e:
|
||||
log.exception(f'Error loading OAUTH_BLOCKED_GROUPS: {e}')
|
||||
blocked_groups = []
|
||||
blocked_groups = _parse_blocked_groups(auth_config.OAUTH_BLOCKED_GROUPS)
|
||||
|
||||
user_oauth_groups = []
|
||||
# Nested claim search for groups claim
|
||||
|
|
@ -1811,7 +1833,7 @@ class OAuthManager:
|
|||
picture = await resp.read()
|
||||
base64_encoded_picture = base64.b64encode(picture).decode('utf-8')
|
||||
try:
|
||||
return validate_profile_image_url(f'data:{upstream_mime};base64,{base64_encoded_picture}')
|
||||
return validate_image_url(f'data:{upstream_mime};base64,{base64_encoded_picture}')
|
||||
except ValueError:
|
||||
log.warning(
|
||||
f'Rejected OAuth profile picture from {picture_url}: '
|
||||
|
|
@ -1917,7 +1939,7 @@ class OAuthManager:
|
|||
if provider == 'feishu' and isinstance(user_data, dict) and 'data' in user_data:
|
||||
user_data = user_data['data']
|
||||
if not user_data:
|
||||
log.warning(f'OAuth callback failed, user data is missing: {token}')
|
||||
log.warning('OAuth callback failed for provider %s, user data is missing', provider)
|
||||
raise HTTPException(400, detail=ERROR_MESSAGES.INVALID_CRED)
|
||||
|
||||
# Extract the "sub" claim, using custom claim if configured
|
||||
|
|
|
|||
|
|
@ -1,140 +0,0 @@
|
|||
import site
|
||||
from datetime import datetime
|
||||
from html import escape
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List
|
||||
|
||||
from fpdf import FPDF
|
||||
from markdown import markdown
|
||||
from open_webui.env import FONTS_DIR, STATIC_DIR
|
||||
from open_webui.models.chats import ChatTitleMessagesForm
|
||||
|
||||
|
||||
class PDFGenerator:
|
||||
"""
|
||||
Description:
|
||||
The `PDFGenerator` class is designed to create PDF documents from chat messages.
|
||||
The process involves transforming markdown content into HTML and then into a PDF format
|
||||
|
||||
Attributes:
|
||||
- `form_data`: An instance of `ChatTitleMessagesForm` containing title and messages.
|
||||
|
||||
"""
|
||||
|
||||
def __init__(self, form_data: ChatTitleMessagesForm):
|
||||
self.html_body = None
|
||||
self.messages_html = None
|
||||
self.form_data = form_data
|
||||
|
||||
self.css = Path(STATIC_DIR / 'assets' / 'pdf-style.css').read_text()
|
||||
|
||||
def format_timestamp(self, timestamp: float) -> str:
|
||||
"""Convert a UNIX timestamp to a formatted date string."""
|
||||
try:
|
||||
date_time = datetime.fromtimestamp(timestamp)
|
||||
return date_time.strftime('%Y-%m-%d, %H:%M:%S')
|
||||
except (ValueError, TypeError) as e:
|
||||
# Log the error if necessary
|
||||
return ''
|
||||
|
||||
def _build_html_message(self, message: Dict[str, Any]) -> str:
|
||||
"""Build HTML for a single message."""
|
||||
role = escape(message.get('role', 'user'))
|
||||
content = escape(message.get('content', ''))
|
||||
timestamp = message.get('timestamp')
|
||||
|
||||
model = escape(message.get('model') if role == 'assistant' else '')
|
||||
|
||||
date_str = escape(self.format_timestamp(timestamp) if timestamp else '')
|
||||
|
||||
# extends pymdownx extension to convert markdown to html.
|
||||
# - https://facelessuser.github.io/pymdown-extensions/usage_notes/
|
||||
# html_content = markdown(content, extensions=["pymdownx.extra"])
|
||||
|
||||
content = content.replace('\n', '<br/>')
|
||||
html_message = f"""
|
||||
<div>
|
||||
<div>
|
||||
<h4>
|
||||
<strong>{role.title()}</strong>
|
||||
<span style="font-size: 12px;">{model}</span>
|
||||
</h4>
|
||||
<div> {date_str} </div>
|
||||
</div>
|
||||
<br/>
|
||||
<br/>
|
||||
|
||||
<div>
|
||||
{content}
|
||||
</div>
|
||||
</div>
|
||||
<br/>
|
||||
"""
|
||||
return html_message
|
||||
|
||||
def _generate_html_body(self) -> str:
|
||||
"""Generate the full HTML body for the PDF."""
|
||||
escaped_title = escape(self.form_data.title)
|
||||
return f"""
|
||||
<html>
|
||||
<head>
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
</head>
|
||||
<body>
|
||||
<div>
|
||||
<div>
|
||||
<h2>{escaped_title}</h2>
|
||||
{self.messages_html}
|
||||
</div>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
def generate_chat_pdf(self) -> bytes:
|
||||
"""
|
||||
Generate a PDF from chat messages.
|
||||
"""
|
||||
try:
|
||||
global FONTS_DIR
|
||||
|
||||
pdf = FPDF()
|
||||
pdf.add_page()
|
||||
|
||||
# When running using `pip install` the static directory is in the site packages.
|
||||
if not FONTS_DIR.exists():
|
||||
FONTS_DIR = Path(site.getsitepackages()[0]) / 'static/fonts'
|
||||
# When running using `pip install -e .` the static directory is in the site packages.
|
||||
# This path only works if `open-webui serve` is run from the root of this project.
|
||||
if not FONTS_DIR.exists():
|
||||
FONTS_DIR = Path('.') / 'backend' / 'static' / 'fonts'
|
||||
|
||||
pdf.add_font('NotoSans', '', f'{FONTS_DIR}/NotoSans-Regular.ttf')
|
||||
pdf.add_font('NotoSans', 'b', f'{FONTS_DIR}/NotoSans-Bold.ttf')
|
||||
pdf.add_font('NotoSans', 'i', f'{FONTS_DIR}/NotoSans-Italic.ttf')
|
||||
pdf.add_font('NotoSansKR', '', f'{FONTS_DIR}/NotoSansKR-Regular.ttf')
|
||||
pdf.add_font('NotoSansJP', '', f'{FONTS_DIR}/NotoSansJP-Regular.ttf')
|
||||
pdf.add_font('NotoSansSC', '', f'{FONTS_DIR}/NotoSansSC-Regular.ttf')
|
||||
pdf.add_font('Twemoji', '', f'{FONTS_DIR}/Twemoji.ttf')
|
||||
|
||||
pdf.set_font('NotoSans', size=12)
|
||||
pdf.set_fallback_fonts(['NotoSansKR', 'NotoSansJP', 'NotoSansSC', 'Twemoji'])
|
||||
|
||||
pdf.set_auto_page_break(auto=True, margin=15)
|
||||
|
||||
# Build HTML messages
|
||||
messages_html_list: List[str] = [self._build_html_message(msg) for msg in self.form_data.messages]
|
||||
self.messages_html = '<div>' + ''.join(messages_html_list) + '</div>'
|
||||
|
||||
# Generate full HTML body
|
||||
self.html_body = self._generate_html_body()
|
||||
|
||||
pdf.write_html(self.html_body)
|
||||
|
||||
# Save the pdf with name .pdf
|
||||
pdf_bytes = pdf.output()
|
||||
|
||||
return bytes(pdf_bytes)
|
||||
except Exception as e:
|
||||
raise e
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Reference in a new issue