* feat: add support for Valkey vector database
Signed-off-by: Riley Des <riley.desserre@improving.com>
* feat: add CLIENT SETNAME to Valkey vector store connections
Set client_name on GlideClientConfiguration for both the main client
and batch client so connections are identifiable in CLIENT LIST,
monitoring dashboards, and CloudWatch metrics.
Signed-off-by: Riley Des <riley.desserre@improving.com>
---------
Signed-off-by: Riley Des <riley.desserre@improving.com>
Firecrawl /search returns either `{"data": [...]}` (flat list — v1, and
what frost19k reported on #23966) or `{"data": {"web": [...]}}` (v2,
current production). The parser only handled the dict shape:
data = response.get('data') or {}
results = data.get('web') or []
On a list-shape response, `data.get('web')` raised AttributeError,
caught by the function's outer try/except, and `search_firecrawl`
silently returned []. Web search worked against v2 endpoints but is one
upstream-format-change away from failing closed again. Accept either.
Open WebUI's collection ACL accepted any unknown name as a
legacy/ephemeral collection. In Milvus multi-tenancy mode that name
becomes the `resource_id` and is interpolated unescaped into a SQL-like
Milvus expression — `resource_id == '<name>'` — so a name like
x' or resource_id != '' or resource_id == 'x
turns the filter into a tautology and returns every tenant's chunks
from the shared collection.
All collection names Open WebUI generates are UUIDs, SHA-256 hex
digests, or fixed-prefix variants of those — they all fit
[A-Za-z0-9_-]. Add a strict format check in
filter_accessible_collections (utils.py) that drops any name outside
that set before any ACL or vector-store lookup, applied even on the
admin bypass path. _validate_collection_access then surfaces the dropped
name as a 403.
As defense in depth, MilvusClient now validates resource_id at every
expression-construction site and escapes single quotes / backslashes in
any other string interpolated into a filter (delete ids, metadata
filter values). Non-string filter values are typed-checked instead of
str()-formatted.
Co-authored-by: Claude <noreply@anthropic.com>
validate_url() resolves DNS to check IPs but discards the result; the
HTTP client resolves again independently. Between those two lookups an
attacker can swap the DNS record from a public IP to an internal one
(DNS rebinding).
Push the IP-is-global check into the actual connection layer so the
validated resolution is the one used for the TCP connect:
- aiohttp (_fetch): _SSRFSafeResolver wraps DefaultResolver and rejects
non-global IPs at resolve time (zero TOCTOU window).
- requests (_scrape): _SSRFSafeAdapter mounts custom urllib3 connection
classes whose _new_conn resolves, validates, and connects to the
validated IP in one shot (zero TOCTOU window).
Both paths respect ENABLE_RAG_LOCAL_WEB_FETCH (skip validation when on).
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
SafePlaywrightURLLoader validated only the initially submitted URL and
then let the browser follow HTTP redirects and client-side navigations
without re-checking them, so a public URL could redirect into the
internal network (cloud metadata, RFC1918, loopback). Intercept
document-type requests, re-run validate_url on each, and apply the same
redirect policy as the requests loader (blocked unless
AIOHTTP_CLIENT_ALLOW_REDIRECTS). Sub-resource requests pass through
unchanged so page rendering performance is unaffected.
Co-authored-by: POV9en <POV9en@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The __main__ block called search_bing() with 4 positional arguments, but
the function requires 5 (subscription_key, endpoint, locale, query,
count). Running `python -m open_webui.retrieval.web.bing` raised a
TypeError and, before failing, silently misrouted every argument. Read
the key/endpoint from environment variables, matching config.py defaults.
Closes#24765
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
urllib.parse.urlparse and requests/aiohttp disagree on how to split URLs
containing backslash, tab, CR, or LF in or around the netloc. urlparse
treats backslash as part of userinfo and uses what follows '@' as the
host; requests treats backslash as the start of the path and connects
to whatever precedes it. The same URL therefore passes the private-IP
filter (urlparse sees a public host) but reaches an internal target
(requests connects to e.g. 127.0.0.1). End result is an SSRF that the
existing IP block list cannot catch because it's evaluating the wrong
host.
PoC: http://127.0.0.1:6666\@1.1.1.1 — urlparse hostname is 1.1.1.1
(global, passes), requests reaches 127.0.0.1 (loopback).
Reject up front any URL containing one of the four documented parser-
confusing characters before either parser gets a chance to interpret
it. None of these characters is valid in an unencoded URL (\ should
always be %5C, whitespace should be %09 / %0A / %0D), so this is a
pure defensive rejection with no legitimate-input false positives.
Reported by Fushuling and RacerZ-fighting in GHSA-8w7q-q5jp-jvgx.
Co-authored-by: Fushuling <Fushuling@users.noreply.github.com>
Co-authored-by: RacerZ-fighting <RacerZ-fighting@users.noreply.github.com>
validate_url() in retrieval/web/utils.py only validates the initial URL.
The HTTP clients used downstream (sync requests, sync requests via the
parent WebBaseLoader._scrape, aiohttp via load_url_image) followed 3xx
redirects by default and did not re-validate the redirect target against
the private-IP / metadata-IP block list. An authenticated user could
submit a public URL that 302-redirected to an internal address (RFC1918,
127.0.0.1, 169.254.169.254, etc.) and the redirected response was returned
to them, enabling SSRF reads of internal services and cloud metadata.
Three call sites needed allow_redirects=False to match the policy already
enforced on the async _fetch() path:
- SafeWebBaseLoader: override requests_kwargs in __init__ so that the
inherited synchronous _scrape() path passes allow_redirects=False to
self.session.get() (the parent WebBaseLoader uses requests' default
allow_redirects=True).
- get_content_from_url (retrieval/utils.py): pass allow_redirects=False
on the streamed requests.get(...) call.
- load_url_image (routers/images.py, image-edits endpoint): pass
allow_redirects=False on the aiohttp session.get(...) call.
Reports consolidated under GHSA-rh5x-h6pp-cjj6:
- GHSA-rh5x-h6pp-cjj6 (tenbbughunters / Tenable) - sync _scrape
- GHSA-5vxg-6gmv-m2qr (YLChen-007) - load_url_image
- GHSA-hf76-c83f-63w2 (tempcollab) - aiohttp _fetch (already fixed)
- GHSA-h55f-h5fh-mvm4 (sneaXOR) - get_content_from_url
Add configurable reranker batch size (env var RAG_RERANKING_BATCH_SIZE,
default 32) following the same pattern as RAG_EMBEDDING_BATCH_SIZE.
- config.py: PersistentConfig for RAG_RERANKING_BATCH_SIZE
- main.py: import, state init, pass to get_reranking_function
- colbert.py: accept batch_size param in predict() (was hardcoded 32)
- utils.py: get_reranking_function passes batch_size at call time
- retrieval.py: expose in config GET/POST endpoints and ConfigForm
- Documents.svelte: add Reranking Batch Size input in admin settings
Closes#23730
Loader.load() dispatches to the underlying langchain document loaders
(PyMuPDF, Unstructured, python-docx, Tika, …) which are all
synchronous and CPU/IO-bound. process_file() awaited it directly on
the event loop, so parsing a non-trivial PDF/DOCX would freeze the
entire FastAPI app for the duration of the parse — which is what users
experience as "the server hangs whenever I upload a file."
Add an `aload()` async wrapper on Loader that runs the sync load on a
worker thread via asyncio.to_thread, and update process_file() to
await it. The sync API is preserved so existing callers that already
run inside run_in_threadpool (e.g. save_docs_to_vector_db) are
unaffected.
https://claude.ai/code/session_01JSr4NZSskEUQvoJnavVXh8
Co-authored-by: Claude <noreply@anthropic.com>
* fix(retrieval): offload sync VECTOR_DB_CLIENT calls in async paths via AsyncVectorDBClient
The vector DB backends (Chroma, pgvector, Qdrant, Milvus, Pinecone,
Weaviate, …) are uniformly synchronous and their methods perform
blocking network or disk I/O. Multiple async route handlers and helpers
were calling them directly on the event loop — file processing,
memories, knowledge bases, hybrid search bookkeeping — so a single
upsert/delete/search would freeze every other in-flight request for the
duration of the call.
Introduce `AsyncVectorDBClient`, a thin async facade that wraps the
existing sync client and dispatches each method through
`asyncio.to_thread`. It mirrors `VectorDBBase` exactly and forwards
*args/**kwargs so backend-specific extra parameters keep working.
Update every async-context call site (routers/retrieval, routers/files,
routers/memories, routers/knowledge, retrieval/utils,
tools/builtin) to await `ASYNC_VECTOR_DB_CLIENT` instead of calling the
sync client directly. Two helpers that were sync-only also acquire
async siblings or are awaited via `asyncio.to_thread` at their async
call site (`remove_knowledge_base_metadata_embedding`,
`get_all_items_from_collections`, `query_doc`).
The original sync `VECTOR_DB_CLIENT` is unchanged, so callers that
already run inside `run_in_threadpool` (e.g. `save_docs_to_vector_db`
and the sync `query_doc`/`get_doc` helpers) are unaffected.
https://claude.ai/code/session_01JSr4NZSskEUQvoJnavVXh8
* fix(retrieval): restore explicit AsyncVectorDBClient signatures matching VectorDBBase
Per PR review: the original *args/**kwargs forwarding lost type
safety and IDE/static-analysis support. Restore explicit signatures
that mirror VectorDBBase exactly, so:
* Bad kwargs fail at the facade boundary instead of inside the
worker thread (where the resulting TypeError tends to be
swallowed by surrounding `try/except`).
* IDE autocomplete and static analysis work as expected.
* The stated intent ("mirror VectorDBBase exactly") now holds at
the API contract level, not just behaviourally.
While doing this, surface a pre-existing bug in
`delete_entries_from_collection` that the stricter typing flagged:
the call passed `metadata={'hash': hash}` which is not a parameter
on `VectorDBBase.delete` nor any backend. The TypeError raised
inside the sync delete was silently swallowed by `except Exception`
so the endpoint always reported `{'status': False}` for every
request instead of actually deleting matching vectors. Replace with
`filter=...` to do what the endpoint name promises.
The thorough review's other note (no concurrency/backpressure on
the shared default threadpool) is intentionally not addressed here:
asyncio.to_thread on the shared executor is the right primitive for
this use case; per-domain bounded executors would add lifecycle
complexity disproportionate to the problem and the loop is no
longer blocked, which was the actual bug.
https://claude.ai/code/session_01JSr4NZSskEUQvoJnavVXh8
* fix(retrieval): parallelize hybrid-search collection prefetch; document async facade contracts
Address PR review findings:
1. Hybrid-search prefetch was sequential
`query_collection_with_hybrid_search` previously awaited
`ASYNC_VECTOR_DB_CLIENT.get(name)` once per collection in a for
loop. Each call already off-loaded to a worker thread, but
awaiting them serially meant total prefetch latency scaled
linearly with the number of collections. Run them concurrently
with `asyncio.gather` so multi-collection queries actually
benefit from the threadpool. Per-collection exception handling
is preserved by wrapping each fetch in a small helper that
logs and returns `(name, None)` on failure, so a single bad
collection cannot poison the whole gather.
2. Document the thread-safety expectation explicitly
The facade now formally states what was always implicit: the
sync `VECTOR_DB_CLIENT` is shared across worker threads, so the
underlying backend driver must be thread-safe. This is not a
new exposure — `save_docs_to_vector_db` already called the sync
client from `run_in_threadpool`. Adding a global lock here
would defeat the responsiveness the facade exists to provide;
backends that cannot tolerate concurrent access should grow
their own internal serialization.
3. Document the API-surface choice and `.sync` escape hatch
The strict `VectorDBBase` mirror was a deliberate choice (the
previous `*args/**kwargs` revision let a `metadata=` typo
silently break an endpoint). Document it, and call out the
`.sync` escape hatch with an example for callers that genuinely
need a backend-specific parameter not on `VectorDBBase`.
https://claude.ai/code/session_01JSr4NZSskEUQvoJnavVXh8
* fix(retrieval): guard /delete against null file.hash and let HTTPException reach the client
Address PR review finding on the `metadata=` → `filter=` change in
`delete_entries_from_collection`.
The new `filter={'hash': hash}` query was correct for files that
have a hash, but did not handle `file.hash is None` (unprocessed,
failed, or legacy records). The match semantics of a null filter
value are backend-dependent — some ignore the key entirely, some
treat it as "metadata field absent" and match every such row — so
issuing the query risked deleting unrelated entries.
* Reject `hash is None` up front with a 400 explaining the file
has no hash to target.
* Narrow the surrounding `except Exception` so it no longer
swallows `HTTPException`. Without this fix the new 400 (and the
pre-existing 404 for missing files) would be silently re-shaped
into `{'status': False}` and the caller could not distinguish a
bad-request input from a backend error.
https://claude.ai/code/session_01JSr4NZSskEUQvoJnavVXh8
---------
Co-authored-by: Claude <noreply@anthropic.com>
