Commit graph

683 commits

Author SHA1 Message Date
Classic298
68a74da70b
Fix: mps inference evaluations (#29735)
* fix(retrieval): serialize local embedding and reranking on MPS

On Apple Silicon the server process is killed outright (SIGSEGV or SIGTRAP, no traceback) partway through answering any question that retrieves from a knowledge base with hybrid search and a local reranking model. The client sees a dropped connection and the answer is lost.

Hybrid search fans its queries out concurrently and every task calls the same shared local model on a worker thread. Torch's Metal shader cache is a process-wide singleton whose lookup tables have no lock, so two of those threads racing inside it corrupt the cache and take the process down with it.

Guard the local SentenceTransformer and CrossEncoder calls with a shared lock that is only a real lock when the selected device is MPS. CPU and CUDA installs keep the concurrency they have today, and external reranking endpoints are untouched. Reranking several queries on a Mac now runs one at a time, which is the cost of the process staying alive.

Verified by driving the real hybrid-search fan-out with 16 concurrent queries: peak simultaneous entries into the local model drops from 16 to 1 on MPS, stays at 16 on CPU, and the returned documents, scores and ordering are byte-identical in every case.

Fixes #29722

* fix(evaluations): serialize the leaderboard embedder against retrieval on MPS

The leaderboard's tag-similarity search builds its own SentenceTransformer, and on Apple Silicon sentence-transformers places it on the MPS device. It runs on a worker thread, so an admin running a leaderboard search while anyone queries a knowledge base puts two threads into torch's Metal backend at the same time, which kills the server process outright with no traceback.

Move the lock added for the retrieval path into env.py, beside the device selection that decides whether MPS is used at all, and take it around the leaderboard's embedding calls as well. Sharing one lock between the two modules is the whole point, since two separate locks would still let a leaderboard search collide with a retrieval query.

Only inference is guarded, matching the retrieval path. Model construction stays as it is here and in the retrieval routers.

Verified by driving the leaderboard similarity path and retrieval reranking from six threads against one instrumented model: peak simultaneous entries drops from six to one on MPS, and the similarity scores are unchanged.

Related to #29722.
2026-09-06 15:20:41 -04:00
Classic298
1bfa59acbd
fix: keep HTML entities in extracted document text (#29736)
Uploading a file whose text contains literal HTML entities stored a rewritten copy of it: ` ` became a non-breaking space, `>` became `>`, and ` ` was decoded twice down to a bare non-breaking space. That stored text is what gets indexed and what the model reads, so notes, specs and source files reached the model differing from the file that was uploaded.

Every loaded document goes through `ftfy.fix_text`, which is there to repair mojibake left by the encoding-detection fallback. Its default configuration also decodes HTML entities, per line and sticky forward: entities are decoded on every line up to the first line holding a literal `<`, then left alone for the rest of the document. The same escape therefore survives or vanishes depending on where it sits in the file. This disables that one behaviour and leaves every other ftfy repair in place.

Text from a third-party extraction engine that returns escaped output now keeps those escapes. Guessing whether an escape is markup or content is the bug being fixed.

Fixes #29732
2026-09-06 15:13:45 -04:00
Classic298
acd03b147c
fix: treat .ino files as source text so knowledge uploads stop failing (#29673)
Uploading an Arduino sketch (`.ino`) to a knowledge base failed with `Expecting value: line 1 column 1 (char 0)` whenever the content extraction engine was Tika or Docling. Browsers send `.ino` as `application/octet-stream`, and the extension was missing from the known source extension list, so the file was handed to the extraction server instead of being read as plain text. The server answered with a non-JSON body and the loader crashed while decoding it. `.cpp` and `.h` sketches in the same folder uploaded fine, because those extensions are already on the list.

Adding `ino` to that list routes it to the plain text loader, the same way the yaml/toml gap was closed in 710320601a. A sketch is plain C++ text, so there is nothing for a document extraction server to do with it.

Verified by dispatch matrix over 35 extensions, 5 content types and all 8 engines against a stub server that reproduces the non-JSON response: the only rows that change are `.ino` under Tika and Docling, which now resolve to the text loader and extract the sketch verbatim. Every other row is unchanged.

Fixes #29670
2026-09-04 19:13:04 -04:00
Classic298
ea1b58eb6a
fix: apply the shared metadata size cap to the last four vector DB backends (#29502)
Chunk metadata inserted into the vector DB carries arbitrary client-
supplied fields (e.g. custom metadata from file uploads), so it needs
the same size cap, type coercion and null-byte sanitization every
other backend already applies through process_metadata(). Four
backends never called it: Qdrant, Qdrant multitenancy, Milvus
multitenancy and Oracle23ai, so an oversized or malformed metadata
blob went into those unfiltered.

Wire process_metadata() in at each backend's single insert/upsert
chokepoint, matching the pattern already used by the other eleven
backends.
2026-09-04 11:42:20 -04:00
Timothy Jaeryang Baek
1caf22b5a8 refac 2026-08-31 00:40:15 -04:00
Timothy Jaeryang Baek
873fb741c2 refac 2026-08-31 00:39:16 -04:00
Timothy Jaeryang Baek
5c62cc0517 chore: format 2026-08-25 16:53:53 -04:00
Timothy Jaeryang Baek
1d6d4e6e66 refac
Co-Authored-By: Classic298 <27028174+Classic298@users.noreply.github.com>
2026-08-25 16:27:17 -04:00
Timothy Jaeryang Baek
6dcc2d5269 refac 2026-08-25 15:48:30 -04:00
Timothy Jaeryang Baek
140d2cf4b5 refac 2026-08-25 15:47:55 -04:00
Timothy Jaeryang Baek
e3a7a64d82 refac 2026-08-25 15:25:07 -04:00
Timothy Jaeryang Baek
278e97589e refac 2026-08-25 15:22:19 -04:00
Classic298
e3e4bd87df
refac: consolidate the web fetch address checks onto the request path (#27823)
* fix: apply the SSRF checks to redirect targets on every web fetch path

Two guards protect server-side fetches: a private-IP check and the operator's `WEB_FETCH_FILTER_LIST`. Neither reached a redirect hop on the aiohttp paths, and the filter list never reached one on the requests paths either.

aiohttp answers IP-literal hosts itself without consulting a resolver, so `_SSRFSafeResolver` was never invoked for a hop such as `http://169.254.169.254/` and the private-IP check simply did not run. With redirect following enabled, a submitted public URL that redirects to an IP literal reached loopback, RFC1918 and cloud-metadata addresses, and the response body was returned to the caller. The filter list was consulted only in `validate_url`, on the originally submitted URL, so a redirect to a filter-listed host was fetched without it ever being applied.

`_SSRFSafeResolver` is replaced by `_SSRFSafeConnector`, which hooks `_resolve_host` so the IP check also covers the IP-literal shortcut and both DNS cache paths. The filter list moves to a per-request hook on each transport, `connect()` for aiohttp and `send()` for the requests adapter, because those see the request destination: at the connection layer a proxied request presents the proxy's host, and a pooled connection skips resolution entirely. This covers every hop, including redirects, on all five aiohttp call sites and both requests sessions. The Playwright loader already validated each hop and is unchanged.

Both gaps required `AIOHTTP_CLIENT_ALLOW_REDIRECTS=true`, which is not the default.

Two behaviour changes for operators. The filter list now applies to redirect targets rather than only to submitted URLs. Under a forward proxy it is evaluated against the request destination instead of the proxy, which also fixes allowlist entries rejecting every fetch in proxied deployments.

* refac: match the web fetch filter list against resolved addresses

The filter list is now evaluated against the hostname together with the addresses it resolves to, at URL validation and on each connection, on both transports. An IPv6 address is also matched by the IPv4 address it carries.

* refac: screen outbound fetch addresses against reserved ranges ipaddress misses

`ipaddress.is_global` was the only test behind the web-fetch address check, and it answers a narrower question than "may we fetch this". Several special-purpose ranges are globally routable by registry while nothing on them is a legitimate destination, so they passed. Classification now screens those ranges on top of `is_global`, and applies the same screen to the IPv4 address embedded in an IPv6 transition encoding rather than only to the literal. All three checkpoints share the predicate, so they all inherit it.

The range list is the exact complement of what CPython's `ipaddress` already models, checked entry by entry against both IANA special-purpose registries. Prefixes IANA marks globally reachable are deliberately left out, so no real destination changes behaviour. Verified against 31 addresses covering every entry, their transition-encoded forms, and public controls in both families: 31/31 expected after, 18/31 before.

* refac: match web fetch filter entries that name an address or a range

A filter entry that parses as an address or a CIDR range is matched by containment rather than by DNS label suffix, so a range covers the addresses inside it and an address matches however it is spelled. A range entry previously matched nothing at all, silently.

The built-in list gains the special-purpose networks that ipaddress.is_global reports as reachable while nothing on them is a legitimate destination, so taking an address out of reach is a WEB_FETCH_FILTER_LIST change rather than a release. Those entries hold whether or not local web fetch is enabled; the private-address rule still follows the toggle.
2026-08-25 11:15:48 -04:00
Timothy Jaeryang Baek
2a0274a0a0 refac 2026-08-24 20:34:59 -04:00
Timothy Jaeryang Baek
170ad0595d refac 2026-08-24 19:48:49 -04:00
Classic298
6b438f1a79
refac: stream web-fetched documents to disk instead of buffering them (#28945)
The binary branch of the web fetch read the entire response body into memory
before writing it out. It now streams in blocks, applies the configured file
size limit the same way the sibling URL endpoint already does, and removes the
temporary file when a download fails partway instead of leaving it behind.
2026-08-24 19:06:13 -04:00
Timothy Jaeryang Baek
97466deea1 refac 2026-08-24 18:38:29 -04:00
Timothy Jaeryang Baek
98ee2bdfd3 refac 2026-08-24 17:56:11 -04:00
Classic298
baeb2dfb83
fix: apply RDS IAM token auth to the pgvector engine (#27754)
With `DATABASE_ENABLE_IAM_TOKEN_AUTH=true` and `VECTOR_DB=pgvector`, startup failed at vector store initialisation with `fe_sendauth: no password supplied`, so the two features could not be used together.

`PgvectorClient` builds its own engine and never got the `do_connect` listener that refreshes the RDS IAM token, and the `ScopedSession` branch that would have reused the instrumented main engine is unreachable because `PGVECTOR_DB_URL` defaults to `DATABASE_URL` and is therefore never falsy.

The pgvector engine now goes through `enable_iam_token_auth()` like the main and Alembic engines. Since a token authenticates exactly one host/port/user, that function now attaches the listener only to engines pointing at the same target, so a `PGVECTOR_DB_URL` aimed at a separate database keeps the password from its own URL instead of having it overwritten; the skip is logged with both identities.

Fixes #27752
2026-08-24 07:35:19 -04:00
Classic298
f73f09a3e0
refac: drop the redundant .keys() from two dict membership tests (#28859)
`x in d` and `x in d.keys()` are identical for a plain dict, so the `.keys()` call builds a throwaway view and reads as if it were doing something. Both sites operate on a plain dict: `combined` in `merge_and_sort_query_results` is a local `dict()`, and `ui_settings` comes from `UserSettings.model_dump()` where `ui` is annotated `dict | None` and is already guarded against None on the preceding line.

No behaviour change, and no measurable speedup either, so this is a readability cleanup rather than a performance one.

Sites where `.keys()` is load-bearing are left alone: the `list(d.keys())` snapshots taken before mutating during iteration, and the places where `.keys()` is the iteration or comprehension source rather than a membership test.
2026-08-23 16:02:09 -04:00
Timothy Jaeryang Baek
886248de36 refac 2026-08-23 13:33:51 -04:00
Classic298
069f49fcd2
refac: remove unreachable rate limit handler from DuckDuckGo web search (#28943)
The DuckDuckGo search path catches RatelimitException from the ddgs library. That exception is defined by the library but never raised anywhere in it, checked against the pinned 9.14.4 and against 9.11.3, so the handler could never run. The two fallbacks around it were dead for the same reason: ddgs.text() returns a non-empty list or raises, so None and an empty list are not outcomes it can produce.

Removing all three leaves one call and changes nothing observable. A refused or rate limited search already came out as a failed search, with the error shown to the user and the traceback in the log, and it still does.

The backend argument is now passed as backend or 'auto' rather than conditionally omitted, because 'auto' is the library's own default for that parameter, so every configured value including unset and empty resolves exactly as before. Verified by running the old and the new function side by side against a stubbed library covering normal results, the domain filter, all four backend settings and a failing search, with identical results in every case.
2026-08-23 12:59:05 -04:00
Classic298
c7f306031d
refactor: remove the unreachable async half of the Mistral loader and an unused Datalab helper (#28839)
The Mistral OCR loader has a full async pipeline beside its synchronous one: an async load, its own upload, signed URL, OCR, delete and retry helpers, a pooled session and a batch loader on top. The only way in was the batch loader, which nothing calls, so the entire async half was unreachable. Everything that loads documents goes through the synchronous path, and the shared loader entry point runs it in a worker thread. The Datalab loader carries a public request status poller with no caller either, since its own load inlines the polling it needs.

With the async half gone, the retry classifier's two aiohttp branches can no longer be reached, since the only retried calls are synchronous, so those go with it along with the aiohttp import that existed solely to feed them, and a timeout attribute that nothing reads any more. The class docstring loses the three bullets that only described the removed pipeline, and four docstrings stop calling themselves the sync version of something that no longer has an async counterpart.

This removes around 350 lines and leaves one code path per loader instead of one live path and one that cannot be entered.
2026-08-20 13:14:14 -07:00
Classic298
27402ff210
refac: issue Playwright web loader requests from the shared HTTP clients (#28634)
The Playwright loader's route interceptor now performs each intercepted request with the same requests/aiohttp clients the other web loader paths already use and fulfills the page with that response, rather than having the browser issue it. Redirect handling, header forwarding and cookie delivery to the browser are unchanged.

Two consequences worth knowing. Page requests now leave from the backend instead of the browser, so with PLAYWRIGHT_WS_URL set they originate from a different host, and TLS is verified against certifi plus AIOHTTP_CLIENT_SSL_CERT_FILE rather than the browser's own trust store. And because the synchronous interceptor blocks, sub-resources on that path fetch one at a time: 30 assets at 40ms went from 2.01s to 3.01s, and 8 assets at 500ms from 1.05s to 4.50s. The asynchronous path is unaffected, at 0.65s and 1.05s respectively.
2026-08-17 01:06:12 -06:00
G30
104a0f2f11
fix: remove vestigial api_base_url param that broke the Tavily web loader (#27636) 2026-08-12 02:05:26 -05:00
Timothy Jaeryang Baek
5cd9a39534 refac 2026-08-10 23:40:31 -06:00
Classic298
3dbb4078b3
fix: repair two broken logging calls, one of which makes VECTOR_DB=opengauss unusable (#27838)
SRC_LOG_LEVELS became an empty dict when per-module log levels were dropped, and env.py keeps it only as a legacy name. opengauss.py is the last thing in the tree that still indexes it, at module scope, so importing the module raises KeyError: 'RAG' and any deployment on VECTOR_DB=opengauss dies the first time it touches the vector store. The factory imports it lazily, which is why nothing else trips over it. Deleting the line is the whole fix: every other vector backend takes getLogger(__name__) and inherits the root level.

colbert.py passes an argument to a message with no placeholder to consume it:

    log.info('ColBERT: Loading model', name)

At INFO, which is the default, logging evaluates 'ColBERT: Loading model' % ('colbert-ir/colbertv2.0',) and raises TypeError: not all arguments converted during string formatting. The record is swallowed by handleError, so loading a ColBERT reranker prints '--- Logging error ---' plus a traceback to stderr instead of the model name. Adding %s prints the name and drops the traceback.
2026-08-10 22:19:41 -06:00
Classic298
c5ec01b1f9
fix: make the aiodns resolver opt-in and pin aiodns to 3.6.1 (#28242)
Since v0.11.0 shipped aiodns, aiohttp silently switched every outbound request from the OS resolver to c-ares. On some Windows hosts the bundled c-ares 1.34.6 (pycares 5) discovers only 127.0.0.1:53 as nameserver, so every external provider lookup fails (#28013). In Docker the long-lived c-ares channel intermittently stops resolving container names while Docker's embedded DNS keeps answering, which wipes the Ollama model list and fails all in-flight chats with a misleading "Model not found" (#28215).

This restores the pre-0.11 ThreadedResolver (OS resolver) by default and gates the c-ares path behind a new env var, AIOHTTP_CLIENT_ASYNC_DNS_RESOLVER, off by default. The event-loop DNS perf improvement is now opt-in for deployments whose resolver setup is known to work with c-ares, instead of a process-wide side effect of the package being installed.

aiodns is also downgraded and pinned to 3.6.1 (pycares<5), the last release before the broken c-ares 1.34.6 build, so opting in does not hit the Windows regression. The hardcoded AsyncResolver in the Mistral OCR loader now follows the same switch. Simply removing aiodns instead was not an option because opting in would then be impossible, and #28215 showed the Docker failure is c-ares itself, not aiodns 4.x.
2026-08-10 19:52:29 -06:00
G30
121f2404ee
fix(retrieval): report why a URL could not be read instead of blaming the knowledge base (#28362)
Fetching a URL and saving it were reported as one thing. Everything from
reading the URL to writing the vector database sat inside a single try,
whose handler blamed the knowledge base, so a page that could not be
fetched, parsed or resolved was reported as a knowledge base error even
though nothing had reached the knowledge base yet. Reading the URL now has
its own handler that names the URL, and the knowledge base message is left
to the step that actually touches it.

When YouTube refused a transcript the reason was discarded earlier still:
the loader caught the error, logged it, and returned an empty document
list, so the empty result failed downstream and even the salvageable
explanation was gone before a message was produced. The loader now raises
YoutubeTranscriptError carrying a readable reason, mapped from the
transcript library's own exception types. Blocked requests mention that a
proxy can be configured, and disabled, age restricted, unavailable and
missing language cases each say what actually happened.

URLs that attach successfully are unaffected.
2026-08-10 19:18:04 -06:00
Timothy Jaeryang Baek
1b72899f24 refac 2026-08-10 00:26:44 -06:00
Classic298
2d18727ab8
perf: build info log messages lazily so raising the log level actually saves work (#27837)
Some checks failed
Python CI / Ruff Format (3.11) (push) Has been cancelled
Python CI / Ruff Format (3.12) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true USE_CUDA_VER=cu126 free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true USE_CUDA_VER=cu126 free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / notify-helm-charts (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (, main) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda, cuda) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda126, cuda126) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-ollama, ollama) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-slim, slim) (push) Has been cancelled
Raising GLOBAL_LOG_LEVEL to WARNING buys quieter output but not less work: 241 INFO call sites interpolate their payload into an f-string before the logging call gets to drop it. The heaviest is get_doc, which logs every chunk id and metadata dict in a collection, so on the full-context retrieval path that is the entire knowledge base, once per chat request.

That one line at WARNING, CPython 3.12:

| knowledge base | payload | before   | after   |
| -------------- | ------- | -------- | ------- |
| top-k of 3     | 1.2 kB  | 3.8 us   | 0.07 us |
| 500 chunks     | 201 kB  | 583.6 us | 0.08 us |
| 5000 chunks    | 2.0 MB  | 5.8 ms   | 0.15 us |

The lazy form log.info('query_doc:result %s %s', result.ids, result.metadatas) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. Output at INFO is byte-identical. Two sites that already built their message eagerly, one str concat and one % operator, move to the same lazy form.
2026-08-02 15:39:10 -05:00
Classic298
52cfb02c72
perf: build debug log messages lazily so disabled debug logs cost nothing (#27834)
GLOBAL_LOG_LEVEL defaults to INFO, so every log.debug(...) in the backend is discarded, but the message is built first: 187 call sites interpolate their payload into an f-string before the logging call runs, so the work happens on every request and the result is thrown away. The worst one sits in process_chat_payload and stringifies the whole request body, full conversation history included, once per chat completion.

That one line with DEBUG disabled, CPython 3.12:

| conversation | payload | before   | after   |
| ------------ | ------- | -------- | ------- |
| 4 messages   | 1.2 kB  | 3.4 us   | 0.07 us |
| 20 messages  | 17 kB   | 24.8 us  | 0.07 us |
| 60 messages  | 123 kB  | 216.6 us | 0.07 us |

The lazy form log.debug('form_data: %s', form_data) hands the payload to record.getMessage(), which the InterceptHandler only reaches once a record has passed the level check. With DEBUG enabled the emitted lines are byte-identical, f'{x=}' sites included: those map to %r. MistralLoader._debug_log callers get the same treatment, since that wrapper already forwards *args.
2026-07-31 19:09:01 -05:00
Timothy Jaeryang Baek
bb0f898b43 refac 2026-07-31 17:41:14 -04:00
Classic298
d03e9af0b7
perf: use the orjson codec to parse Oracle vector metadata (#27813)
`_json_to_metadata` parses the metadata of every result row returned by search and get. It now goes through `JSONCodec`, which selects orjson when `ENABLE_ORJSON` is set.

The text it parses is produced by Oracle's own `JSON_SERIALIZE`, and the column is a native `JSON` type, so the database normalises whatever was written and the reader never depends on the writer's escaping.

The matching `_metadata_to_json` write deliberately keeps stdlib `json`: it passes `default=self._decimal_handler`, orjson accepts none of stdlib's keyword arguments, and dropping the handler would turn a currently successful insert of a `Decimal` into a hard failure. The read side has no such constraint.

With `ENABLE_ORJSON` unset, which is the default, `JSONCodec` is stdlib `json` and this call site behaves exactly as before.
2026-07-31 17:31:14 -04:00
Classic298
6be11d4fc9
chore: remove dead json imports (#27815)
Fourteen modules import `json` without using it. Ruff flags every one with F401, and a word-boundary search for `json` in each file matches only the import line itself, including inside strings, comments and annotations.

Two exclusions, both deliberate. Migration files are left alone: the import is equally dead there, but those files are frozen history and not worth the churn. `models/chats.py` has the same dead import and is handled in its own change, so it is skipped here to avoid two changes touching the same line.

No behaviour change.
2026-07-31 17:25:40 -04:00
Classic298
f4c6a76651
perf: use the orjson codec for Valkey vector metadata (#27805)
The Valkey backend serializes chunk metadata on every insert and parses it back on every result row in `get` and `query`. Both directions now go through `JSONCodec`, which selects orjson when `ENABLE_ORJSON` is set.

The stored `metadata_json` field is never matched against as text. `_build_filter_expression` only emits TAG predicates, and the TAG fields are `id`, `hash`, `file_id`, `source` and `knowledge_base_id`; `metadata_json` appears only as a return field that is immediately re-parsed. So rows written with escaped non-ASCII and rows written raw are indistinguishable to every reader, and no migration is needed.

`process_metadata` already stringifies datetimes and strips null bytes and lone surrogates before the write, so the two backends cannot disagree about what is serializable here.

Both read `except` clauses widen from `(json.JSONDecodeError, TypeError)` to `(ValueError, TypeError)`. The codec falls back to engineio's codec, which installs `parse_int=_safe_int` and raises a bare `ValueError` for integer literals longer than 100 characters; the narrower clause would have let that escape and abort a search instead of yielding empty metadata. `json.JSONDecodeError` is a `ValueError` subclass, so this is a strict superset. That removes the module's last use of stdlib `json`, so the import goes with it.

With `ENABLE_ORJSON` unset, which is the default, `JSONCodec` is stdlib `json` and this call site behaves exactly as before.
2026-07-31 17:24:57 -04:00
Classic298
466e05801b
perf: stop query_collection blocking the event loop (#27824)
RAG vector search runs in a thread pool, but then calls `future.result()` on the event loop thread, so the whole worker freezes until every collection answers. Every other user's token stream stops for that long. It's the default retrieval path.

Now `asyncio.gather` over `asyncio.to_thread`, matching what `routers/retrieval.py:2779` already does for the same call.

Measured with 3 queries across 4 collections, 60 ms search, and a second request wanting a turn every 5 ms:

| | before | after |
|---|---|---|
| RAG call | 62.0 ms | 61.2 ms |
| other request's turns | 0 | 7 |
| its worst stall | 62.5 ms | 16.0 ms |

Same results, same order, same `(result, error)` contract. Cancellation now lands mid-search instead of after every thread finishes. Threads move from an unbounded per-call pool to the loop's bounded shared one.
2026-07-31 17:24:31 -04:00
Timothy Jaeryang Baek
810378c0b8 refac
Some checks failed
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true USE_CUDA_VER=cu126 free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true USE_CUDA_VER=cu126 free_disk:true name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Has been cancelled
Frontend Build / Format & Build (push) Has been cancelled
Frontend Build / Unit Tests (push) Has been cancelled
Python CI / Ruff Format (3.11) (push) Has been cancelled
Python CI / Ruff Format (3.12) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda suffix:-cuda]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:cuda126 suffix:-cuda126]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:main suffix:]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:ollama suffix:-ollama]) (push) Has been cancelled
Create and publish Docker images with specific build args / merge (map[name:slim suffix:-slim]) (push) Has been cancelled
Create and publish Docker images with specific build args / notify-helm-charts (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (, main) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda, cuda) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda126, cuda126) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-ollama, ollama) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-slim, slim) (push) Has been cancelled
2026-07-27 19:39:36 -04:00
Timothy Jaeryang Baek
c004b4ecb5 chore: format 2026-07-27 04:38:46 -04:00
Timothy Jaeryang Baek
56183fcb17 refac 2026-07-27 04:27:13 -04:00
Timothy Jaeryang Baek
4eab2550a0 refac 2026-07-27 04:17:03 -04:00
Timothy Jaeryang Baek
48ee357156 refac 2026-07-27 03:50:18 -04:00
Timothy Jaeryang Baek
7e31f64bc8 refac 2026-07-27 03:50:14 -04:00
Timothy Jaeryang Baek
8710c448a9 refac 2026-07-27 02:17:16 -04:00
Classic298
6d4c02a89e
refac: owner-bind ephemeral web-search RAG collections (#26706)
The web-search-* namespace was the one collection namespace filter_accessible_collections admitted unconditionally for any non-admin user, on both read and write, unlike file-*, user-memory-* and knowledge bases which are owner-scoped. process_web_search now mints these ephemeral per-query collections as web-search-{user.id}-<hash>, and the access helper only admits web-search-{requester.id}-* names, so a web-search collection is readable and writable only by the user who created it (admins keep their bypass). The collections hold transient public web-search results and their names are non-enumerable query hashes, so there was no demonstrated cross-user access path; this removes the namespace exception so the per-user scoping the other namespaces enforce also covers web-search.

Co-authored-by: rexpository <rexpository@users.noreply.github.com>
2026-07-27 02:08:04 -04:00
Classic298
e17db990af
fix: parse .msg uploads via unstructured instead of extract_msg (#26704)
The .msg branch routed to langchain's OutlookMessageLoader, which requires the extract_msg package. extract_msg pins beautifulsoup4<4.14, but we pin unstructured==0.22.31 (needs beautifulsoup4>=4.14.3) and beautifulsoup4==4.14.3, so extract_msg can never be installed alongside the current dependency set. As a result the .msg path could not function on any supported install: uploads failed at runtime with an ImportError, and adding the missing package broke the build with an unsatisfiable resolver error.

Switch to UnstructuredEmailLoader, which parses .msg through unstructured's partition_msg (backed by python-oxmsg). Both are already shipped, so .msg uploads work with no new dependency and no version conflict. Attachment partitioning is disabled to preserve the previous body-only extraction behaviour.

Fixes #26690
2026-07-27 02:01:04 -04:00
Timothy Jaeryang Baek
c4f5ac65ee refac 2026-07-27 01:59:17 -04:00
Timothy Jaeryang Baek
def26ce266 refac 2026-07-27 01:21:32 -04:00
Timothy Jaeryang Baek
1717b493d8 refac
Co-Authored-By: Classic298 <27028174+Classic298@users.noreply.github.com>
2026-07-27 00:54:28 -04:00
Timothy Jaeryang Baek
20647bd2d5 chore: format 2026-07-27 00:12:47 -04:00