Commit graph

38 commits

Author SHA1 Message Date
Classic298
16c2a9eda4
fix: index the chat queries that make large SQLite instances unusable (#27663)
Some checks are pending
Python CI / Ruff Format (3.11) (push) Waiting to run
Python CI / Ruff Format (3.12) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true USE_CUDA_VER=cu126 free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/amd64 runner:ubuntu-latest], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args: free_disk:false name:main suffix:]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true USE_CUDA_VER=cu126 free_disk:true name:cuda126 suffix:-cuda126]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_CUDA=true free_disk:true name:cuda suffix:-cuda]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_OLLAMA=true free_disk:false name:ollama suffix:-ollama]) (push) Waiting to run
Create and publish Docker images with specific build args / build (map[arch:linux/arm64 runner:ubuntu-24.04-arm], map[build_args:USE_SLIM=true free_disk:false name:slim suffix:-slim]) (push) Waiting to run
Create and publish Docker images with specific build args / copy-to-dockerhub (-ollama, ollama) (push) Blocked by required conditions
Create and publish Docker images with specific build args / copy-to-dockerhub (-slim, slim) (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge (map[name:cuda suffix:-cuda]) (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge (map[name:cuda126 suffix:-cuda126]) (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge (map[name:main suffix:]) (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge (map[name:ollama suffix:-ollama]) (push) Blocked by required conditions
Create and publish Docker images with specific build args / merge (map[name:slim suffix:-slim]) (push) Blocked by required conditions
Create and publish Docker images with specific build args / notify-helm-charts (push) Blocked by required conditions
Create and publish Docker images with specific build args / copy-to-dockerhub (, main) (push) Blocked by required conditions
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda, cuda) (push) Blocked by required conditions
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda126, cuda126) (push) Blocked by required conditions
Frontend Build / Format & Build (push) Waiting to run
Frontend Build / Unit Tests (push) Waiting to run
The timer scheduler polls once a second and cancels on every message send and chat open, the sidebar lists chats ordered by `updated_at`, and the folder badges count unread chats per folder. None of those could be served by an index, so each call read most of the `chat` table, and because `meta` sits after the chat payload column SQLite had to walk every row's overflow pages to get there. On a large history that stalls the sidebar, every chat switch and every send, and the idle poll alone burns about a quarter of a CPU core.

Timers now keep their due time in a dedicated `chat.timer_at` column behind a partial index, and the chat list, unread and unfinished-reply queries each get an index matching their filter and ordering. Existing pending timers are backfilled from their meta by the migration. Dropping the `internal` and `type` checks also makes a forked timer chat inert, where a fork used to copy `meta` verbatim and become a second claim target that could fire a duplicate timer.

Measured on SQLite, same rows returned:

| query | before | after |
|---|---|---|
| idle timer poll (2000 chats, 0.43 GB) | 170 ms | 0.04 ms |
| cancel on send and chat open (4000 chats, 377 MB) | 200 ms | 0.04 ms |
| sidebar chat list (15000 chats, 1.26 GB) | 157 ms | 1.8 ms |
| folder unread badges (15000 chats, 1.4 GB) | 54 ms | 0.2 ms |

PostgreSQL 17 serves all of them as index-only scans with no sort node. Exercised through fresh install, upgrade with seeded data, downgrade and re-upgrade on SQLite and PostgreSQL 17.

Fixes #27622
2026-08-23 16:11:54 -05:00
Classic298
81fe43f210
perf: write a chat's messages in one transaction instead of one per message (#28806)
Saving a chat rewrote its message rows one at a time. Each message took its own session out of the pool and committed on its own, and the save endpoint hands over the entire merged history rather than only what changed, so a two hundred message chat cost two hundred sessions and two hundred commits on every save.

The messages now go through a single select and a single commit. The field mapping for the insert and the update branch moved into two small helpers, so the batch and the single-message path cannot drift apart.

Measured on a two hundred message chat with one message edited: 201 queries and 200 transactions before, 2 queries and 1 transaction after, ~149 ms against ~6 ms. Re-saving an unchanged history now costs one select and no writes at all.

One behaviour change worth stating: a message the database cannot store used to be skipped on its own, and now costs the rest of that same save. This table is a rebuildable fast path, so the reader falls back to the history on the chat row and re-triggers the backfill, and the next save reconciles everything still present. A per-message retry was tried and dropped, because a commit that lands but still raises would re-apply the usage merge and double the recorded token counts.
2026-08-19 12:46:49 -05:00
Classic298
6be11d4fc9
chore: remove dead json imports (#27815)
Fourteen modules import `json` without using it. Ruff flags every one with F401, and a word-boundary search for `json` in each file matches only the import line itself, including inside strings, comments and annotations.

Two exclusions, both deliberate. Migration files are left alone: the import is equally dead there, but those files are frozen history and not worth the churn. `models/chats.py` has the same dead import and is handled in its own change, so it is skipped here to avoid two changes touching the same line.

No behaviour change.
2026-07-31 17:25:40 -04:00
Classic298
699d512e2f
perf: drop redundant session.refresh calls after commit across the model layer (#27381)
Both session factories run with expire_on_commit=False, so ORM objects keep their attribute values after commit. Every session.refresh issued right after a commit therefore re-SELECTed a row whose values the session already held, including full chat JSON blobs and user settings, purely to overwrite identical data. Fifty such calls existed across the model layer, covering nearly every write path in the app (chat inserts, title updates, pin/archive toggles, user role and settings updates, tool, prompt, function, model, file, tag, feedback, memory, automation and grant writes).

All fifty are removed. The only refreshes with an actual job were the two update-then-reload paths in tools and skills, where a Core UPDATE statement bypasses the identity map; those now use session.get(..., populate_existing=True), which guarantees a fresh row in one SELECT whether or not the row was already present in the session (the previous code issued get plus refresh, two SELECTs, on the default configuration).

Benchmark (real SQLite DB, per write):

| write path | before | after |
| --- | --- | --- |
| chat title update, ~600 KB chat blob | 2.08 ms | 1.24 ms |
| user role update, small row | 1.21 ms | 0.68 ms |

On Postgres each removed refresh is additionally a network round trip. The chat-blob case also skips re-parsing the entire JSON document per write.

Functionally verified against a fresh database: user insert, role and settings updates, chat insert (including the server-default meta column, which is always provided client-side), title update and pin toggle, tool insert and the Core-update reload path, tag insert and the prompt insert flow that pins version_id after history creation all return correct values and persist correctly.
2026-07-23 18:08:00 -05:00
Timothy Jaeryang Baek
af9a315ac3 refac 2026-07-20 01:33:47 -04:00
Timothy Jaeryang Baek
743b9fd3ce refac 2026-07-16 21:57:43 -04:00
Timothy Jaeryang Baek
4d27bfff92 refac 2026-07-16 01:42:24 -04:00
Timothy Jaeryang Baek
95391221df refac 2026-06-23 23:35:44 +02:00
Timothy Jaeryang Baek
3f0c0e0a0d refac 2026-06-19 00:16:06 +02:00
Taylor Wilsdon
6cd0ba0b6b
implement date picker and additional columns for analytics (#25922)
Co-authored-by: Tim Baek <tim@openwebui.com>
2026-06-17 03:03:18 +02:00
Algorithm5838
4b33d7ebc1
fix: preserve parent_id on chat_message upsert (#25205) 2026-05-31 14:50:05 -07:00
Timothy Jaeryang Baek
aa06200f78 refac 2026-05-19 20:37:53 +04:00
Timothy Jaeryang Baek
6d0295588e refac: modernize type annotations (PEP 604 / PEP 585) 2026-05-12 17:10:15 +09:00
Timothy Jaeryang Baek
7bcc0e2e5c chore: format 2026-05-09 15:25:27 +09:00
Timothy Jaeryang Baek
485d689cfd refac 2026-05-09 07:52:15 +09:00
Timothy Jaeryang Baek
a32d26e61d refac
Some checks failed
Create and publish Docker images with specific build args / build-main-image (linux/amd64, ubuntu-latest) (push) Has been cancelled
Create and publish Docker images with specific build args / build-main-image (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Create and publish Docker images with specific build args / build-cuda-image (linux/amd64, ubuntu-latest) (push) Has been cancelled
Create and publish Docker images with specific build args / build-cuda-image (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Create and publish Docker images with specific build args / build-cuda126-image (linux/amd64, ubuntu-latest) (push) Has been cancelled
Create and publish Docker images with specific build args / build-cuda126-image (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Create and publish Docker images with specific build args / build-ollama-image (linux/amd64, ubuntu-latest) (push) Has been cancelled
Create and publish Docker images with specific build args / build-ollama-image (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Create and publish Docker images with specific build args / build-slim-image (linux/amd64, ubuntu-latest) (push) Has been cancelled
Create and publish Docker images with specific build args / build-slim-image (linux/arm64, ubuntu-24.04-arm) (push) Has been cancelled
Python CI / Format Backend (push) Has been cancelled
Frontend Build / Format & Build Frontend (push) Has been cancelled
Frontend Build / Frontend Unit Tests (push) Has been cancelled
Create and publish Docker images with specific build args / merge-main-images (push) Has been cancelled
Create and publish Docker images with specific build args / merge-cuda-images (push) Has been cancelled
Create and publish Docker images with specific build args / merge-cuda126-images (push) Has been cancelled
Create and publish Docker images with specific build args / merge-ollama-images (push) Has been cancelled
Create and publish Docker images with specific build args / merge-slim-images (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (, main) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda, cuda) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-cuda126, cuda126) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-ollama, ollama) (push) Has been cancelled
Create and publish Docker images with specific build args / copy-to-dockerhub (-slim, slim) (push) Has been cancelled
2026-05-05 04:21:23 +09:00
Timothy Jaeryang Baek
e29d145a1c refac 2026-04-20 08:48:35 +09:00
Timothy Jaeryang Baek
25898116ea chore: format 2026-04-12 18:12:59 -05:00
Timothy Jaeryang Baek
27169124f2 refac: async db 2026-04-12 14:22:11 -05:00
Timothy Jaeryang Baek
4dea4fdf54 refac 2026-04-02 08:34:49 -05:00
Timothy Jaeryang Baek
15883e5229 refac 2026-04-01 06:00:53 -05:00
Timothy Jaeryang Baek
de3317e26b refac 2026-03-17 17:58:01 -05:00
Classic298
caf3362be8
fix: add missing group_id filter to analytics token usage query (#22167)
The get_token_usage_by_user query lacked group_id filtering, while the
companion get_message_count_by_user query already supported it. When an
admin filtered analytics by user group, message counts were correctly
scoped to the group but token usage totals included data from all users.

Add the group_id parameter and subquery filter to get_token_usage_by_user,
matching the pattern used by get_message_count_by_user and other analytics
queries, and pass group_id through from the analytics endpoint.
2026-03-08 17:11:12 -05:00
Timothy Jaeryang Baek
3ceaa107ab chore: format 2026-03-07 20:14:32 -06:00
Classic298
d1975b740b
fix: add deterministic ordering to chat_ids pagination query to prevent duplicates (#22383) 2026-03-07 20:19:44 -05:00
Timothy Jaeryang Baek
7bda6bf767 fix: PostgreSQL cannot use get_chat_ids_by_model_id
Co-Authored-By: EntropyYue <164553692+EntropyYue@users.noreply.github.com>
2026-02-13 11:20:26 -06:00
Timothy Jaeryang Baek
f376d4f378 chore: format 2026-02-11 16:24:11 -06:00
Timothy Jaeryang Baek
e62649f940 enh: analytics 2026-02-05 00:00:49 -06:00
Timothy Jaeryang Baek
68a1e87b66 enh: analytics model modal 2026-02-04 23:42:46 -06:00
Timothy Jaeryang Baek
3c0313f41b refac 2026-02-02 09:31:52 -06:00
Timothy Jaeryang Baek
d938eb0e76 refac 2026-02-02 09:23:46 -06:00
Timothy Jaeryang Baek
b0c5f7b668 refac 2026-02-02 09:02:34 -06:00
Timothy Jaeryang Baek
c22bb4f853 refac 2026-02-02 08:55:13 -06:00
Tim Baek
b2c2f1bd49 refac 2026-02-01 10:24:04 +04:00
Tim Baek
679e56c494 feat: token analytics 2026-02-01 10:19:59 +04:00
Tim Baek
3da4323ef3 refac 2026-02-01 10:11:13 +04:00
Tim Baek
75e5a485d2 refac 2026-02-01 10:06:52 +04:00
Tim Baek
599cd2eeeb feat: analytics backend API with chat_message table
- Add chat_message table for message-level analytics with usage JSON field
- Add migration to backfill from existing chats
- Add /analytics endpoints: summary, models, users, daily
- Support hourly/daily granularity for time-series data
- Fill missing days/hours in date range
2026-02-01 07:04:13 +04:00