Commit graph

60 commits

Author SHA1 Message Date
Timothy Jaeryang Baek
f607337582 refac 2026-05-14 02:56:44 +09:00
Timothy Jaeryang Baek
6d0295588e refac: modernize type annotations (PEP 604 / PEP 585) 2026-05-12 17:10:15 +09:00
Classic298
e7ba8978c6
fix: reject parser-confusing chars in validate_url to close SSRF bypass (#24534)
urllib.parse.urlparse and requests/aiohttp disagree on how to split URLs
containing backslash, tab, CR, or LF in or around the netloc. urlparse
treats backslash as part of userinfo and uses what follows '@' as the
host; requests treats backslash as the start of the path and connects
to whatever precedes it. The same URL therefore passes the private-IP
filter (urlparse sees a public host) but reaches an internal target
(requests connects to e.g. 127.0.0.1). End result is an SSRF that the
existing IP block list cannot catch because it's evaluating the wrong
host.

PoC: http://127.0.0.1:6666\@1.1.1.1 — urlparse hostname is 1.1.1.1
(global, passes), requests reaches 127.0.0.1 (loopback).

Reject up front any URL containing one of the four documented parser-
confusing characters before either parser gets a chance to interpret
it. None of these characters is valid in an unencoded URL (\ should
always be %5C, whitespace should be %09 / %0A / %0D), so this is a
pure defensive rejection with no legitimate-input false positives.

Reported by Fushuling and RacerZ-fighting in GHSA-8w7q-q5jp-jvgx.

Co-authored-by: Fushuling <Fushuling@users.noreply.github.com>
Co-authored-by: RacerZ-fighting <RacerZ-fighting@users.noreply.github.com>
2026-05-11 00:57:48 +09:00
Timothy Jaeryang Baek
df42d96c95 refac 2026-05-09 21:05:49 +09:00
Classic298
8854541508
fix: prevent redirect-based SSRF in web-fetch and image-load call sites (#24491)
validate_url() in retrieval/web/utils.py only validates the initial URL.
The HTTP clients used downstream (sync requests, sync requests via the
parent WebBaseLoader._scrape, aiohttp via load_url_image) followed 3xx
redirects by default and did not re-validate the redirect target against
the private-IP / metadata-IP block list. An authenticated user could
submit a public URL that 302-redirected to an internal address (RFC1918,
127.0.0.1, 169.254.169.254, etc.) and the redirected response was returned
to them, enabling SSRF reads of internal services and cloud metadata.

Three call sites needed allow_redirects=False to match the policy already
enforced on the async _fetch() path:

- SafeWebBaseLoader: override requests_kwargs in __init__ so that the
  inherited synchronous _scrape() path passes allow_redirects=False to
  self.session.get() (the parent WebBaseLoader uses requests' default
  allow_redirects=True).
- get_content_from_url (retrieval/utils.py): pass allow_redirects=False
  on the streamed requests.get(...) call.
- load_url_image (routers/images.py, image-edits endpoint): pass
  allow_redirects=False on the aiohttp session.get(...) call.

Reports consolidated under GHSA-rh5x-h6pp-cjj6:
- GHSA-rh5x-h6pp-cjj6 (tenbbughunters / Tenable) - sync _scrape
- GHSA-5vxg-6gmv-m2qr (YLChen-007) - load_url_image
- GHSA-hf76-c83f-63w2 (tempcollab) - aiohttp _fetch (already fixed)
- GHSA-h55f-h5fh-mvm4 (sneaXOR) - get_content_from_url
2026-05-09 21:01:45 +09:00
RomualdYT
e0d6074cd2
refactor(firecrawl): use v2 API directly (#23934)
Co-authored-by: Tim Baek <tim@openwebui.com>
2026-04-24 18:32:08 +09:00
Timothy Jaeryang Baek
0e311a95a7 refac 2026-04-24 15:16:37 +09:00
Timothy Jaeryang Baek
fd25152076 refac 2026-04-20 08:34:15 +09:00
Timothy Jaeryang Baek
9c64d84ad9 refac 2026-04-13 15:03:22 -05:00
Classic298
0753409e7b
fix: use ipaddress stdlib for IPv6 SSRF protection (#23453)
The validators.ipv6(ip, private=True) call always returns a falsy ValidationError because validators==0.35.0 does not support the private kwarg for IPv6. This means any hostname resolving to a private IPv6 address (::1, fd00::*, ::ffff:169.254.169.254) bypasses SSRF protection entirely, circumventing the fix for CVE-2025-65958.

Replace both the IPv4 and IPv6 validators-based private checks with Python's stdlib ipaddress module using an allowlist approach (not addr.is_global). This blocks all non-globally-routable addresses — private, loopback, link-local, reserved, multicast, and unspecified — for both IPv4 and IPv6, including IPv4-mapped IPv6 addresses.
2026-04-12 16:34:13 -05:00
Timothy Jaeryang Baek
de3317e26b refac 2026-03-17 17:58:01 -05:00
Tim Baek
a214ec40ea fix 2026-02-06 03:34:21 +04:00
Timothy Jaeryang Baek
89ad1c68d1 enh: FIRECRAWL_TIMEOUT 2026-01-01 02:07:22 +04:00
Classic298
823b9a6dd9
chore/perf: Remove old SRC level log env vars with no impact (#20045)
* Update openai.py

* Update env.py

* Merge pull request open-webui#19030 from open-webui/dev (#119)

Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: Claude <noreply@anthropic.com>

---------

Co-authored-by: Tim Baek <tim@openwebui.com>
Co-authored-by: Claude <noreply@anthropic.com>
2025-12-20 08:16:14 -05:00
Timothy Jaeryang Baek
b02397e460 feat: WEB_LOADER_TIMEOUT 2025-12-08 11:49:27 -05:00
Timothy Jaeryang Baek
743199f2d0 feat/enh: tool server function name filter list 2025-11-25 02:31:34 -05:00
Timothy Jaeryang Baek
2af4c4b3c7 refac 2025-11-18 04:42:09 -05:00
Timothy Jaeryang Baek
02238d3113 feat/security: Add SSRF protection with configurable blocklist
Co-Authored-By: Classic298 <27028174+Classic298@users.noreply.github.com>
2025-11-18 04:40:55 -05:00
Timothy Jaeryang Baek
7faf19dad9 refac 2025-11-06 15:21:06 -05:00
Omar Aburub
3bcf9a442a fix: make SSL verification async 2025-10-29 16:14:53 +03:00
wei840222
7a3f4d85f6 refactor: replace requests with Firecrawl SDK in search and requests Firecrawl SDK in scrape rather than langchain_community FireCrawlLoader 2025-10-26 15:05:35 +08:00
Timothy Jaeryang Baek
e000494e48 refac 2025-10-07 11:53:30 -05:00
Timothy Jaeryang Baek
0214c1e66c refac 2025-09-09 16:48:59 +04:00
_00_
093af754e7
FIX: Playwright Timeout (ms) interpreted as seconds
Fix for Playwright Timeout (ms) interpreted as seconds.

To address https://github.com/open-webui/open-webui/issues/16801

In Frontend Playwright Timeout is setted as (ms), but in backend is interpreted as (s) doing a time conversion for playwright_timeout var (that have to be in ms).

& as  _Originally posted by @rawbby in [#16801](https://github.com/open-webui/open-webui/issues/16801#issuecomment-3216782565)_

> I personally think milliseconds are a reasonable choice for the timeout. Maybe the conversion should be fixed, not the label.
> This would further not break existing configurations from users that rely on their current config.
>
2025-08-23 14:15:00 +02:00
tth37
78befd5a2f fix: Default web loader fail when verify_ssl=False 2025-05-20 19:44:18 +08:00
Timothy Jaeryang Baek
b143c71da2 refac: AIOHTTP_CLIENT_SESSION_SSL 2025-05-14 23:33:52 +04:00
Timothy Jaeryang Baek
42382b5167 fix 2025-05-14 22:46:01 +04:00
Timothy Jaeryang Baek
8732b64b6b feat: external document loader support 2025-05-14 22:28:40 +04:00
tth37
8f7195ceda fix: FireCrawlLoader default mode to scrape 2025-04-24 01:17:35 +08:00
Timothy Jaeryang Baek
09874ab83d fix: FireCrawlLoader 2025-04-24 01:40:34 +09:00
tth37
839ba22c90 feat: Backend for Self-Hosted/External Web Search/Loader Engines 2025-04-14 01:49:05 +08:00
Timothy Jaeryang Baek
48a23ce3fe refac: web/rag config 2025-04-12 16:33:36 -07:00
Timothy Jaeryang Baek
9b20ef4922 refac 2025-03-20 14:01:47 -07:00
genjuro
07098c6352 perf: set shorter timeout for playwright and make it configurable 2025-03-20 15:28:09 +08:00
leilibj
3e8546135d
fix: correct incorrect usage of log.exception method 2025-03-19 13:04:34 +08:00
orenzhang
c761e4fd08
feat(trace): opentelemetry instrument 2025-03-10 22:27:31 +08:00
Luke
7917128ed3 enh: enable configuration for tavily extract depth 2025-03-08 00:43:02 -05:00
Luke
987954c817 feat: Add Tavily extract web loader integration 2025-03-06 18:15:18 -05:00
Timothy Jaeryang Baek
eeb00a5ca2 chore: format 2025-02-20 01:01:29 -08:00
Youggls
0fb3c08181 feat: Add Firecrawl web loader integration 2025-02-19 16:54:44 +08:00
Rory
10e0c81de9 Merge remote-tracking branch 'upstream/dev' into playwright
# Conflicts:
#	backend/open_webui/retrieval/web/utils.py
#	backend/open_webui/routers/retrieval.py
2025-02-17 21:53:39 -06:00
Rory
bc82f48ebf refac: RAG_WEB_LOADER -> RAG_WEB_LOADER_ENGINE 2025-02-17 21:43:32 -06:00
Craig Quiter
e67eb89e05 style: black format 2025-02-15 10:53:16 -08:00
Rory
8e9b00a017 Fix docstring 2025-02-14 22:48:15 -06:00
Rory
aa2b764d74 Finalize incomplete merge to update playwright branch
Introduced feature parity for trust_env
2025-02-14 22:32:45 -06:00
Rory
4da220c513 Merge remote-tracking branch 'upstream/dev' into playwright
# Conflicts:
#	backend/open_webui/config.py
#	backend/open_webui/main.py
#	backend/open_webui/retrieval/web/utils.py
#	backend/open_webui/routers/retrieval.py
#	backend/open_webui/utils/middleware.py
#	pyproject.toml
2025-02-14 20:48:22 -06:00
Guofeng Yi
b38acc8559
Merge branch 'dev' into feate-webloader-support-proxy 2025-02-15 09:50:02 +08:00
Yimi81
d3f71930f0 web loader support proxy 2025-02-14 07:15:09 +00:00
Yimi81
ceef600223 support async load for websearch 2025-02-14 07:05:10 +00:00
Rory
ec6fe9939b Merge remote-tracking branch 'upstream/dev' into playwright 2025-02-05 17:47:58 -06:00