mirror of
https://github.com/open-webui/open-webui.git
synced 2026-09-23 00:41:56 +00:00
28a4df7921
`SafeFireCrawlLoader` inherits `RateLimitMixin` and accepts `requests_per_second` in its constructor, but neither `lazy_load` nor `alazy_load` ever invoked `_sync_wait_for_rate_limit` / `_wait_for_rate_limit`. The configured value (env `WEB_LOADER_CONCURRENT_REQUESTS`) was silently ignored: the loader fired requests as fast as the loop ran. `_safe_process_url` is not used because it's redundant for FireCrawl: FireCrawl fetches the target URL on the user's behalf, and `verify_ssl` is already honored in the FireCrawl payload (`skipTlsVerification`). |
||
|---|---|---|
| .. | ||
| loaders | ||
| models | ||
| vector | ||
| web | ||
| utils.py | ||