mirror of
https://github.com/open-webui/open-webui.git
synced 2026-09-23 00:41:56 +00:00
28a4df7921
`SafeFireCrawlLoader` inherits `RateLimitMixin` and accepts `requests_per_second` in its constructor, but neither `lazy_load` nor `alazy_load` ever invoked `_sync_wait_for_rate_limit` / `_wait_for_rate_limit`. The configured value (env `WEB_LOADER_CONCURRENT_REQUESTS`) was silently ignored: the loader fired requests as fast as the loop ran. `_safe_process_url` is not used because it's redundant for FireCrawl: FireCrawl fetches the target URL on the user's behalf, and `verify_ssl` is already honored in the FireCrawl payload (`skipTlsVerification`). |
||
|---|---|---|
| .. | ||
| data | ||
| internal | ||
| migrations | ||
| models | ||
| retrieval | ||
| routers | ||
| socket | ||
| static | ||
| storage | ||
| test | ||
| tools | ||
| utils | ||
| __init__.py | ||
| alembic.ini | ||
| config.py | ||
| constants.py | ||
| env.py | ||
| functions.py | ||
| main.py | ||
| tasks.py | ||