mirror of
https://github.com/open-webui/open-webui.git
synced 2026-09-17 23:52:29 +00:00
061f5e3a6d
* perf: stop re-parsing the whole tool-argument buffer on every streamed chunk Converting an OpenAI stream to Anthropic events buffers each tool call's arguments and, to find out when the JSON is complete, parsed the entire buffer again on every chunk. A tool call with large arguments pays that parse thousands of times, and the cost grows with the square of the argument size. The parse now runs only when the buffer could actually be complete. A JSON object can only close on its final brace, so a chunk that does not end there cannot complete it. Arguments that are not an object, or that start with whitespace, keep parsing on every chunk exactly as before. Measured on CPython 3.12 with 130 KB of tool arguments over 7648 chunks: | | before | after | |---|---|---| | parses | 7648 | 1 | | time | 382 ms | 0.82 ms | The block closes on exactly the same chunk as before, verified by replaying randomized fragmentations of objects with braces inside strings, escaped characters, unicode escapes, arrays, bare scalars, leading and trailing whitespace and a buffer that never completes, against both JSON backends. * refactor: read tool['arguments'] directly in the JSON completion guard Restores the pre-existing comment above the guard to its original wording and drops the `buffered` local, so the guard and the parse call both read `tool['arguments']`, the name the rest of the file already uses for that buffer. Behaviour is unchanged: same three conditions in the same order, same short-circuit result. * perf: strip whitespace in the tool-argument completion guard The character guard only looked at the first and last byte of the buffer, so a chunk that ended in a space still triggered a full parse and a tool argument with leading whitespace fell back to parsing on every chunk. Stripping first collapses both cases to a single parse at the end of the stream. Measured on a streamed tool call, parses and wall time for the whole stream, orjson on the left of the slash and stdlib json on the right: | argument shape | before | after | |---|---|---| | 20 KB string, char-by-char deltas | 3678 parses, 56 / 28 ms | 1 parse, 3.1 / 3.1 ms | | 200 KB, 20-char deltas | 1473 parses, 176 / 63 ms | 1 parse, 3.1 / 2.8 ms | | 8 KB prose, leading whitespace | 715 parses, 5.5 / 2.5 ms | 1 parse, 0.18 ms | | 8 KB of spaces inside a value | 713 parses, 6.0 / 2.9 ms | 1 parse, 0.83 / 0.72 ms | The strip costs about 20 ns per delta on arguments that have no whitespace at either end, which is where the old form was already optimal: a 20 KB compact argument goes from 191 to 216 us over 1786 deltas. Soundness is unchanged, the guard can still only skip a parse that would have failed: 2660892 buffers (exhaustive to length 6 over a JSON-lexical alphabet, every prefix of 26 named cases with a trailing byte appended, and every codepoint below U+3000 after a complete document) with zero cases where a parse would have succeeded. |
||
|---|---|---|
| .. | ||
| data | ||
| internal | ||
| migrations | ||
| models | ||
| retrieval | ||
| routers | ||
| socket | ||
| static | ||
| storage | ||
| tools | ||
| utils | ||
| __init__.py | ||
| alembic.ini | ||
| config.py | ||
| constants.py | ||
| env.py | ||
| events.py | ||
| functions.py | ||
| main.py | ||
| tasks.py | ||