The store retried a connection failure or a timeout on top of the embedding
wrapper and the provider SDK, which already retry both, so a persistent
timeout on a batch reached the provider 36 times instead of 12: the OpenAI
model retries a batch max_retries + 1 times and the SDK retries each of those
attempts.
Transport failures stay provider-wide -- the request is never split per item
-- but the retry is now owned explicitly instead of implicitly:
* a whole-batch request leaves it to the layers below, which already retry it;
* an isolated single-item attempt keeps the store's backoff, because that
attempt is the last chance to keep the one vector the isolation step is
recovering.
Rate limiting is unchanged. The transport cases now pin both budgets, and
three tests drive a real OpenAIAsEmbedding wrapper and a real
openai.AsyncClient over an httpx.MockTransport, so the provider attempt count
is measured rather than assumed.
A store-level retry for a provider 5xx stacked on top of the retries the
embedding wrapper and the provider SDK already perform, so a batch the SDK
answered after 12 attempts became 36. Retry ownership stays with those
layers; the store answers a provider-wide 5xx once and gives up.
Transport failures keep their retry because they describe a connection that
no lower layer has reported, and the opt-in quota policy is untouched.
Tests updated to pin the new boundary: a 503 is answered once for the
OpenAI-shaped, Gemini-shaped and DashScope-shaped errors, while a 429 keeps
its retry from the rate-limit branch.
`_compute_batch()` set `is_healthy = True` whenever the batch produced at least one
vector. Per-item isolation can recover an early item and then meet a provider-wide
failure (exhausted quota, authentication, exhausted transient retries), which
`_call_with_retry()` correctly records as unhealthy before it stops. The recovered
vector then overwrote that state, so a request that ended in an outage reported the
provider as healthy.
The terminal-failure flag is now threaded from `_embed_individually()` back to
`_compute_batch()`. The vectors recovered before the failure are still returned and
cached; only the health state now follows the failure. This matters because
`LocalFileStore._recover_after_real_request()` schedules missing-vector backfill off
the unhealthy-to-healthy transition: a hidden outage would either trigger backfill
while the provider is still unavailable or suppress a later genuine recovery.
Adds a regression test through `get_embeddings()` that checks both the retained
partial vector and the final unhealthy state. The existing isolation test covers
`_call_with_retry()` directly, so it never saw the overwrite.
- `_embed_individually` now stops asking for the remaining items as soon as one isolated
item fails provider-wide (credentials, missing model, exhausted quota, exhausted
transient retries). No remaining input can explain that failure, so continuing only
multiplies the rejected requests. The vectors already recovered are kept and the items
that were never answered stay `None`.
- `_call_with_retry` reports that outcome through an optional `terminal_failure`
out-parameter, so its return type and every existing call site stay unchanged.
- `_error_code` reads the business error code from the attribute, the structured body, or
the serialized message. AgentScope's DashScope embedding raises a plain `RuntimeError`
with the response embedded in the message, so a quota error was classified as a rate
limit and never used the configured `quota_retry_delay`.
Regression tests cover both halves of the isolation boundary (a provider-wide failure
stops the loop; an input-dependent failure does not) and the serialized DashScope quota
path with quota retries enabled.
The per-item fallback still multiplied service failures into one request per item.
Both gaps come from the fallback this PR adds, so they are fixed here.
Status extraction
- `_http_status()` reads the status from `status_code`, an integer `code` (Google GenAI
exposes the HTTP status there), `response.status_code` (httpx-style wrappers), and the
structured `status_code` embedded in the message (AgentScope's DashScope embedding turns
every non-200 response into a plain `RuntimeError`).
- Only integers are read from `code`. A string `code` names an error
(`insufficient_quota`, `rate_limit_exceeded`, `InvalidApiKey`); reading it as a status
would misclassify quota and rate-limit errors.
- `_is_provider_wide_failure`, `_is_transient_provider_failure`, `_is_rate_limited` and
`_is_input_dependent_failure` all classify through it.
Splitting
- The generic fallback splits only when `_is_input_dependent_failure()` matches: a
context-length rejection, the one error a single over-long item explains by itself.
Anything unrecognized is provider-wide and is never answered with one request per item.
- This narrows the isolation this PR adds. An input-dependent error that carries neither a
status nor an input-length signal is no longer split, so it costs its batch instead of
multiplying the requests. Erring towards the batch keeps one provider failure from
becoming N failures and losing the healthy vectors anyway.
- `_embed_individually()` drops the redundant `len(single) == 1` check: the
`_call_with_retry([text])` contract only returns a result of the expected length.
Verified against the real exception classes: `google.genai.errors.APIError.code` is an
`int`, and a real DashScope response yields a `RuntimeError` whose message carries
`"status_code"`. Regression coverage uses the real classes when available and mirrors
their shape otherwise.
Fourth review follow-up on #579.
The classifier added in c9448ae covered 401/403 and 5xx, but not 404. A missing
model or a wrong embedding route answers with 404 (SDK `NotFoundError`), which
no individual input can explain, so the batch still fell through to
`_embed_individually()`: a ten-item request against a bad model name produced
eleven requests, all failing for the same reason.
404 is now part of the provider-wide set (`_PROVIDER_WIDE_STATUS_CODES`) and
`NotFoundError` joins the class-name fallback. Like 401/403 it is not
transient, so the request is reported as failed once instead of being retried
or split. The context-length `400` is deliberately left out of that set: an
over-limit item is the one status a single input can explain by itself, so it
has to stay isolatable.
Regression coverage:
- test_provider_wide_statuses_exclude_input_dependent_bad_request
401/403/404/500 -> provider-wide; an over-limit 400 -> not provider-wide.
- test_provider_wide_not_found_errors_are_not_split
404 -> batch sizes [10], no sleeps, no single-item requests.
The existing context-length isolation, 401/403, 5xx, quota, rate-limit and SDK
transport tests are retained.
Local verification with the repo's own hook arguments:
pylint -> Your code has been rated at 10.00/10, exit 0
black -> 2 files would be left unchanged
flake8 -> clean
Branch-order discriminator check (classifiers extracted from the production
file and executed against each error type):
case old new
400 context-length SPLIT SPLIT
401 / 403 SPLIT STOP
404 NotFoundError SPLIT STOP
500 SPLIT RETRY
SDK APIConnectionError SPLIT RETRY
SDK APITimeoutError SPLIT RETRY
builtin ConnectionError RETRY RETRY
builtin TimeoutError RETRY RETRY
429 RETRY RETRY
insufficient_quota STOP STOP
Third review follow-up on #579.
`_call_with_retry` only caught the built-in `TimeoutError`, `ConnectionError`
and `OSError`. `openai.APIConnectionError` and `openai.APITimeoutError` do not
inherit from any of them (they go APIError -> OpenAIError -> Exception), so
they reached the generic handler: `_is_provider_wide_failure()` did not
recognize them either, and a multi-item request fanned out into per-item
requests that failed for exactly the same reason. The same gap meant a
single-item request got no ReMe-level retry at all after a transport failure.
Transport failures are now recognized by a dedicated classifier,
`_is_transient_transport_failure()`, covering the built-in socket errors as
well as the SDK/httpx exception types (matched across the MRO, so subclasses
such as `APITimeoutError` are included) without importing the provider SDK.
They retry the original request with the existing backoff and never split; the
retry budget is unchanged.
Regression coverage:
- test_sdk_transport_errors_are_retried_without_splitting
APIConnectionError / APITimeoutError -> batch sizes [10, 10, 10], sleeps
[1.0, 2.0], then failure with no single-item requests.
- test_sdk_transport_error_recovers_without_losing_the_batch
one APITimeoutError then success -> [[1.0, 0.0]] * 10, batch sizes [10, 10].
- test_isolated_item_retries_an_sdk_transport_error
context-length rejection, then APIConnectionError on one isolated item ->
that item's vector is retained.
The existing context-length, 401/403, 5xx, quota and rate-limit tests are
retained.
Local verification with the repo's own hook arguments:
pylint -> Your code has been rated at 10.00/10, exit 0
black -> 2 files would be left unchanged
flake8 -> clean
Branch-order discriminator check (classifiers extracted from the production
file and executed against each error type):
case old new
400 context-length SPLIT SPLIT
401 / 403 SPLIT STOP
500 SPLIT RETRY
SDK APIConnectionError SPLIT RETRY
SDK APITimeoutError SPLIT RETRY
builtin ConnectionError RETRY RETRY
builtin TimeoutError RETRY RETRY
429 RETRY RETRY
insufficient_quota STOP STOP
Review follow-up on the P2 raised against 4d8a3c5.
`_call_with_retry` sent every unrecognized exception to `_embed_individually`
whenever the request had more than one item. Only rate limiting and
`insufficient_quota` were excluded, so a 401, a 403 or a 5xx still split a
ten-item request into ten extra single-item requests, none of which can
succeed: credentials, permissions and server outages are properties of the
endpoint, not of any individual input.
The split path is now gated on a dedicated classifier,
`_is_provider_wide_failure()`, which recognizes 401/403 and 5xx by status
code (with the OpenAI SDK class names as a fallback for exceptions that do
not expose one). Transient server errors keep their retry policy through the
existing backoff; the remaining provider-wide failures stop the loop and
report unhealthy, like the quota branch. Input-dependent rejections such as
the per-item context-length limit are unchanged and still isolate one item
at a time.
Regression coverage: test_provider_wide_credentials_errors_are_not_split_into_per_item_requests
(401/403 -> a single request of size 10) and
test_provider_wide_server_errors_are_retried_without_splitting
(500 -> batch sizes [10, 10, 10] with sleeps [1.0, 2.0]). The existing
context-length, quota, 429 and per-item transient-retry tests are retained.
Local verification, using the repo's own hook arguments:
pylint -> Your code has been rated at 10.00/10, exit 0
black -> 2 files would be left unchanged
flake8 -> clean
Classifier discriminator check (old vs new split decision, extracted from the
production file and executed):
400 context-length -> SPLIT / SPLIT
401, 403, 500, 504 -> SPLIT / NO-SPLIT
429 and insufficient_quota keep their own earlier branches.
Review follow-up on the two P2 findings in #579:
* quota exhaustion no longer falls into the per-item split. The store now gives
up explicitly when the provider reports insufficient_quota, including after
the opt-in quota retries are spent, because no single item is to blame and
splitting only multiplied the rejected requests (3 items -> 4 requests).
* the per-item fallback goes back through `_call_with_retry`, so a 429,
connection error or timeout on an isolated item keeps the existing backoff
and retry policy instead of silently dropping that item's vector. A one-item
request still cannot split again.
Regression tests cover both findings: a quota failure stays a single request
with the default policy and with exhausted quota retries, and a transient
failure on one item of a split batch still ends with every healthy sibling
embedded.
python -m pytest tests/unit/test_local_embedding_store.py tests/unit/test_file_store_consistency.py -q
-> 83 passed, 41 skipped
A provider-side request error rejects an entire batch, so a single over-limit
item used to cost the vectors of every item beside it. QwenPaw #8040 shows two
poison chunks turning into exactly twenty lost vectors, while the backfill loop
still reported full success.
Retry a rejected multi-item request one item at a time so only the item that
actually fails is left without a vector. Single-item requests keep their
previous behaviour, and provider-level failures (429 / quota) keep using the
existing retry paths instead of being split.
Also make the backfill loop report what it actually filled, and name the failing
chunk ids in the final reindex error. `processed` counts attempts, so it claimed
success after a partial loss.
* fix(agent_wrapper): release Codex threads after each turn
Loaded Codex threads and their per-thread MCP server subprocesses were
never unloaded from the app-server, growing file descriptors and child
processes monotonically until the wrapper was closed (fd exhaustion,
os error 24). Reply, streaming reply, and compact_session now
unsubscribe the thread in a turn-scoped finally block by default; the
new release_threads=False option restores the previous behavior.
Unsubscribe failures are logged and never break the turn, and threads
remain resumable after being unloaded.
Closesagentscope-ai/ReMe#609
* fix(agent_wrapper): keep tool-context guard on failed release and cover stream startup
A failed thread/unsubscribe left the thread loaded in the app-server
but dropped its tool-context entry, so a later resume with changed
tool_context_id or injected_job_kwargs would silently bypass the
constraint check. Keep the guard entry unless the unsubscribe request
succeeds.
reply_stream now releases the thread from an outer try/finally scoped
right after _open_thread, so threads are also unloaded when turn/start
raises or the caller is cancelled while awaiting it.
* feat(docker): add container deployment and multi-platform release workflow
* fix(llm): initialize providers on model access and preserve runtime injection
* test(llm): keep rejected model updates independent of credentials
* feat(service): bind network services to all interfaces
* fix(service): keep network listeners local by default
* fix(service): make remote access explicit
* feat(plugins): auto-tag generated reports
* fix(logging): forward host records on Python 3.13
* refactor(tags): decouple auto tagging from index updates
* fix(tags): bind auto tagging to configured index
* fix(tags): preserve standalone default index
* fix(tags): make auto tagging best effort
* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* Add configurable tag indexing and tag listing
* Add tag-filtered hybrid search with exact candidate ranking
* Remove obsolete generated files
* Rename tag index key to tag_key and reject reserved fields
* Extract automatic tagging into a dedicated step
* Restrict frontmatter updates to authorized keys
* Refine tag filtering and automatic memory tagging
* Require underscore-separated tags in auto-tag prompts
- Forbid spaces in tags and require underscores (e.g. sam_altman) in both
English and Chinese auto_tag prompts, with English examples switched to
English entities (OpenAI, gold)
- Drop prompt-string assertions superseded by the new tagging rule
- Merge construction/runtime tag_key validation tests into one parametrized case
* Make max_tags_per_file configurable in auto-tag step
* Consolidate tag index tests
* Fix search test fixture lint warnings
* Align tag contracts and index health behavior
* Fall back when tag index is unavailable
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* Add optional tag generation and normalization to auto memory
* Add tag index components and clean up temporary JSONL files
* Preserve tag index state when reconciliation fails
* Refactor and streamline application implementation
* Fix pylint C1803 warnings in tag normalization tests
* Document optional tag index configuration
* Make tag index failures non-blocking and disable auto-memory tags
* fix(tag-index): fail closed and support reindexing
* fix(tag-index): preserve complete query expressions
---------
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* fix(embedding): retry 429 rate-limit errors instead of dropping the batch
openai.RateLimitError is not a TimeoutError/ConnectionError/OSError, so
_call_with_retry's except Exception branch caught it and returned None on
the first attempt with zero backoff. Add _is_rate_limited, mirroring the
existing _is_insufficient_quota duck-typed check, and retry a 429 with the
same exponential backoff used for network errors.
* fix(embedding): insufficient_quota errors carrying status_code=429 no longer bypass quota handling
_is_rate_limited() checked status_code == 429 first, so an OpenAI-compatible
insufficient_quota error (which also carries status_code=429) matched the
generic rate-limit branch before the code=insufficient_quota check ever ran.
That meant a real quota exhaustion retried on the wrong backoff (or not at
all, when quota_retry_delay is unset) instead of the dedicated quota_retry_delay
wait.
_is_rate_limited() now defers to _is_insufficient_quota() first. The existing
quota test double now sets status_code=429 to match the real OpenAI error
shape, which is what exposes the regression without the fix.
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
* fix(embedding): log rate-limit retries
---------
Signed-off-by: Amir Fathi <amirfathi.me@gmail.com>
Co-authored-by: jinli.yl <jinli.yl@alibaba-inc.com>
* feat(core): support live component replacement
* fix(core): serialize replacement with app lifecycle
* fix(core): preserve state during component replacement
* fix(core): roll back partial component startup
* chore(deps): add 'web' extra to reme-ai and remove reme_studio from core dependencies
- Updated reme-ai dependency to include 'as' and 'web' extras
- Removed reme_studio from core dependency list to avoid duplication or unnecessary install
* fix(core): preserve persisted state on startup failure
* chore(release): prepare ReMe 0.4.1.9
* refactor(config): remove daily_cookbook and streamline plugin configs
- Delete the entire daily_cookbook.yaml standalone application config
- Remove qwenpaw dependencies verification and related CI workflow steps
- Simplify release workflows by removing qwenpaw verification and enforcing reme-ai >=0.4.1.9
- Update plugin start commands and examples to use 'default' or 'demo' configs instead of daily_cookbook
- Adjust imports and tests related to daily_cookbook removal and injected_job_kwargs enhancements
- Refactor agent wrapper to support injected_job_kwargs for job parameter injection in auto-fin and daily-paper
- Improve daily_paper digest prompt to include configured daily directory and correct historical search constraints
- Update dependency versions in pyproject.toml files to require reme-ai >=0.4.1.9 and remove qwenpaw optional dependencies
- Clean up unused environment variables and obsolete test cases related to daily_cookbook and verification steps
* fix(local_embedding_store): retry batch computation on vector space changes
- Add up to 3 attempts to recompute embedding batch if vector space changes during processing
- Log warnings when maximum retries reached and discard stale results
- Prevent caching results from outdated vector spaces to maintain consistency
- Add tests to verify retry behavior and abort after continuous vector space churn
fix(daily_paper): update digest search logic and tests
- Change search to query existing memory, not only previous articles in daily_dir
- Allow multiple searches outside daily_dir but limit links to dated markdown in daily_dir before today
- Update test assertions to reflect revised search and linking rules
* fix(embedding): retry vector space changes per request
* feat(service): expose MCP through HTTP backend
Serve JSON/SSE job endpoints and streamable HTTP MCP from one FastAPI application, sharing the same jobs and application lifecycle. Preserve the standalone MCP backend, add configurable MCP HTTP settings, update startup metadata and integration docs, and cover routing, lifecycle, configuration, and compatibility behavior with unit tests.
* fix(service): preserve MCP request protections
Route the exact MCP path through the complete FastMCP ASGI application so its middleware and state remain active. Reject non-literal MCP paths and validate reserved Job conflicts before tolerant service registration. Add regression coverage for middleware preservation, route syntax, and startup failure.
* fix(service): reject encoded MCP paths
Reject percent signs in mcp_path so ASGI path decoding cannot turn an accepted configuration into an unreachable route. Cover encoded slash, space, and double-encoded slash inputs.
* fix: recover embedding after transient health failure
* refactor(embedding_store): remove provider_success_count and simplify health recovery logic
- Deleted provider_success_count attribute and related methods across embedding and file stores
- Updated _recover_after_real_request to rely solely on is_healthy flag for recovery decisions
- Removed redundant counting logic for provider successes during embedding operations
- Cleaned up health status management to streamline provider recovery detection
- Adjusted unit tests to align with removal of provider_success_count and maintain health checks consistency
* refactor(embedding_store): use default health check timeout
* fix(embedding_store): ensure is_healthy remains unchanged on cache hits
- Updated get_embeddings docstring to clarify cache hits must not alter is_healthy state
- Improved code comment for embedding dimension matching method
* fix(file_store): make embedding recovery race-safe
* ci: use default CodeQL query suite
* fix(file_store): preserve queued embedding rebuilds
* fix(file_store): preserve verified recovery without chunks
* feat: add entry-point plugin system
* fix: harden plugin config and client loading
* docs(workflow): add detailed manual for publishing reme-auto-fin to PyPI
- Provide step-by-step instructions for updating project.version and merging branches
- Explain dependency verification for reme-ai on PyPI during build
- Specify requirements for GitHub Actions secret configuration and version uniqueness
- Describe manual workflow triggering and input of version number
- Recommend publishing order for related projects
- Clarify that only manual dispatch triggers publishing, no automatic triggers on push or tag
* feat: support plugin-defined component types
* refactor: simplify plugin configuration
* fix: isolate plugin loading and defer client fallback
* refactor: freeze built-in component registry
* fix: isolate config entry point loading
* fix: complete auto-fin package metadata
* feat: refine local-first research workflows
* fix: delegate structured output tool choice
* refactor(auto-fin): fetch and filter rolling CLS news
* fix(auto-fin): keep imports portable across platforms
* feat(auto-fin): expose CLS fetch controls
* fix(auto-fin): propagate configurable news window
* feat(auto_fin): normalize hybrid wikilinks in report body
- Add _normalize_hybrid_wikilinks method to remove redundant Markdown destinations
- Use regex to identify hybrid wikilinks with optional destinations
- Replace redundant destinations with simpler wikilink format for clarity
- Ensure normalization is failure-safe with exception handling and logging
- Update report body normalization process to apply hybrid wikilink fix
- Add unit tests to verify correct normalization and failure safety behavior
* fix(dream): serialize integration with application-wide asyncio lock
- Add application-wide asyncio.Lock to serialize digest writes during integration
- Update _snapshot_digest to capture metadata per bucket
- Validate bucket association when recovering from file changes
- Add tests ensuring recovery only from the correct bucket
- Add tests confirming integration lock is shared across application context
- Enhance strict topic YAML loading validation in dream utils
- Add tests for strict topic loading rejecting invalid or lossy fields
* fix(cookbook): enable configurable job_tools for digest and merge steps
- Update daily_cookbook.yaml to add job_tools: [memory_search, read] in digest steps
- Modify DailyPaperDigestStep to read job_tools from kwargs instead of fixed list
- Modify AutoFinMergeStep to similarly read job_tools from kwargs
- Update tests to pass job_tools explicitly when invoking these steps
- Remove hardcoded _TOOLS constants and replace with dynamic job_tools handling
* fix: retry incomplete dream receipts
* perf(pdf): increase max PDF pages limit from 20 to 35
- Updated configuration max_pdf_pages from 20 to 35 in daily_cookbook.yaml
- Modified code to extract up to 35 pages instead of 20 in analyze.py
- Updated README and README_ZH to document the increased max_pdf_pages
- Adjusted unit test assertions to reflect new max_pdf_pages limit of 35
* fix memory integration and daily paper links
* docs clarify cookbook tool usage
* fix(embedding): isolate caches by vector space
* fix(embedding): stabilize cache space switching
* Revert "fix(embedding): stabilize cache space switching"
This reverts commit 74193c9a0a.
* fix(embedding): include resolved OpenAI endpoint in cache ID
* fix(embedding): stabilize cache space switching
* fix(embedding): isolate Ollama endpoint caches
* feat(backend): improve workspace support for web clients
* fix(config): preserve the default workspace directory
* chore(reme): bump version to 0.4.1.5
- Update __version__ from 0.4.1.4 to 0.4.1.5 in initialization file
* fix(chat): disable builtin tools in read-only mode
* fix(agent): make builtin tools opt-in
* fix(list): tolerate files removed during mtime sort
* fix(chat): expose complete read-only job set
Always apply conditional-line filtering so tagged prompt lines are removed unless the corresponding boolean flag is explicitly true. Add regressions for omitted flags with and without format variables.
Test: pytest tests/unit/test_prompt_handler.py -q
* feat(file_store): add ZvecLocalFileStore backend
- Implement ZvecLocalFileStore with native zvec collection for ANN search.
- Keep JSONL chunks as the source of truth; rebuild collection from chunks
when sidecar digest/dimension/HNSW M mismatch is detected.
- Add dedicated unit tests in tests/unit/test_zvec_file_store.py.
- Parametrize existing file_store consistency tests to cover both
LocalFileStore and ZvecLocalFileStore.
- Register the new backend in reme/components/file_store/__init__.py.
* fix(file_store): fix zvec collection sync and content validation, declare zvec dependency
* feat: simplify local links and support line anchors
* fix: align line anchor tests with CI lint
* fix: preserve local links across file moves
* fix: encode markdown paths when rewriting links
* refactor(read): keep explicit line range parameters
* fix: simplify legacy link predicate compatibility
* docs: align local link behavior with implementation
* fix: skip unsupported markdown destination escapes
* fix: normalize workspace link paths across platforms
* fix: bound markdown link scanning
* fix: keep local link processing linear
* docs: clarify permissive markdown link parsing
* fix: handle local link processing failures
* refactor: limit file links to wikilink syntax
* docs: align wikilink contract with implementation
* fix: normalize dream and neighbor paths on Windows
* fix: resolve workspace path for neighbor expansion
