ReMe/reme4/steps/file_io/read_image.py
jinliyl 041f957a7f
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
Tests ReMe / Unit Tests - py3.10 (push) Waiting to run
Tests ReMe / Unit Tests - py3.13 (push) Waiting to run
refactor(components) components and file I/O, fix method calls and validation (#268)
* refactor(components): extract shared component state into mixin

- Introduce ComponentMixin class with shared state for components and steps
- Move identity, config, and vault path functionality to ComponentMixin
- Update BaseComponent to inherit from ComponentMixin
- Update BaseStep to inherit from ComponentMixin
- Consolidate vault path helper methods in ComponentMixin
- Remove duplicate vault path implementations from BaseComponent and BaseStep
- Add ComponentMixin to components module exports

* refactor(file_io): implement path locks cache eviction mechanism

- Add _PATH_LOCKS_MAX constant set to 1024 for cache size limit
- Implement cache eviction logic when locks exceed maximum capacity
- Remove half of unlocked entries when cache limit is reached
- Use list comprehension to identify unlocked locks for removal
- Maintain existing path normalization and locking behavior

fix(edit): correct method call from public to private fail method

- Change self.fail to self._fail for internal error handling
- Maintain consistent private method usage within class

fix(mcp_client): change pop to get for optional command and args

- Replace kwargs.pop with kwargs.get to avoid removing keys
- Preserve original kwargs dictionary contents
- Maintain default empty string and list values

feat(reme): add client backend validation with error raising

- Check if client_cls is None before instantiation
- Raise ValueError with descriptive message for unknown backends
- Provide clear error feedback for invalid backend configurations

* fix(components): move directory creation to start method

- Moved component_metadata_path.mkdir call from __init__ to _start in base_keyword_index
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_graph
- Moved component_metadata_path.mkdir call from __init__ to _start in local_file_store
- Ensures directory creation happens after component initialization
- Prevents potential issues with path creation during object construction

* fix(steps): replace assertions with runtime errors for app_context validation

- Replace assert statements with explicit RuntimeError exceptions when app_context is None
- Add descriptive error messages for better debugging when resolving components
- Replace assert in resolve_component method with proper exception handling
- Replace assert in get_file_parser method with proper exception handling
- Maintain same functionality while improving error reporting clarity

* refactor(file_io): split file IO utilities into modular components

- Move daily note helpers to separate _daily_index module
- Extract path validation and resolution to new _path module
- Remove unused code and imports from _file_io module
- Update import statements across affected modules
- Introduce WikilinkHandler utility for link parsing
- Replace regex-based link extraction with WikilinkHandler
- Add integration JSONL files to gitignore
- Consolidate file locking mechanism in _file_io module

* style(formatter): fix spacing issues in file IO and chunked file parser

- Fixed whitespace around colon in slice notation in file_io.py
- Corrected spacing around colon in slice notation in chunked_file_parser.py
- Applied consistent formatting for array slicing operations
- Improved code readability by standardizing space placement in ranges

* refactor(steps): replace property-based component resolution with Ref descriptor

- Introduce Ref descriptor class for lazy component dependency resolution
- Replace _resolve method and individual properties with Ref descriptors
- Add as_llm, as_llm_formatter, as_token_counter, file_store, and embedding Ref attributes
- Remove legacy property methods and resolve logic from BaseStep
- Add cache clearing mechanism for Ref values during step calls
- Update UpdateCatalogStep to use Ref instead of property-based resolution
2026-06-01 11:35:19 +08:00

117 lines
4.4 KiB
Python

"""Read an image file as base64; oversized images degrade to path + metadata."""
import base64
import aiofiles
import aiofiles.os
from ._path import NON_IMAGE_WARNING, gate_image, resolve_path
from ..base_step import BaseStep
from ...components import R
from ...constants import DEFAULT_MAX_IMAGE_BYTES
@R.register("read_image_step")
class ReadImageStep(BaseStep):
"""Read an image file under ``vault_dir`` and return base64 in ``answer``.
Step-level attributes (``kwargs``, configured in yaml under ``steps:`` —
not exposed to LLM):
max_bytes (int, default ``DEFAULT_MAX_IMAGE_BYTES``): cap for the
base64 path. Above this, ``answer`` carries a notice and
``metadata.oversized=True`` (no base64).
Invariants (callers depend on these):
- Normal branch: ``answer`` is pure base64 (no ``data:`` prefix, no
notice suffix). Use ``f"data:{mime};base64,{answer}"`` if you need a
data URL.
- Oversized branch (``metadata.oversized=True``): ``answer`` is a
human-readable notice, **not** base64. Inspect ``metadata.oversized``
before decoding.
- Unknown / missing suffix (compatibility mode): ``answer`` stays pure
base64; ``metadata.non_image_warning=True`` flags it.
"""
def _fail(self, message: str, **meta) -> None:
assert self.context is not None
self.context.response.success = False
self.context.response.answer = f"Error: {message}"
if meta:
self.context.response.metadata.update(meta)
async def execute(self): # pylint: disable=too-many-return-statements
assert self.context is not None
raw = str(self.context.get("path") or "")
max_bytes_raw = self.kwargs.get("max_bytes", DEFAULT_MAX_IMAGE_BYTES)
try:
max_bytes = int(max_bytes_raw)
if max_bytes <= 0:
raise ValueError()
except (TypeError, ValueError):
self._fail(f"`max_bytes` must be a positive integer, got {max_bytes_raw!r}")
return None
target, err = resolve_path(self.vault_path, raw)
if err:
self._fail(err)
return None
target, is_image, mime = gate_image(target)
if not is_image:
self.logger.info(f"[{self.name}] {NON_IMAGE_WARNING} path={target}")
if not target.exists():
self._fail(f"file {target} does not exist", path=str(target))
return None
if not target.is_file():
self._fail(f"path {target} is not a file", path=str(target))
return None
try:
stat = await aiofiles.os.stat(str(target))
except Exception as e: # pylint: disable=broad-except
self._fail(f"read failed: {e}", path=str(target))
return None
size_bytes = stat.st_size
if size_bytes > max_bytes:
self.context.response.success = True
self.context.response.answer = (
f"image exceeds max_bytes ({size_bytes} > {max_bytes}), "
f"base64 omitted; use path={target} to access directly"
)
md = {
"path": str(target),
"size_bytes": size_bytes,
"mime": mime,
"oversized": True,
"max_bytes": max_bytes,
}
if not is_image:
md["non_image_warning"] = True
self.context.response.metadata.update(md)
self.logger.info(
f"[{self.name}] read path={target} size={size_bytes} mime={mime} "
f"oversized=True is_image={is_image}",
)
return self.context.response
try:
async with aiofiles.open(str(target), "rb") as f:
data = await f.read()
except Exception as e: # pylint: disable=broad-except
self._fail(f"read failed: {e}", path=str(target))
return None
b64 = base64.b64encode(data).decode("ascii")
self.context.response.success = True
self.context.response.answer = b64
md = {"path": str(target), "size_bytes": size_bytes, "mime": mime}
if not is_image:
md["non_image_warning"] = True
self.context.response.metadata.update(md)
self.logger.info(
f"[{self.name}] read path={target} size={size_bytes} mime={mime} oversized=False is_image={is_image}",
)
return self.context.response