Commit graph

8 commits

Author SHA1 Message Date
jinli.yl
ca5970c6b0 refactor(component): refactor base components and file store implementations
- Remove abstract methods from base component start/close
- Update BaseJob to remove name parameter and simplify initialization
- Change file modification time field from mtime_ms to modified_time in seconds
- Add type checking imports and improve typing annotations
- Implement LocalFileStore with JSONL persistence for file chunks
- Add MdFileParser with markdown and frontmatter support
- Simplify HttpClient call method with proper kwargs handling
- Remove unused ReMe class methods and create backup version
- Update StreamJob to use step_components instead of steps attribute
2026-04-23 17:44:25 +08:00
jinli.yl
42a3343cb5 feat(core): add core components and application framework
- Introduce Application class for managing application lifecycle
- Add base component classes for LLM formatters and token counters
- Implement embedding model base with caching and batching support
- Create file watcher base with watchfiles integration
- Add job and step base components for workflow execution
- Update base component with async locks and improved lifecycle management
- Register new component types in component registry
- Add application context and runtime context for dependency injection
2026-04-23 16:25:31 +08:00
jinli.yl
fa249a3e38 feat(components): add Anthropic LLM support and enhance component architecture
- Integrate AnthropicChatModel with new AnthropicAsLLM component
- Add component formatters for OpenAI and Anthropic chat models
- Implement token counter component with estimated token counting
- Create base client component for ReMe service communication
- Refactor BaseComponent to remove app_context parameter from _start/_close
- Update embedding model base class to remove retry logic and use npz cache
- Add job component for sequential step execution with BaseJob
- Implement step component base class for LLM workflow execution
- Enhance application context with proper type annotations
- Update component initialization to pass app_context automatically
- Remove asyncio dependency from embedding model cache operations
2026-04-23 14:18:57 +08:00
Sen Huang
ec66fcf2ed
``` (#216)
feat(file-store): add Chroma and SQLite file store implementations

- Add ChromaFileStore and SqliteFileStore classes to support additional
  storage backends
- Export new store classes in __init__.py module

refactor(file-store): enhance BaseFileStore with hybrid search capabilities

- Add keyword scoring utility with word-match ratio and phrase bonus
- Implement hybrid search method that combines vector and keyword results
- Add merge logic for combining vector and keyword search results with
  weighted scoring
- Move abstract methods to separate section for better organization
- Remove redundant search filter parameter from local implementation

refactor(file-store): simplify LocalFileStore implementation

- Remove unused delete_file_chunks and upsert_chunks methods
- Remove redundant update_file_metadata method
- Update chunk counting logic to use file_meta directly
- Simplify keyword search to use new base class scoring utility
- Remove duplicate hybrid search implementation since it's now in base class
```
2026-04-23 10:59:44 +08:00
jinli.yl
33b3aee540 up
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
2026-04-22 14:21:06 +08:00
jinliyl
52f1a33b3a
feat: add file parser support and search filtering (#214)
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
- Add file_parser component with default implementation
- Introduce SearchFilter schema for path and tag filtering
- Implement filter functionality in BaseFileStore and LocalFileStore
- Update file watcher to use parser-based filtering instead of suffix filters
- Register new FILE_PARSER component enum
- Add test_data directory to gitignore

refactor: improve component imports and initialization

- Fix relative imports in application.py
- Add file_parser import to component init
- Initialize registry dict when component type doesn't exist
- Remove circular import in HttpService by using string annotation
- Update config yaml to use proper component names

refactor: enhance file watcher architecture

- Replace MdFileWatcher with more flexible FullFileWatcher and LightFileWatcher
- Remove suffix-based filtering in favor of parser-based approach
- Update BaseFileWatcher to resolve parsers from app context
- Remove unused watch_filter method

refactor: update ReMe core functionality

- Remove memory_path creation
- Simplify dream and proactive methods to return empty strings
- Update config defaults for HTTP service and component backends

docs: update component configuration in paw.yaml

- Change service backend from cmd to http
- Rename components to use correct singular forms
- Add default file parser and file watcher configurations
- Set up local file store with default settings
```

Co-authored-by: huangsen <huangsen.huang@alibaba-inc.com>
2026-04-21 16:44:46 +08:00
jinli.yl
b202fc3fba fix(token-counter): handle non-positive divisor in token estimation
Some checks failed
Pre-commit / run (ubuntu-latest) (push) Has been cancelled
- Changed condition from divisor equals zero to divisor less than or equal to zero
- Prevents division by zero error when estimate_divisor is negative
- Maintains validation logic for invalid divisor values
2026-04-17 11:45:39 +08:00
jinli.yl
baf110e602 feat(components): add token counter and file-based utility components
Some checks are pending
Pre-commit / run (ubuntu-latest) (push) Waiting to run
- Introduce BaseAsTokenCounter and EstimatedAsTokenCounter for token estimation
- Add AsMsgStat and AsBlockStat schema for message statistics tracking
- Implement FileIO class with read/write/append/edit operations
- Create file utility functions for safe async file reading and truncation
- Add MemorySearch component for semantic search in memory files
- Register new component types in ComponentEnum and update imports
- Add constants for default host, port, and truncation limits
- Create BaseService abstract base class for service implementations
- Implement BaseStep with component accessors and lifecycle management
- Add proper __all__ exports for all new modules and components
2026-04-16 20:21:04 +08:00