* test(background_steps): add comprehensive tests for batch processing and memory management
- Add test for catalog upserts in batches of at most 100 files
- Add test for catalog deletes in batches of at most 100 paths
- Add test for index memory budget reducing batches to one file
- Add test for memory target limiting cumulative batch size
- Add test for invalid batch memory settings rejection
- Add test for continuing after one batch fails
- Add test for yielding to event loop while building batch
- Add test for modified file reusing unchanged embedding
- Add test for reporting memory estimation failure without aborting
feat(update_changes): implement bounded batch processing with memory management
- Add configurable batch parameters with default values
- Implement memory budget calculation based on available system memory
- Add file inspection and memory estimation before processing
- Implement batch flushing when limits are reached
- Add proper error handling for batch operations
- Support async yielding during batch building
- Add comprehensive validation for batch configuration parameters
- Implement memory estimation for indexing operations
- Add batch size limiting for delete operations
* test(steps): add tests for memory estimation failure handling
- Add test case for isolated file processing when memory estimation fails
- Add test case for proper release of flushed items before building next file
- Implement weak reference tracking to verify payload lifetime management
- Create parametrized tests for both source and item memory estimation methods
- Add assertions to verify single-item batch behavior on estimation failures
- Include comprehensive error handling verification for memory budget calculations
* chore(version): bump version to 0.4.1.3
- Update __version__ from 0.4.1.2 to 0.4.1.3 in __init__.py
* feat(index): support batch settings from environment
* refactor(index): use direct batch defaults
* refactor(index): configure memory estimates through step args
* ci: simplify Windows smoke dependencies
* feat(index): add file size limits and oversized file handling
- Implement max_file_bytes configuration option for content processing jobs
- Add default 20MB file size limit for background processing in default config
- Skip oversized files during auto_resource step with appropriate metadata
- Clear stale index entries when oversized files are modified
- Add size-based filtering logic to update_changes step with skip reporting
- Include file size validation in UpdateIndexStep with proper response handling
- Add comprehensive tests for oversized file scenarios in auto_resource and update_index
- Document file size limits in constants with appropriate thresholds
* chore(version): bump version to 0.4.1.1
- Update __version__ from 0.4.1.0 to 0.4.1.1 in __init__.py
* fix(index): isolate batch metadata and handle file races
* docs: rename vault_dir to workspace_dir in documentation and examples
* refactor(extract): format long method call across multiple lines
* refactor(extract): format system prompt parameters for better readability
